[Devops] Meetup Meetup Meetup!

[Farhad Shahbazi]

On Aug 29, 2012, at 0:01, Michael Renner <michael.renner at amd.co.at> wrote:

> Hoi,
> 
> here're the collected links and other stuff from today's meet up:

thanks for that!

> graylog2:
> 
> Nice frontend for syslog servers. Farhad will write a summary/howto tomorrow including basic rsyslog setup ;)

as promised:

Logging with rsyslog and Graylog2 on Debian squeeze
===================================================

Components
----------

* rsyslog
* graylog2-server
* graylog2-web-interface
* mongodb
* elasticsearch
* apache
* mod_passenger

This[0] is what we are aiming for! 

building Debian packages for graylog2 and graylog2-server
---------------------------------------------------------

We used fpm[1] for easy source to .deb conversion. Install it via

   gem install fpm

For creating the packages you will also need to

   aptitude install ruby1.8 rubygems rake make libopenssl-ruby ruby-dev build-essentia
   gem install bundler

You should built this on a Debian machine similar to the machine you are going to install the packages on.
Download graylog2-server.tar[2] and graylog2-web-interface.tar[3], unpack them somwhere and run `make` in the directories.
The resulting debs will install graylog2-server and graylog2-web-interface to /opt

putting things together
---------------------------

installing dependencies:

   aptitude install openjdk-6-jre mongodb rsyslog apache2 libapache2-mod-passenger

fortunately elasticsearch offers upstream .debs

   wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.8.deb
   dpkg -i elasticsearch-0.19.8.deb

install our previously built debs

configuring mongodb:
enable authentication in '/etc/mongodb.conf'

   auth = true

creating the graylog2 user in mongodb

   mongo
   use admin
   db.addUser('admin', 'myAdminPassword')
   use graylog2
   db.addUser('graylog', 'myGraylogPassword')

followed by a `/etc/init.d/mongodb restart`

configuring elasticsearch:

add the following lines to '/etc/elasticsearch/elasticsearch.yml'

   ...
   network:
   host: localhost
   path:
   logs: /var/log/elasticsearch
   data: /var/data/elasticsearch
   cluster:
   name: graylog2
   ...

followed by a `/etc/init.d/elasticsearch restart`

configuring graylog2:
/etc/graylog2.conf

   ...
   syslog_listen_port = 5140
   syslog_protocol = tcp

   elasticsearch_url = https://127.0.0.1:9200/
   elasticsearch_index_name = graylog2
   ...
   mongodb_useauth = true
   mongodb_user = graylog
   mongodb_password = mongodb_password
   mongodb_host = 127.0.0.1
   mongodb_database = graylog
   mongodb_port = 27017
   ...

/opt/graylog2-web-interface/config/mongoid.yml

   ...
   production:
       host: 127.0.0.1
       port: 27017
       username: graylog
       password: mongodb_pass
       database: graylog2
   ...

you might also want to take a look at the other .yml files in '/opt/graylog2-web-interface/config/'

configuring rsyslog:

configure your rsyslog to receive syslog messages and forward them to the graylog2-server. Since we wanted to have the logs in files as well we configured rsyslog accordingly '/etc/rsyslog.d/logcatch.conf'

   $ModLoad imtcp                                                                                                       
   $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200)
   $InputTCPServerRun 514

   $template DailyPerHostLogs,"/srv/syslog/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"
   *.* -?DailyPerHostLogs;RSYSLOG_FileFormat

   $template GRAYLOG2,"<%PRI%>1 %timegenerated:::date-rfc3339% %HOSTNAME% %syslogtag% - %APP-NAME%: %msg:::drop-last-lf%\n"
   $ActionForwardDefaultTemplate GRAYLOG2
   *.* @@127.0.0.1:5140


serving graylog2-web-interface with apache and mod_passenger:

   aptitude install apache2 libapache2-mod-passenger

enable mod_passenger

   a2enmod passenger

configure the vhost

   <VirtualHost *:80>
       ServerName graylog2.example.com
       ServerAlias graylog2.example.com
       DocumentRoot /opt/graylog2-web-interface/public

       <Directory /opt/graylog2-web-interface/public>
           Allow from all
           Options -MultiViews
       </Directory>

       LogLevel warn
   </VirtualHost>


configuring the clients
---------------------------------

   aptitude install rsyslog

create '/etc/rsyslog.d/logsend.conf'

   *.*  @@ip.of.the.server

This will send all messages (\*.\*) via tcp (@@) to our rsyslog server

If I missed anything or if you have any questions feel free to ask and I'll try to answer to the best of my ability.

Best regards,
Farhad

[0] https://dl.dropbox.com/u/45717448/talks/20120828-graylog2/setup.png
[1] https://github.com/jordansissel/fpm
[2] https://dl.dropbox.com/u/45717448/talks/20120828-graylog2/graylog2-server.tar
[3] https://dl.dropbox.com/u/45717448/talks/20120828-graylog2/graylog2-web-interface.tar