2013. szeptember 25., szerda

The road so far


We are getting closer to the end of the project, so I decided to write a post/tutorial about using syslog-ng with redis and what I’ve implemented so far:
After I've learned some tricks at the beginning of this summer in C by implementing redis client, I started to develop redis destination for syslog-ng. We can collect now log messages and send them to redis via publish or set command, or we can just use incr or hincrby to make statistics.
But first, you’ll have to compile syslog-ng based on this former post.

A short summary without explanation:
git clone -b redis-destination https://github.com/ptichy/syslog-ng-3.4.git
or download: https://github.com/ptichy/syslog-ng-3.4/archive/redis-destination.zip

./autogen.sh && ./configure --prefix <install dir> --enable-debug && make && make install
or if you want just to be sure:
./autogen.sh && ./configure --prefix <install dir> --enable-debug --enable-redis && make && make install

If you want to make statistics in redis from the amount of log messages separated by programs, you can easily use the following config file (if you don’t want to change the default config file, save the following file as mynewconf.conf and run it with sbin/syslog-ng -f mynewconf.conf command):

@version: 3.4
options {
 threaded(no);
}; 

source s_test1 {
 system();
 tcp( port(514));
};

destination d_redis_hincrby {
 redis( command("HINCRBY" "programs" "${PROGRAM}" "1"));
};

log {
 source(s_test1);
 destination(d_redis_hincrby);
};
 
Of course, you can use another type of sources, you can choose for e.g. tcp(ip(10.1.2.3) port(1999)); instead of unix-stream("/tmp/test-log");
The source statement above receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address. See syslog-ng guide for further information (Chapter 6: Collecting log messages — sources and source drivers).
In this case you will get a programs list with the monitored program names and the amount of them. Hence, if you type hgetall programs in redis client, you’ll get a similar message with the name of programs and the number of collected messages:

redis 127.0.0.1:6379> hgetall programs
1) "xchat"
2) "4"
3) "program2"
4) "10"


If you type redis( set("$PROGRAM" "$MSG")); in config file, then it’s completely equivalent with redis( command("SET" "${PROGRAM}" "${MESSAGE}")); 

And finally, see another example from everyday: let's make a stat about browsers that the visitors use: (from apache log, per minutes)

@version: 3.4
options {
 threaded(no);
}; 

source s_apache{
 file("/var/log/apache2/access.log");
};

parser p_apache {
 csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",
  "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",
  "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
  "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
  flags(escape-double-char,strip-whitespace)
  delimiters(" ")
  quote-pairs('""[]')
  );
};

log {
 source(s_apache);
 parser(p_apache);
 destination(d_redis);
};

destination d_redis {
 redis( command("HINCRBY" "${MONTH_ABBREV} ${DAY} ${HOUR}:${MIN}"  "${APACHE.USER_AGENT}" "1"));
};

If you have experience/opinion and want to share with us as a feedback, please send it to balabit syslog-ng mail list and/or me.

2013. július 26., péntek

Redis destination

After I've learned some tricks in the last weeks in C by implementing redis client, I started to develop redis destination for syslog-ng.
The first aim was to develop the basis of the plugin, write makefiles and compile it. For this purpose the afsmtp plugin was a good take-off. I wrote the initialization functions, that called by syslog-ng on startup to load the destination. This way I managed to write a test message in redis.
My mentor, Viktor wrote me a tiny test program in python to check the exist of a predetermined key.
It starts the syslog-ng, that writes a given key in redis. The program checks the exist of this key, shutting down syslog-ng and return with a message, if the writing was successful.

tichy@nb:~/syslog-ng-3.4-install-redis$ sudo ./test.py
Pid of syslog-ng process: 29410
PING: PONG
syslog-ng starting up; version='3.4.2'
syslog-ng shutting down; version='3.4.2'
The test was successful.

Update:  I managed to write messages in redis via syslog-ng the following way: I sent a log message with logger command and I got it in redis.
Of course, I have a lot of things to do yet, choose naming method of keys properly, optimizing, etc.

Update #2: Pub/sub implemented, we can send now the log messages to channels separated by program names.
Error handling fixed, so if redis isn't up on program starting or disconnect while running, syslog-ng collects log messages, try to reconnect and resend all messages after connection restored.

Update #3: Set custom redis command added.

2013. június 29., szombat

Redis client in C

I made a simple redis client in C based on the hiredis example (redis-2.6.14/deps/hiredis/example.c) to demonstrate how to use hiredis. You can compile it with cc -o redisclient redisclient.c libhiredis.a command or run makefile. If you want to specify the host from terminal, run ./redisclient 127.0.0.1:6379. If you don't define, it will use the adress and port in this example. Type exit to quit. Supported commands: get, set, del, incr, decr.

PING: PONG
redis: get key
> value
redis: incr key
> ERR value is not an integer or out of range
redis: set key value 2
> SET: OK
redis: get key
> value 2
redis: set age 21
> SET: OK
redis: decr age
> 20
redis: incr age
> 21
redis: del age
> (null)

2013. június 25., kedd

Compiling syslog-ng OSE 3.4/3.5 on ubuntu

Download the latest version of syslog-ng OSE from the BalaBit website or from GitHub.

You will need build-essential, dpkg-dev, gcc, glib-dev, flex and bison, libnet, libffi packages.
You can install them with apt-get install command or with synaptic.
Hint: You should increase the terminal buffer up to 1000 lines.

Then download the latest version of syslog-ng (if you want to download from github through terminal, git must be installed on your system; you can install it with: )
sudo apt-get install git
then
git clone https://github.com/balabit/syslog-ng-3.4.git

However, in some descriptions you can read: "Download and install the latest version of eventlog", it's enough to install libevtlog-dev and libevtlog0.

Then cd to your syslog-ng-3.4 directory and run
dpkg-checkbuilddeps
to check build dependencies and conflicts.

I needed to install the following packages on ubuntu 13.04:
libtool, debhelper, libevtlog-dev, libevtlog0,  libevtlog0-dbg, libnet1-dev, libglib2.0-dev, libdbi0-dev, libssl-dev.

I got an error message during compiling, so you should install xsltproc before next step.

Type to compile syslog-ng with debug option (you can find more options in syslog-ng ose 3.4 admin guide):
./autogen.sh && ./configure --prefix <install dir> --enable-debug && make && make install
where <install dir> is the destination folder.

Note: If you get permission denied messages, you need to change the user ownership.
chown -R <user> syslog-ng-3.4

Sources:
    Syslog-ng OSE 3.4 admin guide

2013. június 12., szerda

Short introduction and first steps

Hi,

I'm tichy and I participate in gsoc 2013 with project name syslog-ng: redis destination.
It's about to develop a new syslog-ng destination plugin, one that can add or change data in redis and have a reliably database with the function of getting real-time counters, statistics or in other cases caching.
I started the zero phase of the project:
  • I read tutorials and documentation about redis, syslog-ng
  • compiled the syslog-ng 3.4 on ubuntu
  • made a github account for code hosting
Some thoughts about me
I provided the hungarian translation of pigeonplanner and I think this project is a very good opportunity to get involved in open source development.
I got programming experience (c based languages) at university level which is a solid base to start the project, learn and develop something new on a higher level and get experience, which would be a good reference after graduating the university.
I decided to write a blog post for each milestone or when I just find a problem, that's worth to write some thoughts about.
I have exams at the university, but I will be free from next week so I can focus all of my power on this project.