Subject: [BEHAVE] syslog scalability

From: Rainer Gerhards <[email-masked]>
Date: Mon, 1 Apr 2013 16:20:20 +0000

Hi all,

I am the author of rsyslog (the default syslogd on most
Linuxes) and have been involved in the recent syslog RFC series. David
Harrington has asked my to comment on syslog scalability.

With rsyslog, I routinely do around 200,000 messages per second (mps) on
low-end (notebook) hardware. I know that users do much larger traffic
volumes in production systems. One user report can be found in [1],
where 1 million mps were processed (with an older and slower version).
The message sizes affect message rate. My testing was with 60 to 80 byte
messages (the usual size for linux syslog entries). As another example,
I know a customer who does heavy logging of call data records, size
between 0.5 and 1.5k, with around 150,000 mps. They have deliberately
chosen to stick to this number so that they have headroom in case of
traffic spikes. Note that in that case, the output is automatically
zipped, what obviously takes some toll on processing time. They also use
an older and slower version.

These scenarios, as far as I know, were taken via UDP and plain TCP
syslog. If TLS is used, the numbers obviously are reduced due to the
extra encryption requirement. Of course, this is not syslog specific but
applies to all protocols utilizing TLS (and with hardware crypto boxes
the performance hit is of course much lower).

Regarding TLS, I need to mention that it is supported by at least
rsyslog and syslog-ng, which together have 90+ percent "market share" on
Linux and Unix. Adiscon also supports TLS on Windows in our WinSyslog,
EventReporter and MonitorWare Agent products. I am not sure who else has
implemented it, but I am sure there are many more implementations.

Actual deployments currently favor legacy TCP syslog (As described in
RFC6587). There are various reasons for this; one reason customers often
mention is that they run their syslog traffic in an already-secured
backend environment. Plain TCP syslog is very widely deployed and
supported by almost all syslog tools that have at least some market
share.

I hope this comment is useful. Feel free to ask if information is
missing. I did *not* read many mails on this mailing list here. So I may
have missed some important points. I am currently subscribed to the list
and will follow up.

Best regards,
Rainer Gerhards

[1]http://kb.monitorware.com/000-000-character-messages-sec-with-rsyslog-t10740.html