[squid-users] Squid logging to UDP logs multiple lines at the same time

[Amos Jeffries]

On 04/05/18 20:44, Matt Pson wrote:
> Hello,
> I have set up several Squid servers running 3.5.27 on Ubuntu 16.04 LTS (behind keepalived and haproxy to create a ha/load balanced setup) and having are some problems with logging. In order to have a single logfile I decided to send the log directly using UDP to a separate server running rsyslogd. 
> 
> Each Squid is configured with:
> 
> access_log     udp://squid-logserver.domain.com:10514
> 
> The rsyslogd server is configured with:
> 
> Module (load="imudp")
> # rsyslog Input Modules
> input(type="imudp" port="10514" ruleset="Squid")
> # Squid RulesSet
> # make all squid input from the frontends end up here
> ruleset(name="Squid") {
>         action(type="omfile"
>                 file="/var/log/squid/access.log")
>         }
> 
> This works quite well and logs are sent to the server which receives them and puts them in the log file as expected. But the logs are not sent line by line but instead multiple lines at the same time (the size is approx. a UDP packet of lines, ~1300 bytes) which makes the log file quite messed up.

1400 bytes or your network UDP MSS - whichever is smaller. Squid tries
to fit liens in there until one does not fit then sends off the batch.

It's a bug if the buffered_logs directive and access_log buffer-size=
option do not work to control what Squid is *sending*, but UDP can also
be aggregated by the network stacks I/O buffers of both sender and
receiver machines - so make sure you check for that.

<http://www.squid-cache.org/Doc/config/buffered_logs/>
"
Note that even when buffered_logs are off, Squid may have to buffer
records if it cannot write/send them immediately due to pending I/Os
(e.g., the I/O writing the previous log record) or connectivity loss.
"

> 
> What am I missing here? 
> I want my log to be readable ;)

FWIW; The UDP module is intended for a logger which is capable of
handling the log as a raw data stream. syslog systems are not very good
at that.

You will probably find sending these to the Squid machines syslog and
having that relay them out to the remote server in the syslog format
works better than sending a raw UDP stream.

Or maybe using the TCP module and avoiding the complexity of syslog
entirely.

Amos