[squid-users] Squid logging to UDP logs multiple lines at the same time

[Matt Pson]

----- Original Message -----
> From: "Amos Jeffries" <squid3 at treenet.co.nz>
> To: "squid-users" <squid-users at lists.squid-cache.org>
> Sent: Friday, 4 May, 2018 11:45:50
> Subject: Re: [squid-users] Squid logging to UDP logs multiple lines at the same time

> On 04/05/18 20:44, Matt Pson wrote:

>> 
>> This works quite well and logs are sent to the server which receives them and
>> puts them in the log file as expected. But the logs are not sent line by line
>> but instead multiple lines at the same time (the size is approx. a UDP packet
>> of lines, ~1300 bytes) which makes the log file quite messed up.
> 
> 1400 bytes or your network UDP MSS - whichever is smaller. Squid tries
> to fit liens in there until one does not fit then sends off the batch.

So in theory it could work if I set the buffer-size low enough, like 16 bytes, so the log line would never fit in the buffer and thus being sent asap?

> It's a bug if the buffered_logs directive and access_log buffer-size=
> option do not work to control what Squid is *sending*, but UDP can also
> be aggregated by the network stacks I/O buffers of both sender and
> receiver machines - so make sure you check for that.
> 
> <http://www.squid-cache.org/Doc/config/buffered_logs/>
> "
> Note that even when buffered_logs are off, Squid may have to buffer
> records if it cannot write/send them immediately due to pending I/Os
> (e.g., the I/O writing the previous log record) or connectivity loss.
> "

Thanks for the clarification. Reading "To send each log line as text data to a UDP receiver." (at http://www.squid-cache.org/Doc/config/access_log/ ) made me think that each line would be sent individually without buffering.

>> 
>> What am I missing here?
>> I want my log to be readable ;)
> 
> FWIW; The UDP module is intended for a logger which is capable of
> handling the log as a raw data stream. syslog systems are not very good
> at that.

I see. It will be very handy when we get around to make our own logging/stats engine take the input from Squid like we do with syslog messages from our firewalls. But that is in the future of another project.

> You will probably find sending these to the Squid machines syslog and
> having that relay them out to the remote server in the syslog format
> works better than sending a raw UDP stream.
> 
> Or maybe using the TCP module and avoiding the complexity of syslog
> entirely.

Yep, I will go via the syslog on the squid machines. I was trying to remove as many dependencies as possible but I guess syslog is one I can live with since it it required by other services.

Thanks for a quick and very helpful answer.

/Matt Pson