[squid-users] Problem with access.log and when using SMP

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 27/11/2014 1:25 a.m., Giorgi Tepnadze wrote:
> Hello,
> 
> I have own squid log parsing software which worked flawlessly
> before we moved to squid 3.3.8 (old was 2.7). Log parser stopped
> working. This is the problem I found.
> 
> Problem lines: 1416971036.802 184393 10.xx.xx.xx TCP_MISS/200 10963
> CONNECT pixel.facebook.com:1416971036.390      0 10.yy.yy.yy
> TCP_DENIED/407 3751 CONNECT clients4.google.com:443 - HIER_NONE/-
> text/html 1416971036.645   7164 10.zz.zz.zz TCP_MISS/200 55438 GET 
> http://s40.radikal.ru/i089/1107/9b/2fbf1779ead9.jpg
> user at domain.com HIER_DIRECT/81.176.238.141 image/jpeg 443
> user2 at domainc.com HIER_DIRECT/31.13.93.3 -
> 
> As it should be: 1416971036.390      0 10.yy.yy.yy TCP_DENIED/407
> 3751 CONNECT clients4.google.com:443 - HIER_NONE/- text/html 
> 1416971036.802 184393 10.xx.xx.xx TCP_MISS/200 10963 CONNECT 
> pixel.facebook.com:443 user2 at domainc.com HIER_DIRECT/31.13.93.3 - 
> 1416971036.645   7164 10.zz.zz.zz TCP_MISS/200 55438 GET 
> http://s40.radikal.ru/i089/1107/9b/2fbf1779ead9.jpg
> user at domain.com HIER_DIRECT/81.176.238.141 image/jpeg
> 
> I think that's because of SMP as I have 4 squid kids running and
> they all write to same access.log file and there is concurrency
> problem. Sometimes one is writing log line when previous log line
> is not finished yet or something like this. Such errors are rare
> but enough to stop my parser with  error.
> 
> Is it a bug? Is there any workaround or solution?

What logging module are you using to write?

Squid log lines are generated into a buffer and emitted in what should
be close to atomic (but not exactly locking) per-line write() actions.

You could write a custom daemon to receive the log lines and send them
directly through your parser system as units. This has the added
benefit of being a lot closer to real-time log processing.

Amos

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUercvAAoJELJo5wb/XPRjZVEH/RtrMBTH5XfPi6MmIVf2/Yzb
DZg+ObG64J/SmWMc99YvTL8Bg5SrB4GNKZEt5JM3jKYOkjS1lg7G90JUfwmHi8xI
xnX/+iayr19QVmdRb5LNZdpgn1spCoodq9RXQKVLKQnxvYVrV5SCnPx51xeevG2g
X7gBa+l/nltEQy1cc2Dc5UScvMHdXh8tShG8CINX/oyXr6XM0qa8jCe3Tczq966K
GsEnicp017lwSDVSiXPU866Gf/3xexXGUzPIxlWe3OIbEwp/LP0U63jaF2lUz6cu
xY10bAUbOTGE76UmhHRSmePlhZcWwsHHwbXy7Z5OQCe2UX/MRIZYzN7GkqSVgf4=
=iKx3
-----END PGP SIGNATURE-----