[squid-users] Do squid logs performence affect general request performece?

[roee klinger]

Thank you for the advice, Alex.

I just wanted to update anyone following or who might follow that the problem ended up being in
my Logstash server, not in Squid, it performed too slow due to a miss configuration, so Squid kept
sending logs but the slow server got over flooded.

After fixing the server I simply set these configuration normally:
access_log tcp://server:port logformat=logs

and everything seems to be working fine by default.

I did however notice that UDP does aggregate logs before sending them, and TCP does not,
even when set explicitly.

If anybody runs into this issue in the future it might be worth to follow Alex advice on how to get
this to work.

Thanks,
Roee


On 13 Sep 2022, 6:30 +0300, Alex Rousskov <rousskov at measurement-factory.com>, wrote:
> On 9/12/22 18:20, roee klinger wrote:
>
> > Thank you for your advice, as suggested I tested a both TCP and UDP, and
> > found TCP to be so slow
> > that is it practically unusable, UDP however works fine, but has the
> > normal disadvantages of UDP.
>
> Glad that test yielded useful results in your environment!
>
>
> > It seems that TCP sends each message by itself, and UDP sends them in
> > bulk (10-20 at a time),
> > which seems to be a big part of the performance difference, in fact TCP
> > is so slow that it causes Squid to crash after a while.
> >
> > After carefully reading the logs, I tried to get TCP to send in bulk as
> > well by configuring:
> >
> > logformat logstash %ts.%03tu %6tr %>a %<la %>lp %Ss/%03>Hs %<st %rm
> > %ru %[un %Sh/%<a %mt %{Client-Tags}>h
> > access_log tcp://127.0.0.1:5400 logformat=logstash buffer-size=64KB
> > buffered_logs on
> >
> > But it seems to still be sending the logs line by line, am I missing
> > something?
>
>
> I have not tested this, but if my interpretation of the underlying code
> is correct, then the TCP logging module code does not flush the buffer
> when it accumulates (close to) buffer-size bytes. Instead, the module
> starts writing its buffer after accumulating more than 16KB (2*MAX_URL).
> How long are your typical access.log records?
>
> Each write(2) call of a buffering TCP logging module during normal Squid
> operation should be 16KB, regardless of the buffer-size setting.
>
> FWIW, the official access_log buffer-size documentation hints at
> module-specific flushing algorithms. I cannot describe the exact
> algorithm used by the TCP logging module in a few words, but it is not
> as simple as "accumulate buffer-size bytes and then write the
> accumulated bytes".
>
>
> HTH,
>
> Alex.
>
>
> > On 9 Sep 2022, 19:57 +0300, Alex Rousskov wrote:
> > > On 9/9/22 08:49, roee klinger wrote:
> > > > Hello,
> > > >
> > > > I have just recently started exploring ingesting logs from Squid via
> > > > Logstash (TCP log ingestion)
> > > >
> > > > I now have to make the decision of how to deploy Logstash, it seems to
> > > > me that I have two options:
> > > >
> > > > 1. Deploy a Logstash instance for every region where I have a Squid
> > > > instance running, resulting in the lowest latency possible between
> > > > Squid and Logstash.
> > > > 2. Deploy a central Logstash instance for all Squid instance worldwide,
> > > > which is the cheapest and easiest option, but will have high latency
> > > > for some of the Squid instance which are located far away
> > > > geographically from the Logstash instance.
> > > >
> > > > I already know that if the log server crashes in Squid, then Squid will
> > > > crash with a fatal error,
> > >
> > > FYI: Logging error handling is configurable via "access_log on-error".
> > >
> > >
> > > > so Squid takes logging very seriously, so my
> > > > question is: does TCP logging performance in Squid effect the total
> > > > request performance, or are they independent from each other?
> > >
> > > TCP logger uses asynchronous non-blocking I/O. Individual HTTP
> > > transactions do not wait for individual log records to be sent. Needless
> > > to say, logging does consume resources and, hence, may affect overall
> > > HTTP transaction performance. And if the log record recipient is too
> > > slow, then Squid will die and/or log records will be dropped.
> > >
> > >
> > > > What would be you recommendations, if request performance is crucial
> > > > (and log performance is not)?
> > >
> > > I would not be surprised if you would not be able to measure meaningful
> > > performance difference among these modules. From general performance
> > > point of view, and module-specific bugs and limitations notwithstanding,
> > > I would expect "daemon" and "udp" modules to have smaller performance
> > > overhead/cost for Squid workers than the "tcp" module because I expect
> > > that TCP has to "do more" than stdout I/O and UDP. However, I would not
> > > base this decision on such high-level speculations and test various
> > > options instead.
> > >
> > >
> > > HTH,
> > >
> > > Alex.
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> > > http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220918/ca9a3fc9/attachment.htm>