[squid-users] [Troubleshoot] Squid 3.3 - Lots of 403 erros when reducing the workers number

[Alex Rousskov]

On 9/9/22 04:20, Xavier Lecluse wrote:
>>> In order to address some issues with Java clients, we tried to lower
>>> the worker directive from 8 to 1, because of the relative low number
>>> of simultaneous connections on our SSquid servers (about 100rq/s)
>>> After reducing the worker value to 1, and restarting the proxies, we
>>> observed a great number of 403 errors, so we decided to rollback to 8
>>> workers.

>>> - How the number of workers and these 403 errors can be correlated ?

>> I do not know the exact correlation vector in your environment, but
>> fewer workers means, among other things, smaller _aggregate_
>> authentication cache size and higher load on individual authentication
>> helpers. To pinpoint the correlation, we would need to know _why_ Squid
>> is generating 403 (Forbidden) errors.

> Is there a specific way to find the reason ? Do I need to activate a certain logging level to see these traces ?
> I will check in the actual logs and I'll come back if I don't find anything

There are many ways to investigate this. If you cannot figure it out, I 
hope that others on this mailing list can guide you.



>>> The inital problem is from some java clients, which are using two TCP
>>> sessions, one for the authentication, and another one for the HTTP(s)
>>> requests. The fact is that the "second" session is not always opened
>>> on the same worker, so ot considers that the authentication step has
>>> not already been done.

>>> Is there a way to address this issue ?

>> If (a request on) the second connection has enough information to link
>> it to the first/authenticated request/connection, then it may be
>> possible to configure Squid and write authentication helpers in such a
>> way that the "other" worker knows that the client of the second
>> connection has already authenticated. The details would depend on the
>> authentication scheme and that "linking" mechanism.

> Do you have any example of an authentication helper I can start with
> ? or an example on how to "link" events between workers ?

Sorry, I do not.


> I was thinking that workers were quite "individuals" and did not
> exchange anything (or verry little) with each other.

In this case, the exchange of information would be performed by your 
authentication helpers, not workers. A helper is a custom program that 
can do nearly anything as long as it obeys the helper API[1]. For 
example, helpers from different workers can share a custom 
authentication database. Squid workers will just read annotations sent 
to Squid by your helpers (e.g., already_authenticated_=yes). Your 
configuration can act on those annotations via the note ACL.

[1] https://wiki.squid-cache.org/Features/AddonHelpers


HTH,

Alex.


> ----- Mail original -----
> De: "Alex Rousskov" <rousskov at measurement-factory.com>
> À: squid-users at lists.squid-cache.org
> Envoyé: Jeudi 8 Septembre 2022 17:19:19
> Objet: Re: [squid-users] [Troubleshoot] Squid 3.3 - Lots of 403 erros when reducing the workers number
> 
> On 9/8/22 10:15, Xavier Lecluse wrote:
> 
>> We are using two squid proxies (Squid 3.3)
> 
> Squid v3 is not officially supported. My answers below may apply to
> Squid v3, but they are based on Squid v5+.
> 
> 
>> In order to address some issues with Java clients, we tried to lower
>> the worker directive from 8 to 1, because of the relative low number
>> of simultaneous connections on our SSquid servers (about 100rq/s)
>> After reducing the worker value to 1, and restarting the proxies, we
>> observed a great number of 403 errors, so we decided to rollback to 8
>> workers.
> 
>> - How the number of workers and these 403 errors can be correlated ?
> 
> I do not know the exact correlation vector in your environment, but
> fewer workers means, among other things, smaller _aggregate_
> authentication cache size and higher load on individual authentication
> helpers. To pinpoint the correlation, we would need to know _why_ Squid
> is generating 403 (Forbidden) errors.
> 
> 
>> - Is there any "recommandations" about the number of workers to use,
>> for a given number of request/s ?
> 
> Workers are primarily a performance optimization. For related tuning
> suggestions, please see
> https://wiki.squid-cache.org/Features/SmpScale#How_to_configure_SMP_Squid_for_top_performance.3F
> 
> 
>> The inital problem is from some java clients, which are using two TCP
>> sessions, one for the authentication, and another one for the HTTP(s)
>> requests. The fact is that the "second" session is not always opened
>> on the same worker, so ot considers that the authentication step has
>> not already been done.
> 
>> Is there a way to address this issue ?
> 
> If (a request on) the second connection has enough information to link
> it to the first/authenticated request/connection, then it may be
> possible to configure Squid and write authentication helpers in such a
> way that the "other" worker knows that the client of the second
> connection has already authenticated. The details would depend on the
> authentication scheme and that "linking" mechanism.
> 
> 
> HTH,
> 
> Alex.
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users