[squid-users] [Troubleshoot] Squid 3.3 - Lots of 403 erros when reducing the workers number

Mon Sep 12 11:29:22 UTC 2022

He Xavier,

I believe that there are couple logformat options that you should add to understand better this issue.
The first thing in this scenario is to have the squid.conf and access.log output.

If you have a load of about 100 rps you shouldn't be required to have more then 1-2 workers tops.
To know the actual load you can dump the cache manager details.
With the next script:
https://gist.github.com/elico/8790bdc835d8e9ecbc57e72fc31effc0

it would be possible to probably dump all the relevant sections and much more.
Just be advised to review them and the squid.conf and the access.log before send a link to a compressed archive
on the public list.

If you prefer you can send the link to there in a private email so I can try to review them.
Depends on the proxy network access it would be pretty simple to understand where this 403 is coming from.
It's also possible that it's from the origin server and not from the proxy.

I assume this 403 is not based on the authentication level but it might be possible to use a special deny_info to
"mark" the culprit deny acl in your squid.conf.
Ie we can set a special response code in the ranges of 40x or 50x to make sure if it's from the proxy side.

If I remember you are using basic authentication so it's pretty simple to write a threaded helper that will 
be responsible on all requests but I still need to see the squid.conf and access.log

Eliezer

----
Eliezer Croitoru
NgTech, Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd at gmail.com
Web: https://ngtech.co.il/
My-Tube: https://tube.ngtech.co.il/

-----Original Message-----
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Xavier Lecluse
Sent: Friday, 9 September 2022 11:21
To: Alex Rousskov <rousskov at measurement-factory.com>
Cc: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] [Troubleshoot] Squid 3.3 - Lots of 403 erros when reducing the workers number

Hello Alex,

Thanks for your answer.

>> We are using two squid proxies (Squid 3.3)

>Squid v3 is not officially supported. My answers below may apply to 
>Squid v3, but they are based on Squid v5+.

>> In order to address some issues with Java clients, we tried to lower
>> the worker directive from 8 to 1, because of the relative low number
>> of simultaneous connections on our SSquid servers (about 100rq/s) 
>> After reducing the worker value to 1, and restarting the proxies, we
>> observed a great number of 403 errors, so we decided to rollback to 8
>> workers.

>> - How the number of workers and these 403 errors can be correlated ?

>I do not know the exact correlation vector in your environment, but 
>fewer workers means, among other things, smaller _aggregate_ 
>authentication cache size and higher load on individual authentication 
>helpers. To pinpoint the correlation, we would need to know _why_ Squid 
>is generating 403 (Forbidden) errors.

Is there a specific way to find the reason ? Do I need to activate a certain logging level to see these traces ?
I will check in the actual logs and I'll come back if I don't find anything

>> - Is there any "recommandations" about the number of workers to use,
>> for a given number of request/s ?

>Workers are primarily a performance optimization. For related tuning 
>suggestions, please see 
>https://wiki.squid-cache.org/Features/SmpScale#How_to_configure_SMP_Squid_for_top_performance.3F

Yes, I saw and read this thread on the Wiki.
We were already using some of these recommandations.
I think we are quite far from a "top performance" but we didn't observed any performance issue when we used 8 workers.
I understand that an individual worker should be able to handle our actual load, but we have to find what is provoking the 403 errors first.

>> The inital problem is from some java clients, which are using two TCP
>> sessions, one for the authentication, and another one for the HTTP(s)
>> requests. The fact is that the "second" session is not always opened
>> on the same worker, so ot considers that the authentication step has
>> not already been done.

>> Is there a way to address this issue ?

>If (a request on) the second connection has enough information to link 
>it to the first/authenticated request/connection, then it may be 
>possible to configure Squid and write authentication helpers in such a 
>way that the "other" worker knows that the client of the second 
>connection has already authenticated. The details would depend on the 
>authentication scheme and that "linking" mechanism.

Do you have any example of an authentication helper I can start with ? or an example on how to "link" events between workers ?
I was thinking that workers were quite "individuals" and did not exchange anything (or verry little) with each other.
Any start point would be welcome.

Regards,
Xavier

>HTH,

>Alex.

----- Mail original -----
De: "Alex Rousskov" <rousskov at measurement-factory.com>
À: squid-users at lists.squid-cache.org
Envoyé: Jeudi 8 Septembre 2022 17:19:19
Objet: Re: [squid-users] [Troubleshoot] Squid 3.3 - Lots of 403 erros when reducing the workers number

On 9/8/22 10:15, Xavier Lecluse wrote:

> We are using two squid proxies (Squid 3.3)

Squid v3 is not officially supported. My answers below may apply to 
Squid v3, but they are based on Squid v5+.

> In order to address some issues with Java clients, we tried to lower
> the worker directive from 8 to 1, because of the relative low number
> of simultaneous connections on our SSquid servers (about 100rq/s) 
> After reducing the worker value to 1, and restarting the proxies, we
> observed a great number of 403 errors, so we decided to rollback to 8
> workers.

> - How the number of workers and these 403 errors can be correlated ?

I do not know the exact correlation vector in your environment, but 
fewer workers means, among other things, smaller _aggregate_ 
authentication cache size and higher load on individual authentication 
helpers. To pinpoint the correlation, we would need to know _why_ Squid 
is generating 403 (Forbidden) errors.

> - Is there any "recommandations" about the number of workers to use,
> for a given number of request/s ?

Workers are primarily a performance optimization. For related tuning 
suggestions, please see 
https://wiki.squid-cache.org/Features/SmpScale#How_to_configure_SMP_Squid_for_top_performance.3F

> The inital problem is from some java clients, which are using two TCP
> sessions, one for the authentication, and another one for the HTTP(s)
> requests. The fact is that the "second" session is not always opened
> on the same worker, so ot considers that the authentication step has
> not already been done.

> Is there a way to address this issue ?

If (a request on) the second connection has enough information to link 
it to the first/authenticated request/connection, then it may be 
possible to configure Squid and write authentication helpers in such a 
way that the "other" worker knows that the client of the second 
connection has already authenticated. The details would depend on the 
authentication scheme and that "linking" mechanism.

HTH,

Alex.

_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users