[squid-users] squid 4.1 default queue-size should consider concurrency

[Marcus Kool]

If an admin finds it necessary to configure
    url_rewrite_children 16 concurrency=4
the helper subsystem is theoretically capable of processing 64 messages simultaneously.
It does not makes sens to use max(2*4,2*16)=32 for queue-size, but should be _at least_ 64.

Since Squid (before introducing the concurrency parameter) wanted to have a queue that is twice the capacity of the helper subsystem (most likely since this was found to be a good value), the logical 
thing to do is what I proposed and use 2*NCONC*NCHILD, which in the above example is 2*4*16 = 128.

There are many messages on this mailing list about problems with helpers.
One refers to
    external_acl_type session concurrency=100 ttl=3 negative_ttl=0 children-max=1 %LOGIN /usr/lib64/squid/ext_session_acl -a -T 60 -b /var/lib/squid/sessions/
with NCONC=100, NCHILD=1, it is not hard to imagine that a default queue-size of 2 is problematic.
Another one has
    store_id_children 100 startup=0 idle=1 concurrency=1000
with NCONC=1000 and NCHILD=100 the admin wants to process max 1 million requests simultaneously.
*I have a suspicion that admins use these large numbers in an attempt to have a good performance but they never reach it since the queue-size stays small.*
So, the impact of queue-size should be very clearly documented, better is to use a sane default.

Marcus


On 03/07/18 12:07, Amish wrote:
> 2*NCONC*NCHILD will possibly lead to too high value as a default and the busy-ness will never be logged.
> 
> My proposal of higher of (2*NCONC) and (2*NCHILD) would mean that load is now regularly high enough that atleast 2 more children are needed.
> 
> We can start with that and then find a better formula.
> 
> Amish
> 
> 
> On Tuesday 03 July 2018 07:49 PM, Marcus Kool wrote:
>> The original intention of this default value is have a queue that is twice the size of the messages being processed, so for helpers with concurrency=NCONC and num_children=NCHILD it makes a lot of 
>> sense to set the default queue length to 2*NCONC*NCHILD.
>> I do not understand that "compatibility" with doing the wrong calculation is a good thing.
>>
>> Marcus
>>
>>
>> On 03/07/18 05:16, Amos Jeffries wrote:
>>> On 03/07/18 20:00, Amish wrote:
>>>> Hello,
>>>>
>>>> In squid 4.1 new option "queue-size" was introduced.
>>>>
>>>> In most (or all) cases default "queue-size" is set to children-max*2.
>>>>
>>>> But I believe it should be higher of (children-max*2) OR (concurrency*2)
>>>>
>>>> Or it can be some better formula but the point I am trying to make is
>>>> that, "concurrency" should be taken in to account for calculating
>>>> default value of "queue-size".
>>>>
>>>> Please consider.
>>>
>>> FYI; When we add a directive or option to control some behaviour that
>>> already happens the default is usually set to the value all existing
>>> Squid are using so nobody gets an unexpected surprise with upgrade.
>>>
>>> That can change later once more people have experimented to find what a
>>> better value actually is.
>>>
>>> Cheers
>>> Amos
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users