[squid-users] squid 4.1 default queue-size should consider concurrency

[Amish]

Umm, may be I mis-interpreted queue-size.

I thought queue-size indicates messages "waiting" in the queue and not 
those are currently being processed.

So in case of:
    url_rewrite_children 16 concurrency=4

When redirect process is busy --- its currently processing 64 urls.

So max(2*4,2*16)=32 means that 32 URLs are "waiting" in queue. (in 
addition to 64 being processed)

But if queue-size means ALL urls (64+32) then my formula would be

Default queue-size = (NCHILD*NCONC)+max(2*NCHILD, 2*NCONC)

Regards,

Amish.


On Tuesday 03 July 2018 09:17 PM, Marcus Kool wrote:
> If an admin finds it necessary to configure
>    url_rewrite_children 16 concurrency=4
> the helper subsystem is theoretically capable of processing 64 
> messages simultaneously.
> It does not makes sens to use max(2*4,2*16)=32 for queue-size, but 
> should be _at least_ 64.
>
> Since Squid (before introducing the concurrency parameter) wanted to 
> have a queue that is twice the capacity of the helper subsystem (most 
> likely since this was found to be a good value), the logical thing to 
> do is what I proposed and use 2*NCONC*NCHILD, which in the above 
> example is 2*4*16 = 128.
>
> There are many messages on this mailing list about problems with helpers.
> One refers to
>    external_acl_type session concurrency=100 ttl=3 negative_ttl=0 
> children-max=1 %LOGIN /usr/lib64/squid/ext_session_acl -a -T 60 -b 
> /var/lib/squid/sessions/
> with NCONC=100, NCHILD=1, it is not hard to imagine that a default 
> queue-size of 2 is problematic.
> Another one has
>    store_id_children 100 startup=0 idle=1 concurrency=1000
> with NCONC=1000 and NCHILD=100 the admin wants to process max 1 
> million requests simultaneously.
> *I have a suspicion that admins use these large numbers in an attempt 
> to have a good performance but they never reach it since the 
> queue-size stays small.*
> So, the impact of queue-size should be very clearly documented, better 
> is to use a sane default.
>
> Marcus
>
>
> On 03/07/18 12:07, Amish wrote:
>> 2*NCONC*NCHILD will possibly lead to too high value as a default and 
>> the busy-ness will never be logged.
>>
>> My proposal of higher of (2*NCONC) and (2*NCHILD) would mean that 
>> load is now regularly high enough that atleast 2 more children are 
>> needed.
>>
>> We can start with that and then find a better formula.
>>
>> Amish
>>
>>
>> On Tuesday 03 July 2018 07:49 PM, Marcus Kool wrote:
>>> The original intention of this default value is have a queue that is 
>>> twice the size of the messages being processed, so for helpers with 
>>> concurrency=NCONC and num_children=NCHILD it makes a lot of sense to 
>>> set the default queue length to 2*NCONC*NCHILD.
>>> I do not understand that "compatibility" with doing the wrong 
>>> calculation is a good thing.
>>>
>>> Marcus
>>>
>>>
>>> On 03/07/18 05:16, Amos Jeffries wrote:
>>>> On 03/07/18 20:00, Amish wrote:
>>>>> Hello,
>>>>>
>>>>> In squid 4.1 new option "queue-size" was introduced.
>>>>>
>>>>> In most (or all) cases default "queue-size" is set to children-max*2.
>>>>>
>>>>> But I believe it should be higher of (children-max*2) OR 
>>>>> (concurrency*2)
>>>>>
>>>>> Or it can be some better formula but the point I am trying to make is
>>>>> that, "concurrency" should be taken in to account for calculating
>>>>> default value of "queue-size".
>>>>>
>>>>> Please consider.
>>>>
>>>> FYI; When we add a directive or option to control some behaviour that
>>>> already happens the default is usually set to the value all existing
>>>> Squid are using so nobody gets an unexpected surprise with upgrade.
>>>>
>>>> That can change later once more people have experimented to find 
>>>> what a
>>>> better value actually is.
>>>>
>>>> Cheers
>>>> Amos
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users