[squid-users] Originserver load balancing and health checks in Squid reverse proxy mode

Alex Rousskov rousskov at measurement-factory.com
Tue Feb 9 20:59:00 UTC 2021 

On 2/9/21 11:35 AM, Chris wrote:
> This is what I'm seeing in peer_select in cache_log with 44,3 debug
> options:

Add (at least) "15,3" to your debug_options and then look for
getWeightedRoundRobinParent lines. Looking at mgr:server_list Cache
Manager page may also be useful.


> Does the weighted-round-robin need some time to use those rtt values? 

I am not 100% sure, but I think the answer to that question is "no".

If you want to see the details of that peer selection algorithm, look
for the Squid function with that name. I bet it has _some_ undocumented
surprises, but I do not know whether they are relevant to your specific
use case.


HTH,

Alex.


> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(258)
> peerSelectDnsPaths: Find IP destination for: '[the_request]' via
> [ip_cache_peer_srv1]
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(280)
> peerSelectDnsPaths: Found sources for '[the_request]'
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(281)
> peerSelectDnsPaths:   always_direct = DENIED
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(282)
> peerSelectDnsPaths:    never_direct = DENIED
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv1]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv2]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv3]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv1]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(295)
> peerSelectDnsPaths:        timedout = 0
> 2021/02/09 16:25:11.588 kid1| 44,3| peer_select.cc(79) ~ps_state:
> [the_request]
> 
> and than in access.log I have:
> 
> TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv1]
> 
> TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv2]
> 
> TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv3]
> 
> evenly distributed.
> 
> So it's not using the weighted-round-robin that should have srv1 at
> 11ms, while srv2 and srv3 are at about 150ms in regard to pinger.
> 
> What did I miss in configuring weighted-round-robin?
> 
> Best Regards,
> 
> Chris
> 
> 
> 
> 
> 
> 
> 
> On 09.02.21 17:09, Chris wrote:
>> Hi Elizer, this helped, it seems as if I got the pinger working.
>>
>> It's now owned by root in the same group as the squid user and the
>> setuid set.
>>
>> So I used chown root:squidusergroup and chmod u+s on the pinger (and
>> in ubuntu it is actually found under /usr/lib/squid/pinger ).
>>
>> Now with debug 42,3 I get some values as:
>>
>> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2]
>>                                 0 Echo Reply      155ms 7 hops
>>
>> and
>>
>> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1]
>>                                 0 Echo Reply      11ms 9 hops
>>
>> but squid is still allocating the requests evenly and not using those
>> ping times in weighted-round-robin.
>>
>> Does the weighted-round-robin need some time to use those rtt values?
>>
>> Best Regards,
>>
>> Chris
>>
>>
>> On 09.02.21 16:19, NgTech LTD wrote:
>>> Maybe its apparmor.
>>> pinger needs to have a setuid permission as root.
>>> its a pinger and needs root privleges as far as i remember.
>>>
>>> Eliezer
>>>
>>>
>>> On Tue, Feb 9, 2021, 17:03 Chris <ml+squidusers at kisswebdev.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> thank you Amos, this is bringing me into the right direction.
>>>>
>>>> Now I know what I'll have to debug: the pinger.
>>>>
>>>> Cache.log shows:
>>>>
>>>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>>>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>>>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>>>> 2021/02/09 14:49:27| Pinger exiting.
>>>>
>>>> and that last line "pinger exiting" looks like a problem here.
>>>>
>>>> Squid is used as a package from ubuntu bionic, it's configured with
>>>> "--enable-icmp" as stated by squid -v.
>>>>
>>>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>>>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as well
>>>> as icmp_query on) and reconfigured but the cache.log still shows:
>>>>
>>>> "Pinger exiting"
>>>>
>>>> So I don't understand why the pinger is exiting. The pinger_program is
>>>> owned by root and has 0755 execution rights. Normal ping commands do
>>>> work and show the one originserver at ttl=53 and time=50 while the
>>>> other
>>>> is at ttl=56 and time=155 - so a RTT comparison for
>>>> weighted-round-robin
>>>> should work here.
>>>>
>>>> Any hints on how I can find out why the pinger is exiting? Right now
>>>> I'm
>>>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>>>> the pinger exits.
>>>>
>>>> The Originservers are defined by (with icp/htcp disabled):
>>>>
>>>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>>>> no-netdb-exchange weighted-round-robin originserver name=srv1
>>>> forceddomain=[domainname]
>>>>
>>>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>>>> no-netdb-exchange weighted-round-robin originserver name=srv2
>>>> forceddomain=[domainname]
>>>>
>>>>
>>>> Thank you for your help,
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 09.02.21 04:23, Amos Jeffries wrote:
>>>>> On 9/02/21 3:40 am, Chris wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>>>> in reverse proxy mode in regard to originserver health checks and
>>>>>> load balancing.
>>>>>>
>>>>>> So far I had been using the round-robin originserver cache peer
>>>>>> selection algorithm while using weight to favor originservers with
>>>>>> closer proximity/lower latency.
>>>>>>
>>>>> Ok.
>>>>>
>>>>>
>>>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>>>> choose the second originserver. It does look as if (e.g. if one
>>>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>>>> server several times before accessing the other one.
>>>>>>
>>>>> The DEAD check by default requires 10 failures in a row to trigger.
>>>>> This is configurable with the connect-fail-limit=N option.
>>>>>
>>>>>
>>>>>> Now instead of using round-robin plus weight it would be best to use
>>>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>>>> originserver if (as it's normally the case) the originserver won't
>>>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>>>> background-ping work?
>>>>> Well, kind of.
>>>>>
>>>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>>>> but some do. Especially if the server is not a true origin but a
>>>>> reverse-proxy.
>>>>>
>>>>>
>>>>>> I tried weighted-round-robin and background-ping on originservers but
>>>>>> got only an evenly distributed request handling even if ones
>>>>>> originservers rtt would be less than half of the others. But then
>>>>>> again, those originservers won't handle icp requests.
>>>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>>>> Echo traffic is working on all possible network routes between your
>>>>> Squid and the peer server(s).
>>>>>
>>>>>
>>>>>> So what's the best solution to a) choose the originserver with the
>>>>>> lowest rtt and b) still have a fast switch if one of the
>>>>>> originservers switches into dead state?
>>>>>
>>>>> Check whether the RTT is actually being measured properly by Squid
>>>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>>>> responding or close enough in the network RTT could come out as a 0
>>>>> value or some N value equal for both peer. ie. neither being "closer".
>>>>>
>>>>>
>>>>> Amos
>>>>> _______________________________________________
>>>>> squid-users mailing list
>>>>> squid-users at lists.squid-cache.org
>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

More information about the squid-users mailing list