[squid-users] Originserver load balancing and health checks in Squid reverse proxy mode

Tue Feb 9 16:35:47 UTC 2021

This is what I'm seeing in peer_select in cache_log with 44,3 debug options:

2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(258) 
peerSelectDnsPaths: Find IP destination for: '[the_request]' via 
[ip_cache_peer_srv1]
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(280) 
peerSelectDnsPaths: Found sources for '[the_request]'
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(281) 
peerSelectDnsPaths:   always_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(282) 
peerSelectDnsPaths:    never_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv2]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv3]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(295) 
peerSelectDnsPaths:        timedout = 0
2021/02/09 16:25:11.588 kid1| 44,3| peer_select.cc(79) ~ps_state: 
[the_request]

and than in access.log I have:

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv1]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv2]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv3]

evenly distributed.

So it's not using the weighted-round-robin that should have srv1 at 
11ms, while srv2 and srv3 are at about 150ms in regard to pinger.

What did I miss in configuring weighted-round-robin?

Best Regards,

Chris

On 09.02.21 17:09, Chris wrote:
> Hi Elizer, this helped, it seems as if I got the pinger working.
>
> It's now owned by root in the same group as the squid user and the 
> setuid set.
>
> So I used chown root:squidusergroup and chmod u+s on the pinger (and 
> in ubuntu it is actually found under /usr/lib/squid/pinger ).
>
> Now with debug 42,3 I get some values as:
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2] 
>                                 0 Echo Reply      155ms 7 hops
>
> and
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1] 
>                                 0 Echo Reply      11ms 9 hops
>
> but squid is still allocating the requests evenly and not using those 
> ping times in weighted-round-robin.
>
> Does the weighted-round-robin need some time to use those rtt values?
>
> Best Regards,
>
> Chris
>
>
> On 09.02.21 16:19, NgTech LTD wrote:
>> Maybe its apparmor.
>> pinger needs to have a setuid permission as root.
>> its a pinger and needs root privleges as far as i remember.
>>
>> Eliezer
>>
>>
>> On Tue, Feb 9, 2021, 17:03 Chris <ml+squidusers at kisswebdev.com> wrote:
>>
>>> Hi,
>>>
>>> thank you Amos, this is bringing me into the right direction.
>>>
>>> Now I know what I'll have to debug: the pinger.
>>>
>>> Cache.log shows:
>>>
>>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>>> 2021/02/09 14:49:27| Pinger exiting.
>>>
>>> and that last line "pinger exiting" looks like a problem here.
>>>
>>> Squid is used as a package from ubuntu bionic, it's configured with
>>> "--enable-icmp" as stated by squid -v.
>>>
>>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as well
>>> as icmp_query on) and reconfigured but the cache.log still shows:
>>>
>>> "Pinger exiting"
>>>
>>> So I don't understand why the pinger is exiting. The pinger_program is
>>> owned by root and has 0755 execution rights. Normal ping commands do
>>> work and show the one originserver at ttl=53 and time=50 while the 
>>> other
>>> is at ttl=56 and time=155 - so a RTT comparison for 
>>> weighted-round-robin
>>> should work here.
>>>
>>> Any hints on how I can find out why the pinger is exiting? Right now 
>>> I'm
>>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>>> the pinger exits.
>>>
>>> The Originservers are defined by (with icp/htcp disabled):
>>>
>>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv1
>>> forceddomain=[domainname]
>>>
>>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv2
>>> forceddomain=[domainname]
>>>
>>>
>>> Thank you for your help,
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>> On 09.02.21 04:23, Amos Jeffries wrote:
>>>> On 9/02/21 3:40 am, Chris wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>>> in reverse proxy mode in regard to originserver health checks and
>>>>> load balancing.
>>>>>
>>>>> So far I had been using the round-robin originserver cache peer
>>>>> selection algorithm while using weight to favor originservers with
>>>>> closer proximity/lower latency.
>>>>>
>>>> Ok.
>>>>
>>>>
>>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>>> choose the second originserver. It does look as if (e.g. if one
>>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>>> server several times before accessing the other one.
>>>>>
>>>> The DEAD check by default requires 10 failures in a row to trigger.
>>>> This is configurable with the connect-fail-limit=N option.
>>>>
>>>>
>>>>> Now instead of using round-robin plus weight it would be best to use
>>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>>> originserver if (as it's normally the case) the originserver won't
>>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>>> background-ping work?
>>>> Well, kind of.
>>>>
>>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>>> but some do. Especially if the server is not a true origin but a
>>>> reverse-proxy.
>>>>
>>>>
>>>>> I tried weighted-round-robin and background-ping on originservers but
>>>>> got only an evenly distributed request handling even if ones
>>>>> originservers rtt would be less than half of the others. But then
>>>>> again, those originservers won't handle icp requests.
>>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>>> Echo traffic is working on all possible network routes between your
>>>> Squid and the peer server(s).
>>>>
>>>>
>>>>> So what's the best solution to a) choose the originserver with the
>>>>> lowest rtt and b) still have a fast switch if one of the
>>>>> originservers switches into dead state?
>>>>
>>>> Check whether the RTT is actually being measured properly by Squid
>>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>>> responding or close enough in the network RTT could come out as a 0
>>>> value or some N value equal for both peer. ie. neither being "closer".
>>>>
>>>>
>>>> Amos
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users