[squid-users] Originserver load balancing and health checks in Squid reverse proxy mode

Tue Feb 9 17:41:22 UTC 2021

This is more of Amos and Alex area.
In general I think that haproxy does load balancing much more efficiently then squid.
It is being used in production for years so I'm not sure why you should use Squid for LB.
If you want to resolve this issue then be my guest I can only offer so QA and advice here and there.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd at gmail.com
Zoom: Coming soon

-----Original Message-----
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Chris
Sent: Tuesday, February 9, 2021 6:36 PM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Originserver load balancing and health checks in Squid reverse proxy mode

This is what I'm seeing in peer_select in cache_log with 44,3 debug options:

2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(258) 
peerSelectDnsPaths: Find IP destination for: '[the_request]' via 
[ip_cache_peer_srv1]
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(280) 
peerSelectDnsPaths: Found sources for '[the_request]'
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(281) 
peerSelectDnsPaths:   always_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(282) 
peerSelectDnsPaths:    never_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv2]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv3]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292) 
peerSelectDnsPaths:      cache_peer = local=0.0.0.0 
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(295) 
peerSelectDnsPaths:        timedout = 0
2021/02/09 16:25:11.588 kid1| 44,3| peer_select.cc(79) ~ps_state: 
[the_request]

and than in access.log I have:

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv1]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv2]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv3]

evenly distributed.

So it's not using the weighted-round-robin that should have srv1 at 
11ms, while srv2 and srv3 are at about 150ms in regard to pinger.

What did I miss in configuring weighted-round-robin?

Best Regards,

Chris

On 09.02.21 17:09, Chris wrote:
> Hi Elizer, this helped, it seems as if I got the pinger working.
>
> It's now owned by root in the same group as the squid user and the 
> setuid set.
>
> So I used chown root:squidusergroup and chmod u+s on the pinger (and 
> in ubuntu it is actually found under /usr/lib/squid/pinger ).
>
> Now with debug 42,3 I get some values as:
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2] 
>                                 0 Echo Reply      155ms 7 hops
>
> and
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1] 
>                                 0 Echo Reply      11ms 9 hops
>
> but squid is still allocating the requests evenly and not using those 
> ping times in weighted-round-robin.
>
> Does the weighted-round-robin need some time to use those rtt values?
>
> Best Regards,
>
> Chris
>
>
> On 09.02.21 16:19, NgTech LTD wrote:
>> Maybe its apparmor.
>> pinger needs to have a setuid permission as root.
>> its a pinger and needs root privleges as far as i remember.
>>
>> Eliezer
>>
>>
>> On Tue, Feb 9, 2021, 17:03 Chris <ml+squidusers at kisswebdev.com> wrote:
>>
>>> Hi,
>>>
>>> thank you Amos, this is bringing me into the right direction.
>>>
>>> Now I know what I'll have to debug: the pinger.
>>>
>>> Cache.log shows:
>>>
>>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>>> 2021/02/09 14:49:27| Pinger exiting.
>>>
>>> and that last line "pinger exiting" looks like a problem here.
>>>
>>> Squid is used as a package from ubuntu bionic, it's configured with
>>> "--enable-icmp" as stated by squid -v.
>>>
>>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as well
>>> as icmp_query on) and reconfigured but the cache.log still shows:
>>>
>>> "Pinger exiting"
>>>
>>> So I don't understand why the pinger is exiting. The pinger_program is
>>> owned by root and has 0755 execution rights. Normal ping commands do
>>> work and show the one originserver at ttl=53 and time=50 while the 
>>> other
>>> is at ttl=56 and time=155 - so a RTT comparison for 
>>> weighted-round-robin
>>> should work here.
>>>
>>> Any hints on how I can find out why the pinger is exiting? Right now 
>>> I'm
>>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>>> the pinger exits.
>>>
>>> The Originservers are defined by (with icp/htcp disabled):
>>>
>>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv1
>>> forceddomain=[domainname]
>>>
>>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv2
>>> forceddomain=[domainname]
>>>
>>>
>>> Thank you for your help,
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>> On 09.02.21 04:23, Amos Jeffries wrote:
>>>> On 9/02/21 3:40 am, Chris wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>>> in reverse proxy mode in regard to originserver health checks and
>>>>> load balancing.
>>>>>
>>>>> So far I had been using the round-robin originserver cache peer
>>>>> selection algorithm while using weight to favor originservers with
>>>>> closer proximity/lower latency.
>>>>>
>>>> Ok.
>>>>
>>>>
>>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>>> choose the second originserver. It does look as if (e.g. if one
>>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>>> server several times before accessing the other one.
>>>>>
>>>> The DEAD check by default requires 10 failures in a row to trigger.
>>>> This is configurable with the connect-fail-limit=N option.
>>>>
>>>>
>>>>> Now instead of using round-robin plus weight it would be best to use
>>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>>> originserver if (as it's normally the case) the originserver won't
>>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>>> background-ping work?
>>>> Well, kind of.
>>>>
>>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>>> but some do. Especially if the server is not a true origin but a
>>>> reverse-proxy.
>>>>
>>>>
>>>>> I tried weighted-round-robin and background-ping on originservers but
>>>>> got only an evenly distributed request handling even if ones
>>>>> originservers rtt would be less than half of the others. But then
>>>>> again, those originservers won't handle icp requests.
>>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>>> Echo traffic is working on all possible network routes between your
>>>> Squid and the peer server(s).
>>>>
>>>>
>>>>> So what's the best solution to a) choose the originserver with the
>>>>> lowest rtt and b) still have a fast switch if one of the
>>>>> originservers switches into dead state?
>>>>
>>>> Check whether the RTT is actually being measured properly by Squid
>>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>>> responding or close enough in the network RTT could come out as a 0
>>>> value or some N value equal for both peer. ie. neither being "closer".
>>>>
>>>>
>>>> Amos
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users