[squid-users] Squid with SMP, CARP and a forwarding loop

Sun Nov 1 21:57:01 UTC 2015

Thank you Amos, I was pretty brain drained by the time I posted so please 
excuse my pasting slip up.

I tried adding  no-netdb-exchange and no-digest to the cache_peer lines, 
the first one eliminated the forward loop warnings but I am still 
experiencing the first request coming from the cache and then subsequent 
requests going direct. Also I did disable the tproxy port.

I now suspect some sort of logic bug in the code as is shows in the cache 
logs carp.cc is not called a second time before peer_select.cc on the 
second attempt. Unfortunately my programming skills are poor and have 
limited time to look at this issue.

For now to work-around this behaviour I will use the never_direct 
directive, but if you would like to investigate further I have provided 
level 2 and 3 debug cache logs that you could look at.
https://droplet-wlg.solnetsolutions.co.nz/public.php?service=files&t=8a3b73eff46a9cf1a91829c0b9d0016a

Cheers

Mike Hodgkinson
Internal Support Engineer

Mobile  +64 21 754 339
Phone  +64 4 462 5064
Email   mike.hodgkinson at solnet.co.nz

Solnet Solutions Limited
Level 12, Solnet House
70 The Terrace, Wellington 6011
PO Box 397, Wellington 6140

www.solnet.co.nz  

From:   Amos Jeffries <squid3 at treenet.co.nz>
To: 
Date:   30/10/2015 11:03 p.m.
Subject:        Re: [squid-users] Squid with SMP, CARP and a forwarding 
loop
Sent by:        "squid-users" <squid-users-bounces at lists.squid-cache.org>

On 30/10/2015 1:45 p.m., Mike.Hodgkinson wrote:
> I have been attempting to setup a squid forward proxy with one frontend 
> and two backends as per configuration example 
> http://wiki.squid-cache.org/ConfigExamples/SmpCarpCluster
> 
> My issue is that only the first attempt comes from the cache and then 
> additional requests are downloaded direct by the frontend instead of 
from 
> the backend caches. I suspect it is due to a detected forwarding loop 
> which shows up in the logs:
> 
> 2015/10/30 13:07:49.239 kid1| 44,3| peer_select.cc(137) peerSelect: 
> e:=XIWV/0x7f7bfee2e730*2 http://127.0.0.1:40
> 02/squid-internal-dynamic/netdb
> 2015/10/30 13:07:49.239 kid1| 20,3| store.cc(466) lock: peerSelect 
locked 
> key 64AAA11C8DEF57153B10BA2C9D2F3D60 e:=XIWV/0x7f7bfee2e730*3
> 2015/10/30 13:07:49.240 kid1| 44,3| peer_select.cc(441) peerSelectFoo: 
GET 
> 127.0.0.1
> 2015/10/30 13:07:49.240 kid1| 44,3| peer_select.cc(468) peerSelectFoo: 
> peerSelectFoo: direct = DIRECT_YES (forwarding loop detected)
> 2015/10/30 13:07:49.240 kid1| 44,3| peer_select.cc(477) peerSelectFoo: 
> peerSelectFoo: direct = DIRECT_YES
> 2015/10/30 13:07:49.240 kid1| 44,2| peer_select.cc(258) 
> peerSelectDnsPaths: Find IP destination for: 
> http://127.0.0.1:4002/squid-internal-dynamic/netdb' via 127.0.0.1
> 
> I can force the backend caches to be used successfully with this option 
> "never_direct allow all" however I would like to resolve the underlying 
> issue.

The above is an internal Squid request from the fontend to its backends.
This one specifically is going direct due to a hack in the code.

You can avoid it by adding "no-netdb-exchange" to the cache_peer lines.
I'm not sure if that will affect the CARP selection since these requests
are one of the types feeding into the peer up/down/overloaded
monitoring. With "no-query" also in use you will be left with the client
HTTP traffic being the only source of that data which carp depends on.

Or using that "never_direct allow all" will override that code hack.

> 
> I have no iptables configured on this server and have made sure the 
> environment variable http_proxy is not set. Also I have tested this on 
> Squid 3.4.8 and 3.5.10 on Debian.

Since you have no iptables rules configured the traffic arriving in port
3129 will be completely borked.

Either, remove that port 3129 line from the frontend config and use port
3128 for testing until you are ready to setup TPROXY properly;

Or, setup the TPROXY iptables and routing rules and test the proxy
exactly as the clients would be using it.

> 
> My config is below:
> #/etc/squid/squid.conf#
> debug_options = ALL,3
> cachemgr_passwd ******

NOTE: if that was your actual password you now need to change it.

> acl localnet src 10.1.0.0/16
> acl localnet src 10.2.0.0/16
> acl localnet src 192.168.0.0/23
> acl localnet src fe80::/10
> acl squid_servers src 10.1.209.0/24

 See below...

<snip>
> #/etc/squid/squid-frontend.conf#
> http_port 3128
> http_port 3129 tproxy
> http_access allow localhost manager
> http_access deny manager
> http_access allow localhost
> http_access allow localnet
> http_access allow squid_servers

With squid_servers IP range being entirely within "localnet" this
"http_access allow squid_servers" line is not doing anything.
You can simplify by removing it.

> htcp_access allow squid_servers
> htcp_access deny all
> cache_peer 127.0.0.1 parent 4002 0 carp login=PASS name=backend-kid2 
> no-query
> cache_peer 127.0.0.1 parent 4003 0 carp login=PASS name=backend-kid3 
> no-query
> prefer_direct off
> nonhierarchical_direct off
> memory_replacement_policy heap LRU
> cache_mem 2048 MB
> access_log /var/log/squid3/frontend.access.log
> cache_log /var/log/squid3/frontend.cache.log
> visible_hostname frontend.cloud.solnet.nz
> 
> #/etc/squid/squid-backend.conf#
> http_port 127.0.01:400${process_number}
> http_access allow localhost
> cache_mem 5 MB
> cache_replacement_policy heap LFUDA
> maximum_object_size 1 GB
> cache_dir rock /cache/rock 20480 max-size=32768
> cache_dir aufs /cache/${process_number} 20480 128 128 min-size=32769
> visible_hostname backend${process_number}.cloud.solnet.nz
> access_log /var/log/squid3/backend${process_number}.access.log
> cache_log /var/log/squid3/backend${process_number}.cache.log
> 
> I did have visible_hostname set to backend.cloud.solnet.nz but that did 
> not help either.

Nod. All that did previously was prevent forwarding loops between the
backends in case something unusually nasty happened in the
routing/iptabels layers. What you have now is okay and slightly better
for diagnosing the issues.

Amos
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Attention:
This email may contain information intended for the sole use of
the original recipient. Please respect this when sharing or
disclosing this email's contents with any third party. If you
believe you have received this email in error, please delete it
and notify the sender or postmaster at solnetsolutions.co.nz as
soon as possible. The content of this email does not necessarily
reflect the views of Solnet Solutions Ltd.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20151102/3a138562/attachment.html>