[squid-users] squid sibling peers and digest requests

Fri Dec 30 00:15:00 UTC 2016

Here are some debug logs from FwdState which handles digest request.

172.22.13.210 – original squid
172.22.8.145 – sibling squid
127.0.0.1:18070 – parent

As you can see it uses connection to parent for this request (reusing pconn
local=127.0.0.1:44120 remote=127.0.0.1:18070 FD 16 flags=1) which is
probably a bug.

2016/12/29 15:57:41.121| 17,3| FwdState.cc(332) Start: '
http://172.22.8.145:3128/squid-internal-periodic/store_digest'
2016/12/29 15:57:41.121| 17,2| FwdState.cc(133) FwdState: Forwarding client
request , url=http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 15:57:41.121| 17,3| FwdState.cc(387) startConnectionOrFail:
http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 15:57:41.121| 17,3| FwdState.cc(806) connectStart:
fwdConnectStart:
http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 15:57:41.121| 17,3| FwdState.cc(875) connectStart: reusing pconn
local=127.0.0.1:44120 remote=127.0.0.1:18070 FD 16 flags=1
2016/12/29 15:57:41.121| 17,3| FwdState.cc(908) dispatch: : Fetching GET
http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 15:57:41.124| 17,3| FwdState.cc(447) unregister:
http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 15:57:41.124| 17,2| FwdState.cc(655)
handleUnregisteredServerEnd: self=0x1450738*2 err=0
http://172.22.8.145:3128/squid-internal-periodic/store_digest

And peer_select logs:

2016/12/29 16:12:41.843| 44,3| peer_select.cc(137) peerSelect:
e:=IWV/0x148bae0*2
http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 16:12:41.843| 44,3| peer_select.cc(441) peerSelectFoo: GET
172.22.8.145
2016/12/29 16:12:41.843| 44,3| peer_select.cc(446) peerSelectFoo:
peerSelectFoo: direct = DIRECT_UNKNOWN (always_direct to be checked)
2016/12/29 16:12:41.844| 44,3| peer_select.cc(194)
peerCheckAlwaysDirectDone: peerCheckAlwaysDirectDone: DENIED
2016/12/29 16:12:41.844| 44,3| peer_select.cc(441) peerSelectFoo: GET
172.22.8.145
2016/12/29 16:12:41.844| 44,3| peer_select.cc(454) peerSelectFoo:
peerSelectFoo: direct = DIRECT_UNKNOWN (never_direct to be checked)
2016/12/29 16:12:41.844| 44,3| peer_select.cc(171)
peerCheckNeverDirectDone: peerCheckNeverDirectDone: ALLOWED
2016/12/29 16:12:41.844| 44,3| peer_select.cc(177)
peerCheckNeverDirectDone: direct = DIRECT_NO (never_direct allow)
2016/12/29 16:12:41.844| 44,3| peer_select.cc(441) peerSelectFoo: GET
172.22.8.145
2016/12/29 16:12:41.844| 44,3| peer_select.cc(110) peerSelectIcpPing:
peerSelectIcpPing:
http://172.22.8.145:3128/squid-internal-periodic/store_digest
2016/12/29 16:12:41.844| 44,3| peer_select.cc(121) peerSelectIcpPing:
peerSelectIcpPing: counted 0 neighbors
2016/12/29 16:12:41.844| 44,3| peer_select.cc(685) peerGetSomeParent: GET
172.22.8.145
2016/12/29 16:12:41.844| 44,3| peer_select.cc(709) peerGetSomeParent:
peerSelect: FIRSTUP_PARENT/127.0.0.1
2016/12/29 16:12:41.844| 44,5| peer_select.cc(938) peerAddFwdServer:
peerAddFwdServer: adding 127.0.0.1 FIRSTUP_PARENT
2016/12/29 16:12:41.844| 44,5| peer_select.cc(938) peerAddFwdServer:
peerAddFwdServer: adding 127.0.0.1 ANY_OLD_PARENT
2016/12/29 16:12:41.844| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find
IP destination for:
http://172.22.8.145:3128/squid-internal-periodic/store_digest' via 127.0.0.1
2016/12/29 16:12:41.844| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find
IP destination for:
http://172.22.8.145:3128/squid-internal-periodic/store_digest' via 127.0.0.1
2016/12/29 16:12:41.844| 44,2| peer_select.cc(280) peerSelectDnsPaths:
Found sources for '
http://172.22.8.145:3128/squid-internal-periodic/store_digest'
2016/12/29 16:12:41.844| 44,2| peer_select.cc(281) peerSelectDnsPaths:
always_direct = DENIED
2016/12/29 16:12:41.844| 44,2| peer_select.cc(282) peerSelectDnsPaths:
 never_direct = ALLOWED
2016/12/29 16:12:41.844| 44,2| peer_select.cc(292) peerSelectDnsPaths:
 cache_peer = local=0.0.0.0 remote=127.0.0.1:18070 flags=1
2016/12/29 16:12:41.844| 44,2| peer_select.cc(292) peerSelectDnsPaths:
 cache_peer = local=0.0.0.0 remote=127.0.0.1:18070 flags=1
2016/12/29 16:12:41.844| 44,2| peer_select.cc(295) peerSelectDnsPaths:
   timedout = 0
2016/12/29 16:12:41.844| 44,3| peer_select.cc(79) ~ps_state:
http://172.22.8.145:3128/squid-internal-periodic/store_digest

On Thu, Dec 29, 2016 at 2:21 PM, Ivan Larionov <xeron.oskom at gmail.com>
wrote:

> Thank you for helping.
>
> After some experiments and tcpdumping it looks like it's not sibling
> sending request to the parent, but original squid!
>
> So instead of asking sibling about his digests squid asks parent.
>
> And your trick with urlpath_regex didn't help. I even tried:
>
> acl internal_digest urlpath_regex +i /.*store_digest.*/
> always_direct allow internal_digest
> never_direct deny internal_digest
>
> but no luck. It still asks parent.
>
>
> On Thu, Dec 29, 2016 at 1:00 AM, Amos Jeffries <squid3 at treenet.co.nz>
> wrote:
>
>> On 2016-12-29 20:51, Ivan Larionov wrote:
>>
>>> I'm sure about forwarding because I see requests to
>>> http://172.22.15.88:3128/squid-internal-periodic/store_digest [1] in
>>> parent logs and my parent returns 502 because we do not allow requests
>>> to internal IPs. Logs from the parent:
>>>
>>> Got request: GET
>>> http://172.22.15.88:3128/squid-internal-periodic/store_digest
>>> Not allowing blacklisted IP 172.22.15.88
>>> GET http://172.22.15.88:3128/squid-internal-periodic/store_digest 502
>>> 0ms
>>>
>>> I do not have "global_internal_static off" in my config and also I'm
>>> able to get
>>> http://172.22.15.88:3128/squid-internal-periodic/store_digest [1]
>>> using curl or telnet (with telnet I do "GET
>>> /squid-internal-periodic/store_digest" – note relative URL).
>>>
>>
>> Okay, thats good.
>>
>>
>>> However according to debug logs squid does this request using absolute
>>> URL which probably works if target sibling can do direct requests (so
>>> it will request itself for digest and return response to original
>>> squid). But I do have "never_direct allow all" which probably makes
>>> sibling to forward such request to a parent.
>>>
>>
>> Hmm, I think you might be right about that.
>> You can test it by adding:
>>
>>  acl foo urlpath_regex +i /squid.internal.digest/
>>  never_direct deny foo
>>
>>
>>
>>> If my theory about absolute vs relative URL is correct then I believe
>>> original squid should make store_digest request using relative URL
>>> (like I can do with telnet) so sibling squid will return response
>>> right away w/o asking itself for result.
>>>
>>
>> Whats happening with the URL is that the sending peer generates it from
>> the cache_peer IP/host name and port.
>>
>> The receiving peer checks the pathstarts with "/squid-internal-" and that
>> the hostname portion matches its own visible_hostname or unique_hostname.
>> If those match its marked for special handling as an internal request,
>> otherwise global_internal_static is used to determine if the hostname not
>> matching is ignored and it gets marked anyway.
>>
>> Since the digest needs to be targeted at the specific peer and not
>> anything which may inject itself in between them the hostname does need to
>> be sent. The relative URLs are for things that don't vary between proxies,
>> like the Squid icons.
>>
>> If you configure cache_peer with the hostname of the receiving peer
>> instead of its raw-IP the requests should be sent with that hostname
>> instead of raw-IP.
>>
>>
>>
>> The config looks okay. Thanks for that.
>>
>> Amos
>>
>>
>
>
> --
> With best regards, Ivan Larionov.
>

-- 
With best regards, Ivan Larionov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20161229/a2d6b76a/attachment-0001.html>