[squid-users] squid sibling peers and digest requests

Fri Dec 30 00:40:50 UTC 2016

Ok, sorry for so many messages. This is the last one :)

In the end what helped was this:

acl internal_digest urlpath_regex +i ^/squid-internal-periodic/store_digest$
always_direct allow internal_digest
never_direct deny internal_digest

So Amos' original idea with ACL was correct, I just had to adjust it a bit.

Looks like "never_direct allow all" which I have later in config affects
store_digest requests. Not sure if it's a bug or feature.

Thank you for helping again.

On Thu, Dec 29, 2016 at 4:15 PM, Ivan Larionov <xeron.oskom at gmail.com>
wrote:

> Here are some debug logs from FwdState which handles digest request.
>
> 172.22.13.210 – original squid
> 172.22.8.145 – sibling squid
> 127.0.0.1:18070 – parent
>
> As you can see it uses connection to parent for this request (reusing
> pconn local=127.0.0.1:44120 remote=127.0.0.1:18070 FD 16 flags=1) which
> is probably a bug.
>
> 2016/12/29 15:57:41.121| 17,3| FwdState.cc(332) Start: '
> http://172.22.8.145:3128/squid-internal-periodic/store_digest'
> 2016/12/29 15:57:41.121| 17,2| FwdState.cc(133) FwdState: Forwarding
> client request , url=http://172.22.8.145:3128/
> squid-internal-periodic/store_digest
> 2016/12/29 15:57:41.121| 17,3| FwdState.cc(387) startConnectionOrFail:
> http://172.22.8.145:3128/squid-internal-periodic/store_digest
> 2016/12/29 15:57:41.121| 17,3| FwdState.cc(806) connectStart:
> fwdConnectStart: http://172.22.8.145:3128/squid-internal-periodic/store_
> digest
> 2016/12/29 15:57:41.121| 17,3| FwdState.cc(875) connectStart: reusing
> pconn local=127.0.0.1:44120 remote=127.0.0.1:18070 FD 16 flags=1
> 2016/12/29 15:57:41.121| 17,3| FwdState.cc(908) dispatch: : Fetching GET
> http://172.22.8.145:3128/squid-internal-periodic/store_digest
> 2016/12/29 15:57:41.124| 17,3| FwdState.cc(447) unregister:
> http://172.22.8.145:3128/squid-internal-periodic/store_digest
> 2016/12/29 15:57:41.124| 17,2| FwdState.cc(655)
> handleUnregisteredServerEnd: self=0x1450738*2 err=0
> http://172.22.8.145:3128/squid-internal-periodic/store_digest
>
> And peer_select logs:
>
> 2016/12/29 16:12:41.843| 44,3| peer_select.cc(137) peerSelect:
> e:=IWV/0x148bae0*2 http://172.22.8.145:3128/squid-internal-periodic/store_
> digest
> 2016/12/29 16:12:41.843| 44,3| peer_select.cc(441) peerSelectFoo: GET
> 172.22.8.145
> 2016/12/29 16:12:41.843| 44,3| peer_select.cc(446) peerSelectFoo:
> peerSelectFoo: direct = DIRECT_UNKNOWN (always_direct to be checked)
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(194)
> peerCheckAlwaysDirectDone: peerCheckAlwaysDirectDone: DENIED
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(441) peerSelectFoo: GET
> 172.22.8.145
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(454) peerSelectFoo:
> peerSelectFoo: direct = DIRECT_UNKNOWN (never_direct to be checked)
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(171)
> peerCheckNeverDirectDone: peerCheckNeverDirectDone: ALLOWED
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(177)
> peerCheckNeverDirectDone: direct = DIRECT_NO (never_direct allow)
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(441) peerSelectFoo: GET
> 172.22.8.145
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(110) peerSelectIcpPing:
> peerSelectIcpPing: http://172.22.8.145:3128/squid-internal-periodic/store_
> digest
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(121) peerSelectIcpPing:
> peerSelectIcpPing: counted 0 neighbors
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(685) peerGetSomeParent: GET
> 172.22.8.145
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(709) peerGetSomeParent:
> peerSelect: FIRSTUP_PARENT/127.0.0.1
> 2016/12/29 16:12:41.844| 44,5| peer_select.cc(938) peerAddFwdServer:
> peerAddFwdServer: adding 127.0.0.1 FIRSTUP_PARENT
> 2016/12/29 16:12:41.844| 44,5| peer_select.cc(938) peerAddFwdServer:
> peerAddFwdServer: adding 127.0.0.1 ANY_OLD_PARENT
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(258) peerSelectDnsPaths:
> Find IP destination for: http://172.22.8.145:3128/
> squid-internal-periodic/store_digest' via 127.0.0.1
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(258) peerSelectDnsPaths:
> Find IP destination for: http://172.22.8.145:3128/
> squid-internal-periodic/store_digest' via 127.0.0.1
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(280) peerSelectDnsPaths:
> Found sources for 'http://172.22.8.145:3128/squid-internal-periodic/store_
> digest'
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(281) peerSelectDnsPaths:
> always_direct = DENIED
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(282) peerSelectDnsPaths:
>  never_direct = ALLOWED
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(292) peerSelectDnsPaths:
>  cache_peer = local=0.0.0.0 remote=127.0.0.1:18070 flags=1
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(292) peerSelectDnsPaths:
>  cache_peer = local=0.0.0.0 remote=127.0.0.1:18070 flags=1
> 2016/12/29 16:12:41.844| 44,2| peer_select.cc(295) peerSelectDnsPaths:
>    timedout = 0
> 2016/12/29 16:12:41.844| 44,3| peer_select.cc(79) ~ps_state:
> http://172.22.8.145:3128/squid-internal-periodic/store_digest
>
>
> On Thu, Dec 29, 2016 at 2:21 PM, Ivan Larionov <xeron.oskom at gmail.com>
> wrote:
>
>> Thank you for helping.
>>
>> After some experiments and tcpdumping it looks like it's not sibling
>> sending request to the parent, but original squid!
>>
>> So instead of asking sibling about his digests squid asks parent.
>>
>> And your trick with urlpath_regex didn't help. I even tried:
>>
>> acl internal_digest urlpath_regex +i /.*store_digest.*/
>> always_direct allow internal_digest
>> never_direct deny internal_digest
>>
>> but no luck. It still asks parent.
>>
>>
>> On Thu, Dec 29, 2016 at 1:00 AM, Amos Jeffries <squid3 at treenet.co.nz>
>> wrote:
>>
>>> On 2016-12-29 20:51, Ivan Larionov wrote:
>>>
>>>> I'm sure about forwarding because I see requests to
>>>> http://172.22.15.88:3128/squid-internal-periodic/store_digest [1] in
>>>> parent logs and my parent returns 502 because we do not allow requests
>>>> to internal IPs. Logs from the parent:
>>>>
>>>> Got request: GET
>>>> http://172.22.15.88:3128/squid-internal-periodic/store_digest
>>>> Not allowing blacklisted IP 172.22.15.88
>>>> GET http://172.22.15.88:3128/squid-internal-periodic/store_digest 502
>>>> 0ms
>>>>
>>>> I do not have "global_internal_static off" in my config and also I'm
>>>> able to get
>>>> http://172.22.15.88:3128/squid-internal-periodic/store_digest [1]
>>>> using curl or telnet (with telnet I do "GET
>>>> /squid-internal-periodic/store_digest" – note relative URL).
>>>>
>>>
>>> Okay, thats good.
>>>
>>>
>>>> However according to debug logs squid does this request using absolute
>>>> URL which probably works if target sibling can do direct requests (so
>>>> it will request itself for digest and return response to original
>>>> squid). But I do have "never_direct allow all" which probably makes
>>>> sibling to forward such request to a parent.
>>>>
>>>
>>> Hmm, I think you might be right about that.
>>> You can test it by adding:
>>>
>>>  acl foo urlpath_regex +i /squid.internal.digest/
>>>  never_direct deny foo
>>>
>>>
>>>
>>>> If my theory about absolute vs relative URL is correct then I believe
>>>> original squid should make store_digest request using relative URL
>>>> (like I can do with telnet) so sibling squid will return response
>>>> right away w/o asking itself for result.
>>>>
>>>
>>> Whats happening with the URL is that the sending peer generates it from
>>> the cache_peer IP/host name and port.
>>>
>>> The receiving peer checks the pathstarts with "/squid-internal-" and
>>> that the hostname portion matches its own visible_hostname or
>>> unique_hostname. If those match its marked for special handling as an
>>> internal request, otherwise global_internal_static is used to determine if
>>> the hostname not matching is ignored and it gets marked anyway.
>>>
>>> Since the digest needs to be targeted at the specific peer and not
>>> anything which may inject itself in between them the hostname does need to
>>> be sent. The relative URLs are for things that don't vary between proxies,
>>> like the Squid icons.
>>>
>>> If you configure cache_peer with the hostname of the receiving peer
>>> instead of its raw-IP the requests should be sent with that hostname
>>> instead of raw-IP.
>>>
>>>
>>>
>>> The config looks okay. Thanks for that.
>>>
>>> Amos
>>>
>>>
>>
>>
>> --
>> With best regards, Ivan Larionov.
>>
>
>
>
> --
> With best regards, Ivan Larionov.
>

-- 
With best regards, Ivan Larionov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20161229/326d9c46/attachment.html>