[squid-users] ACL dst handled differently in intercept after rewrite

[Amos Jeffries]

On 15/02/2017 6:27 a.m., Craig Gowing wrote:
> Hi all,
> 
> I've got a squid server running which allows direct proxy and also can
> intercept traffic:
> 
> http_port 10.0.0.1:3128
> http_port 10.0.0.1:3129 intercept
> 
> ---
> 
> There is a URL rewriter which allows the incoming requests (this is just an
> example, I don't really allow all):
> 
> url_rewrite_access allow all
> url_rewrite_program /usr/bin/myrewriter
> 
> ---
> 
> This rewriter will rewrite some URLs to a host on the same network, with
> the intention that the request should not be cached by squid, eg
> http://example.net/somefile.bin -> http://10.0.0.2/example.net/somefile.bin
> So a cache_deny directive is used for this:
> 
> acl local_store dst 10.0.0.2
> cache deny local_store
> 
> ---
> 
> Now when requesting this URL using a defined proxy ...

Where the client sends a URL and expects the proxy to perform DNS
lookups. The destination is any one of a set of IPs returned by DNS from
a lookup performed by Squid.

> ... the ACL matches and the request is not cached.

> If using intercept ...

Where the destination is a specific IP address provided by the NAT
system on the machine Squid is running on. It was selected by the client
from a DNS lookup from that clients machine.

> ... the ACL does not match and it
> does get cached (which caused some storage duplication on the network)
> The debug info shows the following:
> 
> Proxy:
> curl -x "10.0.0.1:3128" "http://example.net/somefile.bin" > /dev/null
> Ip.cc(95) aclIpAddrNetworkCompare: aclIpAddrNetworkCompare: compare:
> 10.0.0.2/[ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff] (10.0.0.2)  vs
> 10.0.0.2-[::]/[ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff]
> 
> Intercept:
> curl "http://example.net/somefile.bin" > /dev/null # Intercepted on the NAT
> tables
> Ip.cc(95) aclIpAddrNetworkCompare: aclIpAddrNetworkCompare: compare:
> 93.184.216.34:80/[ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff] (93.184.216.34:80)
>  vs 10.0.0.2-[::]/[ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff]
> 
> This seems to show that the ACL is processed at a different stage for the
> two different modes. Now I'm wondering if this is intentional and I
> shouldn't be using the 'dst' ACL here, or should it be more consistant and
> give the same result regardless?

It is intentional.

A NAT intercept proxy has a single specific dst-IP selected by the
client from a DNS lookup done before anythign got near to Squid.

An explicit-proxy has a whole set of potential dst-IPs found from
looking up the URL domain name (after URL rewriter changes) - any one of
which can match.

The URL-rewriter is doing exactly that. Re-writing the *URL*, not the
destination the client was attempting to reach.


> 
> I have a solution to use the 'url_regex' ACL instead which seems consistant
> between the two modes, but it may slightly affect performance.

Why not use dstdomain ACL? it was designed for matching the domain name
which is being requested by the client.


In fact whats the point of the rewriter pre-pending a raw-IP address in
the first place?

> 
> I couldn't find a huge amount of info on what order the ACLs are processed,
> so if anybody could let me know what the expected behaviour should be that
> would be much appreciated.
> 

<http://wiki.squid-cache.org/ProgrammingGuide/Architecture#Transaction_Processing>
is the best we have other than the code itself.


Amos