[squid-users] Get IP of denied request

[Alex Rousskov]

On 2023-10-30 13:08, magri at web.de wrote:
> Am 27.10.23 um 16:22 schrieb Alex Rousskov:
>> 1. Enhance Squid to resolve transaction destination address once (on
>> first use/need). Remember/reuse resolved IP addresses. Log them via some
>> new %resolved_dst and %dst_resolution_detail codes.
>>
>> This improvement will help address a few use cases unrelated to this
>> discussion, but it will _not_ tell you which of the resolved addresses
>> actually matched which ACL. You will be able to guess in many cases, but
>> there will be exceptions.
>>
>>
>> 2. Add a Squid feature where an evaluated ACL can be configured to
>> annotate the transaction with the information about that evaluation.
>>
>> To start with, we can support annotations for _matched_ dst ACLs. For
>> example:
>>
>>      # When matches, sets the following master transaction annotations:
>>      # * badDestination_input: used destination address (IP or name)
>>      # * badDestination_match: the matched destination IP
>>      # * badDestination_value: the matched ACL parameter value
>>      # * badDestination_ips: resolved destination IP(s)
>>      # * badDestination_errors: DNS resolution and other error(s)
>>      # * badDestination_matches: number of matches (this transaction)
>>      # * badDestination_evals: number of evaluations (this transaction)
>>      acl badDestination dst --on-match=annotate 127.0.0.0/24 10.0.0.1
>>
>> If the same ACL matches more than once, the last(?) match wins, but the
>> aclfoo_matches annotation can be used to detect these cases. The
>> aclfoo_evals annotation can be used to detect whether this ACL was
>> "reached" at all.
>>
>> If really needed, we can support turning individual annotations on and
>> off, but I doubt that complexity is worth associated performance
>> savings. After all, most ACLs will only match once per transaction
>> lifetime (for correctly written configurations). Access.log will be
>> configured to only log annotations of interest to the admin, of course.
>>
>>
>> The above approach can be extended to provide ACL debugging aid:
>>
>>      # Dumps every mismatch information to cache.log at level 1
>>      acl goodDestination dst --on-mismatch=log 127.0.0.0/24
>>
>>      # Dumps every evaluation information to cache.log at level 1
>>      acl redDestination dst --on-eval=log 127.0.0.0/24
>>
>>
>> 3. Add a Squid feature where Squid (optionally) maintains an internal
>> database of recent ACL evaluation history and makes that information
>> accessible via cache manager queries like "which ACLs matched
>> transaction X?" (where X is logged %master_xaction ID).
>>
>>
>> The three sketched options are not mutually exclusive, of course. All
>> require non-trivial code changes.



> Let me first ask some questions for clarification:
> - Does squid cache all ips from dns responses with multiple ips or only
> the one it uses for the request?

All.


> - If it caches more than one ip - does squid use more than one of these
> ips (e.g. as fallback or round robin) inside a single transaction or for
> multiple transactions?

A single transaction will go through several single-name IPs after 
certain transaction failures, but not all failures are treated the same.

Multiple transactions may use multiple single-name IPs due to 
round-robin rotation of cached IPs and other factors.


> Supposing squid uses only a single dst ip inside a single transaction,
> your first option would be sufficient for our purpose!

In my first option, logged %resolved_dst (and the corresponding master 
transaction metadata) may contain multiple IPs (i.e. all resolved ones). 
AFAICT, if any one of those IPs match a "dst" ACL address, then that 
"dst" ACL should match, even if the transaction does not actually use 
the matched address: The "dst" ACL matches based on request-target info, 
not the _use_ of that info.

One of the difficulties here is to decide whether a transaction should 
wait for both IPv4 and IPv6 addresses to be resolved:

If a transaction proceeds with one family of addresses, then there will 
be no delays associated with waiting for the second one. However, in 
that case, Squid may declare a "dst" ACL mismatch even though there 
would have been a match if we waited for the second DNS answer! It feels 
like for correctness sake, we must wait if a "dst" ACL needs checking or 
a similar decision has to be made...


> Resolving the ip only once and storing it inside the transaction would
> also avoid ambiguous cases where different ips could theoretically be
> used in different acls or worse in acls and the real connection.

Yes, that is one of the positive side effects of that option.


> It could also improve the performance of acl evaluation because after
> the resolution of the dst ip all following dst acls would be evaluated
> fast(?).

A speedup is possible in some corner cases, but, in most cases, there 
will be no difference in this area because "all following dst ACLs" 
should "immediately" use _cached_ IP(s) in the current code AFAICT.


> Your second option sounds really nice for debugging purposes but IMHO
> these annotations should be optional because even if the performance
> penalty may(?) be neglectable they increase the size of the transaction
> object and this could accumulate if many acls are in use.
> Hence I would propose this as optional extension to the first option.

Yes, the second feature is optional (and orthogonal): If one does not 
add "--on-match=annotate" to a "dst" ACL declaration, then they do not 
get those annotations (and their overheads).


> Your third option seems not feasible for us, because the delay between a
> failed request and reporting of the failure often takes more than a day.
> The needed history would be quite excessive (with over 80 million
> requests a day).

Agreed.

Alex.



>>>>> Is there any way to get the ip logged that was used in the dst-acl
>>>>> aside
>>>>> from debug logging? Maybe through some annotation mechanism?
>>>>>
>>>>> Squid version is 6.2, as 6.4 crashes with assertion errors here, too.
>>>>>
>>>>> thanks,
>>>>> Martin
>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> https://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users