[squid-users] Get IP of denied request

[magri at web.de]

Am 27.10.23 um 16:22 schrieb Alex Rousskov:
> On 2023-10-27 07:14, magri at web.de wrote:
>> Am 26.10.23 um 21:11 schrieb Alex Rousskov:
>>> On 2023-10-26 08:37, magri at web.de wrote:
>>>> TL;DR: is there a way to get/log the resolved ip of a denied request?
>>>
>>> TLDR: Bugs notwithstanding, use %<a.
>
>> %<a doesn't work :-(
>
> Sorry, my first response was wrong: As you have correctly explained, %<a
> is the destination address of the used next hop connection while dst ACL
> uses an address derived from the request-target. The two addresses do
> not have to match and, more importantly, it is wrong to expect a dst ACL
> to update/store that "used next hop address" information when the ACL
> itself does not open or reuse a connection to that address.
>
>
>>>      acl matchDst1 dst 127.0.0.1
>>>      acl markDst1 note matched=127.0.0.1
>>>      http_access deny matchDst1 markDst1
>>>
>>>      acl matchDst2 dst 127.0.0.2
>>>      acl markDst2 note matched=127.0.0.2
>>>      http_access deny matchDst2 markDst2
>>>
>>>      logformat myFormat ... matched_dst=%note{matched}
>>>      access_log ...
>>>
>>> For long dst lists, the above approach will require scripting the
>>> generation of the corresponding squid.conf portions or include files, of
>>> course.
>
>> I don't think this scales to blacklists with 6-digit count sizes
>
> I think that depends on the definition of "to scale".
>
>
>> and it also doesn't work for blacklisted networks :-(
>
> Agreed. This approach can only log which dst ACL data matched, and that
> data may be different from the IP.
>
>
>> I hoped there would be a way to get the ip as some kind of variable like
>> the header fields in logformat.
>
> Yes, I know. I was just documenting an existing workaround for cases
> where it is usable.
>
>
>> Any ideas?
>
> I have three:
>
> 1. Enhance Squid to resolve transaction destination address once (on
> first use/need). Remember/reuse resolved IP addresses. Log them via some
> new %resolved_dst and %dst_resolution_detail codes.
>
> This improvement will help address a few use cases unrelated to this
> discussion, but it will _not_ tell you which of the resolved addresses
> actually matched which ACL. You will be able to guess in many cases, but
> there will be exceptions.
>
>
> 2. Add a Squid feature where an evaluated ACL can be configured to
> annotate the transaction with the information about that evaluation.
>
> To start with, we can support annotations for _matched_ dst ACLs. For
> example:
>
>      # When matches, sets the following master transaction annotations:
>      # * badDestination_input: used destination address (IP or name)
>      # * badDestination_match: the matched destination IP
>      # * badDestination_value: the matched ACL parameter value
>      # * badDestination_ips: resolved destination IP(s)
>      # * badDestination_errors: DNS resolution and other error(s)
>      # * badDestination_matches: number of matches (this transaction)
>      # * badDestination_evals: number of evaluations (this transaction)
>      acl badDestination dst --on-match=annotate 127.0.0.0/24 10.0.0.1
>
> If the same ACL matches more than once, the last(?) match wins, but the
> aclfoo_matches annotation can be used to detect these cases. The
> aclfoo_evals annotation can be used to detect whether this ACL was
> "reached" at all.
>
> If really needed, we can support turning individual annotations on and
> off, but I doubt that complexity is worth associated performance
> savings. After all, most ACLs will only match once per transaction
> lifetime (for correctly written configurations). Access.log will be
> configured to only log annotations of interest to the admin, of course.
>
>
> The above approach can be extended to provide ACL debugging aid:
>
>      # Dumps every mismatch information to cache.log at level 1
>      acl goodDestination dst --on-mismatch=log 127.0.0.0/24
>
>      # Dumps every evaluation information to cache.log at level 1
>      acl redDestination dst --on-eval=log 127.0.0.0/24
>
>
> 3. Add a Squid feature where Squid (optionally) maintains an internal
> database of recent ACL evaluation history and makes that information
> accessible via cache manager queries like "which ACLs matched
> transaction X?" (where X is logged %master_xaction ID).
>
>
> The three sketched options are not mutually exclusive, of course. All
> require non-trivial code changes.
>
> Would any of the above options address your needs? Any preferences or
> spec adjustments?

Very nice and detailed options!

Let me first ask some questions for clarification:
- Does squid cache all ips from dns responses with multiple ips or only
the one it uses for the request?
- If it caches more than one ip - does squid use more than one of these
ips (e.g. as fallback or round robin) inside a single transaction or for
multiple transactions?

Supposing squid uses only a single dst ip inside a single transaction,
your first option would be sufficient for our purpose!

Resolving the ip only once and storing it inside the transaction would
also avoid ambiguous cases where different ips could theoretically be
used in different acls or worse in acls and the real connection.
It could also improve the performance of acl evaluation because after
the resolution of the dst ip all following dst acls would be evaluated
fast(?).

Before the introduction of annotations we hat to use some acls twice
(for http_access and dedicated logs for this acl) but that's not
necessary anymore. We still have several dst acls that could benefit
from storing the once resolved dst ip.

Your second option sounds really nice for debugging purposes but IMHO
these annotations should be optional because even if the performance
penalty may(?) be neglectable they increase the size of the transaction
object and this could accumulate if many acls are in use.
Hence I would propose this as optional extension to the first option.

Your third option seems not feasible for us, because the delay between a
failed request and reporting of the failure often takes more than a day.
The needed history would be quite excessive (with over 80 million
requests a day).

Thank you,
Martin

>
>
>
> Thank you,
>
> Alex.
>
>
>>>> Is there any way to get the ip logged that was used in the dst-acl
>>>> aside
>>>> from debug logging? Maybe through some annotation mechanism?
>>>>
>>>> Squid version is 6.2, as 6.4 crashes with assertion errors here, too.
>>>>
>>>> thanks,
>>>> Martin
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users