[squid-dev] [PATCH] New format code %acl_matched to log the last matched acl

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18/11/2014 5:55 a.m., Alfonso Ali wrote:
> On 11/16/2014 06:14 AM, Amos Jeffries wrote:
> 
>> What exactly are those use-cases please? Accounting what
>> exactly?
> 
> We have a lot of sites classified in some categories (tech,
> health, culture, etc.) and we need to generate traffic reports
> based on those categories. Each category can be composed of more
> than one acl type (dstdomain, url_regex). When parsing the logs we
> can extract to which category belongs each url based on the matched
> acl.
> 
> The other user case is related with quota accounting, we have a
> lot sites that are free (mainly the ones described above) for each
> user and the quota program use the acl matche to know if the
> request have to be accounted or not.
> 
> 
>> This is important to document as it informs us whether this patch
>> is only useful for you, or could be useful for others. It may
>> also be
> 
> I understand, we decided to send it now becouse other institutions
> asked us about our solution, so i though it could be useful for
> other too.
> 
>> that your use-case is far better served by some other feature or
>> patch addition.
> 
> Agreed, as i said before our first approach to the problem was to
> use an external acl that generated a log=%s reply to be used on the
> log files with the %ea format code.

The traffic classification use-case is best served by the annotation
feature now available in Squid-3.4. It is like the tag= response from
external ACL helpers. But annotations can be added from any helper at
all, or explicitly by the note directive.

> 
>> ? sounds like a bug. In which case fixing the bug is the right 
>> approach. Some details on what you found would be appreciated.
>> You can report through bugzilla to keep this thread clean.
> 
> We looked into it, but is was very difficult to debug since it
> only ocurred at high loads and the way to verify it is parsing the
> whole log file to see if each url match the correct acl (we only
> found this issue becouse some users complained about been accounted
> incorrectly), at first we though it was a problem related with the
> acl's ttl or buffering but since squid already knows which acl was
> used to allow the request and logging it from squid will have the
> benefit of reducing the load associated with the external programs
> used for the external acl's we decided for this solution.

Hm. If an external ACL helper is used multiple times ony the last log=
delivered will be used in the log. This may have been part of the
problem. Older Squid would often repeat ACL tests with new data when
resuming from external lookups.

> 
>> Please be aware the named ACL is not valid to be used outside of 
>> Checklist matching sequences. It contains the last ACL to be
>> named *including* any ACLs tested in figuring out where to log
>> the traffic. Even if no ACLs are tested determining the log to
>> write to, the name may be altered at any time by a concurrent
>> request being processed through some other ACLs.
> 
> I don't have much knowledge of squid's code, i just looked at how
> others format codes where defined and how the debug info about the
> matched acl was generated (line 739 in client_side_request.cc).
> 
> I though (but don't checked) that the http object and
> AclMatchedName variable where unique for each request or at least
> until the log line was generated.
> 
> Do you think that if i make a copy of AclMatchedName's value on the
> http object i can ensure that the correct info is generated in the
> log file?

Possibly. But it is just adding another custom tag onto the
transaction. We are trying at present to reduce the number of those
type of mechanisms. Centering on the annotation feature as a single
unified way to do it instead.

Amos
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUam+fAAoJELJo5wb/XPRjcKYIALNHi9K8SE8MrEx4FHymuItS
rGsPCnK27vN5WXcbV3zHHB5W3MtFL7a6CUEqlDVdlO5DJ3pZo2Ja3fN9ZAP/sG3P
4NWbLwS4lslmR38bpe8wN0IagEMz+KMQZ4UE2V0gBslJSI+I7DrrD+XiK0yMig+y
He+Zwl1R+EXQuUaIB1wbgrppcoq32jjLIgu+N/WloX2oj6qYD28FWVcImQVqC3ps
NAKdVp8ocZ03Tj8naEwO2d0qzXue9kE74W47P3ENms/J/kNOleFH4jVux7ekmeKK
bANgw/3ggN4uXJAhRpLmfrMwcAODYP6gjgRrkaPrDggjBWT7OHPu2ZEQPgj+nns=
=HbM5
-----END PGP SIGNATURE-----