[squid-users] ACL / http_access rules stop work using Squid 6+
Alex Rousskov
rousskov at measurement-factory.com
Fri Apr 19 04:04:41 UTC 2024
Hi Andre,
I did not receive your new reply, but I did find it in the mailing
list archives in time to download the logs. Thank you for sharing them!
FWIW, you may want to CC me in future emails (but I do not know whether
that will help).
Something really strange is going on. Squid runtime logs (both versions)
do not match the two identical "squid -k parse" logs you have shared
earlier. For example, at runtime, Squid v5 checks the following rules,
in this order:
1. http_access ... ArticaTSTSite ...
2. http_access ... NoCacheMGR ...
3. http_access ... LocalClient ...
4. http_access ... MyIPSrc ...
5. http_access ... internal_icons ...
6. http_access ... MgrInfoUri ...
The first 10 rules in "squid -k parse" output are different! I pasted
them below and marked those that are missing in runtime output:
1. http_access deny ArticaTSTSite ArticaTSTSiteAll
2. http_access allow NoCacheMGR ToArticaWWW
3. http_access allow LocalClient NoCacheMGR
4. http_access allow MyIPSrc NoCacheMGR
5. http_access allow internal_icons
6. Missing: http_access allow internal_icons
7. Missing: http_access deny NormalPorts to_localhost CVEFix all
8. Missing: http_access deny NormalPorts manager ManagerDeny all
9. Missing: http_access allow MgRPort manager all
10. http_access allow MgRPort MgrInfoUri all
The differences continue...
N.B. Rules 5 and 6 in parsing logs are the same, but Squid does not know
that (yet?). Squid will parse, consume, and evaluate both (if they are
reached and do not match). In runtime logs, Squid v5 only evaluates the
first one (#5) and then goes to a MgrInfoUri rule that is listed as #6
in Squid v5 runtime logs and as #10 in parse logs, skipping four rules.
Runtime logs do not contain just the ACL names. They also contain rule
numbers assigned at parsing time. The number should be incremented for
every http_access rule Squid parses. Those numbers do not match "squid
-k parse" output, reflecting the same "missing rules" problem, and
elevating my confidence that other rules are missing at runtime.
Squid v6 runtime logs show the same problem, but the differences start
later (and are more profound). For example, Squid v6 logs do not show
Group212 ACL being evaluated (rule #151 in parse logs) but do show
AnnotateFinalAllow ACL evaluated (rule #168 in parse logs and rule #43
in v6 runtime logs). Group98 (rule #116 in parse logs) is also missing
from v6 runtime logs.
Parsing logs show 168 http_access rules.
Running Squid v5 appears to have 163 http_access rules.
Running Squid v6 appears to have 43 http_access rules!
Bugs notwithstanding, Squid does not "skip" http_access rules that it
successfully parsed/consumed at startup or reconfiguration. Squid does
not evaluate higher-numbered rules after a rule matches, but that is not
what it happening here: For example, I see sequences where Squid v6
checks 43 rules from the first (in all logs) ArticaTSTSite rule to the
last (in all logs) AnnotateFinalAllow rule; the other 125 rules (seen in
"squid -k parse" logs) appear to be "missing" from v6 runtime logs.
I speculate that one of these things have happened (in no particular order):
A. I am confused by these partial logs.
B. Squid failed to parse/consume some http_access rules in your
configuration files. The set of rules Squid failed to parse differs
across v5 and v6. The logs do not detail parsing, so I cannot validate
this theory, but it would explain the missing rules. In the vast
majority of cases, there ought to be at least some level-0/1 cache.log
errors or warnings about failing to parse/consume an http_access rule!
One way to validate this theory is to compare "squid -k parse" output
with http_access lines in mgr:config cache manager report received from
a running Squid instance. You may need to modify squid.conf to allow
that cache manager query (see cachemgr_passwd). Squids, especially older
ones, have various bugs in mgr:config output so a manual comparison may
be needed.
Another way is to study "squid -k parse -X -d9" output that should have
more details for Squid v6. This is similar to the trick in (D) below,
but without runtime debugging.
C. Your "squid -k parse" commands do not match the actual configurations
that Squids are parsing and then using at runtime. The same mgr:config
trick from (B) above can be used to validate this theory.
D. Both Squids are suffering from some serious bug. The effects of that
bug result in different actual/used http_access rule sets in each version.
One way to confirm theory D is to study ALL,9 logs collected from Squid
v6 with "squid ... -X -d9" command line option (or a similar trick that
will include initial parsing in the log). The logs should detail at
least one transaction that should evaluate most http_access rules. Such
logs will address concern (A) as well, but you will probably have to
share them privately if you are using production configuration/instance.
HTH,
Alex.
On 2024-04-15 19:49, Andre Bolinhas wrote:
> Hi Alex,
> Thnks for your reply.
>
> Logs uploaded again, you can find it here.
>
> https://we.tl/t-QiSKMgclOb
>
> Best regards
>
> On 15/04/2024 14:12, Alex Rousskov wrote:
>> On 2024-04-14 17:23, Andre Bolinhas wrote:
>>
>>> Any tip on this matter? I want to upgrade to squid 6.9 but due to
>>> this issue, i'm stuck.
>>
>>
>> Hi Andre,
>>
>> Please note that I did _not_ receive your email quoted below. It
>> is in the email archive, so the problem is not on your end, but I just
>> wanted to mention that I was not (knowingly) ignoring you.
>>
>> > I have re-uploaded the cache.log files.
>>
>> The files have expired again. I have reviewed the diff you shared, but
>> cannot make further progress without those test logs. Hopefully, your
>> next list post reaches me.
>>
>> Alex.
>>
>>
>>> On 01/04/2024 11:53, Andre Bolinhas wrote:
>>>>
>>>> Hi Alex
>>>>
>>>> Thanks for your help on the matter.
>>>>
>>>>
>>>>> The logs archive you shared previously has expired, so I cannot
>>>>> double check, but from what I remember, the shared logs did not
>>>>> support the above assertion, so there may be more to the story
>>>>> here. However, to make progress, let's assume that v5 configuration
>>>>> files are identical to v6 configuration files.
>>>> If you want, I can run the same test with in a different debug
>>>> parameters, just tell which ones.
>>>>
>>>> I have re-uploaded the cache.log files.
>>>> https://we.tl/t-AB4XuUwuf7
>>>>
>>>>> One way to answer all of the above questions is to look at the
>>>>> following output:
>>>>>
>>>>> squid -k parse ... |& grep Processing:.http_access
>>>> There is no diff between both squid version, you can check it here
>>>> DiffNow - Compare Files, URLs, and Clipboard Contents Online
>>>> <https://www.diffnow.com/report/jsrva>
>>>>
>>>>> The logs archive you shared previously has expired, so I cannot
>>>>> double check, but from what I remember, the shared logs did not
>>>>> support the above assertion, so there may be more to the story
>>>>> here. However, to make progress, let's assume that v5 configuration
>>>>> files are identical to v6 configuration files.
>>>> The configuration files / folder are the same, the server is the
>>>> same, the only thing that changes is the Squid version
>>>>
>>>> On 29/03/2024 17:40, Alex Rousskov wrote:
>>>>> On 2024-03-25 15:13, Bolinhas André wrote:
>>>>>
>>>>>> Yes, the configuration is the same for both versions.
>>>>>
>>>>> The logs archive you shared previously has expired, so I cannot
>>>>> double check, but from what I remember, the shared logs did not
>>>>> support the above assertion, so there may be more to the story
>>>>> here. However, to make progress, let's assume that v5 configuration
>>>>> files are identical to v6 configuration files.
>>>>>
>>>>> 1. Is there an "http_access allow all AnnotateFinalAllow" rule?
>>>>>
>>>>> 2. Is there an "http_access deny HTTP Group38 AnnotateRule28" rule?
>>>>>
>>>>> 3. Assuming the answers are "yes" and "yes", which rule comes
>>>>> first? If you use include files, this question applies to the
>>>>> imaginary preprocessed squid.conf file with all the include files
>>>>> inlined (recursively if needed). That kind of preprocessed
>>>>> configuration is what Squid effectively sees when compiling
>>>>> http_access rules, one by one. Which of the two rules will Squid
>>>>> see first?
>>>>>
>>>>> One way to answer all of the above questions is to look at the
>>>>> following output:
>>>>>
>>>>> squid -k parse ... |& grep Processing:.http_access
>>>>>
>>>>> Replace "..." with your regular squid startup command line options
>>>>> and adjust standard error redirection (|&) as needed for your
>>>>> shell. Run the above command for both Squid v5 and v6 binaries. You
>>>>> should see output like this:
>>>>>
>>>>>
>>>>>> 2024/03/29 13:31:05| Processing: http_access allow manager
>>>>>> 2024/03/29 13:31:05| Processing: http_access deny all
>>>>>
>>>>>
>>>>> HTH,
>>>>>
>>>>> Alex.
>>>>>
>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> *De:* Alex Rousskov <rousskov at measurement-factory.com>
>>>>>> *Enviado:* segunda-feira, 25 de março de 2024 19:12
>>>>>> *Para:* squid-users at lists.squid-cache.org
>>>>>> *Assunto* Re: [squid-users] ACL / http_access rules stop work
>>>>>> using Squid 6+
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2024-03-22 09:38, Andre Bolinhas wrote:
>>>>>>
>>>>>> > In previous versions of squid, from 3 to 5.9, I use this kind
>>>>>> of deny
>>>>>> > rules and they work like charm
>>>>>> >
>>>>>> > acl AnnotateRule28 annotate_transaction accessrule=Rule28
>>>>>> > http_access deny HTTP Group38 AnnotateRule28
>>>>>> >
>>>>>> > This allows me to deny objects without bump / show the error page
>>>>>> > (deny_info)
>>>>>> >
>>>>>> > But using squid 6+ this rules stop to work and everything is
>>>>>> allowed.
>>>>>> >
>>>>>> > Example:
>>>>>> > Squid 5.9 (OK)
>>>>>> > https://ibb.co/YdKgL1Y
>>>>>> >
>>>>>> > Squid 6.8 (NOK)
>>>>>> > https://ibb.co/tbyY2GV
>>>>>> >
>>>>>> > Sample of both cache.log in debug mode
>>>>>> >
>>>>>> > https://we.tl/t-T7Nz1rVbVu
>>>>>>
>>>>>>
>>>>>> In you v6 logs, most logged transactions are allowed because a rule
>>>>>> similar to the one reconstructed below is matching:
>>>>>>
>>>>>> http_access allow all AnnotateFinalAllow
>>>>>>
>>>>>>
>>>>>> There are similar cases in v5 logs as well, but most denied v5
>>>>>> transactions match the following rule instead (i.e. the one you
>>>>>> shared
>>>>>> above):
>>>>>>
>>>>>> http_access deny HTTP Group38 AnnotateRule28
>>>>>>
>>>>>>
>>>>>> In your Squid configuration, v6 allow rule is listed much higher
>>>>>> than v5
>>>>>> deny rule (#43 vs #149). I do not see any signs of Group38 or
>>>>>> AnnotateRule28 ACL evaluation in v6 logs, as if the rule sets are
>>>>>> different for two different Squid instances. Are you using the
>>>>>> same set
>>>>>> of http_access rules for both Squid versions?
>>>>>>
>>>>>> Alex.
>>>>>>
>>>>>> _______________________________________________
>>>>>> squid-users mailing list
>>>>>> squid-users at lists.squid-cache.org
>>>>>> https://lists.squid-cache.org/listinfo/squid-users
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> https://lists.squid-cache.org/listinfo/squid-users
>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> https://lists.squid-cache.org/listinfo/squid-users
>>
More information about the squid-users
mailing list