[squid-users] cache_peer_access by dynamic ACL
Alexeyяр Gruzdov
my.shellac at gmail.com
Thu Apr 20 14:54:15 UTC 2023
The cpu is around 100% even no any requests is going....
For now I left just one cache_peer in configuration and got the new error:
commBind Cannot bind socket FD 28 to [::]: (13) Permission denied
чт, 20 апр. 2023 г. в 17:21, Alex Rousskov <rousskov at measurement-factory.com
>:
> On 4/20/23 04:23, Alexeyяр Gruzdov wrote:
>
> > cache_peer peerG1.com parent 40001 0 no-query no-digest name=peerG1
> >
> > external_acl_type ext_proxy_g1_type %LOGIN %DST /usr/local/bin/g1.py
> >
> > acl proxy_g1_ext_mark_acl ext_proxy_g1_type
> >
> > acl proxy_g1_ext_marked_acl annotate_transaction proxy=g1
> >
> > acl proxy_peerG1_acl note proxy g1
> >
> > http_access deny proxy_g1_ext_mark_acl proxy_g1_ext_marked_acl !all
> > .....
> > others http_access rules
> >
> > And this above works.
>
> Glad to hear that. ( If others are going to use the above as a guiding
> example, I would recommend naming these ACLs very differently, but that
> is not important to Squid. )
>
>
> > BUT
> > I am worried about why this my external script for ACL type loads the
> > one of core of CPU to 100%.....???
>
> External ACL caching aside, the script will be contacted once for every
> Squid transaction. Does your script CPU usage go down to zero when there
> is no traffic? If not, then there is a bug in the script itself.
>
> If you use the script from the command line, without Squid, does it
> consume a lot of CPU and/or take a lot of time per fake query? You can
> adjust the script to log the real query (when the script is used by
> Squid), so that you can easily replicate that query when running the
> script without Squid...
>
> The cache key in your case is (the expansion of) "%LOGIN %DST". It is
> enabled by default IIRC. Look for "cache" related options at
> http://www.squid-cache.org/Doc/config/external_acl_type/
>
>
> > ( I used three of workers in config,
> > but I can see a six process called like my external helper script, looks
> > like squid runs x2 process for external ACL )
>
> See external_acl_type children-* options:
> http://www.squid-cache.org/Doc/config/external_acl_type/
>
> In most environments, I recommend setting all three of them to the same
> value. Please note that these options are not SMP-aware (yet), so Squid
> will _not_ divide their values by the number of workers and give each
> worker as many children as you state in squid.conf.
>
>
> > Because if I will put the one more group of users (that must to use
> > another cache_peer ) - I will need to create one more external script
> > that will making to check an existed users from an other DB table
>
> Once you get the basic setup above working for one group to your
> satisfaction, I would recommend migrating from (one script and one
> matching annotate_transaction ACL) per group to a single script for all
> groups. That single external ACL script will send the right
> annotation(s) to Squid.
>
>
> HTH,
>
> Alex.
>
>
> > ср, 19 апр. 2023 г. в 22:39, Alex Rousskov:
> >
> > On 4/19/23 13:30, Alexeyяр Gruzdov wrote:
> >
> > > cache_peer peerG1.com parent 40001 0 no-query no-digest
> name=peerG1
> >
> > > external_acl_type ext_proxy_g1_type %LOGIN %DST
> /usr/local/bin/g1.py
> >
> > > acl proxy_g1_ext_acl ext_proxy_g1_type
> >
> > OK. I assume that /usr/local/bin/g1.py will only match users that
> > should
> > go to cache_peer called peerG1.
> >
> >
> > > acl proxy_g1_ext_acl_mark annotate_transaction proxy=g1
> >
> > Please note that the name of this annotate_transaction ACL --
> > "proxy_g1_ext_acl_mark" -- implies a relationship to the external ACL
> > named "proxy_g1_ext_acl", but there is no such relationship. Squid
> does
> > not care about ACL names, but this naming problem may indicate a
> > misunderstanding. To follow your naming scheme, this ACL should be
> > called something like "proxy_g1_mark_acl" or "mark_for_proxy_g1_acl".
> >
> >
> > > acl proxy_peerG1_acl note proxy g1
> >
> > OK. FWIW, a more consistent ACL name would have been
> > "proxy_g1_marked_acl" or "marked_for_proxy_g1_acl". Again, Squid does
> > not really care about these names, so use whatever you think is
> > consistent/meaningful/etc.
> >
> >
> > > http_access deny proxy_g1_ext_acl !all
> >
> > This line has no (positive) effect. Squid will evaluate the external
> > ACL, but since the rule, as a whole, will never match due to "!all",
> > and
> > since the external ACL has no (relevant) side effects, you can just
> > delete this line from your configuration.
> >
> > Needless to say, if you delete this line, then proxy_g1_ext_acl will
> be
> > unused, which should tell you that this configuration is not doing
> what
> > you want. See below for a fix recommendation.
> >
> >
> > > http_access deny proxy_g1_ext_acl_mark !all
> >
> > This line will mark _all_ transactions. You only want to mark
> > transactions that also matched proxy_g1_ext_acl. That "b only if a"
> > logic is accomplished by using _both_ ACLs in the same rule:
> >
> > http_access deny proxy_g1_ext_acl proxy_g1_ext_acl_mark !all
> >
> > With the above http_access rule (instead of the earlier two), Squid
> > will
> > evaluate the external ACL, and, if it matches, Squid will also
> evaluate
> > the annotation-setting ACL. The whole rule will then be rejected due
> to
> > "!all", but not until it annotates the transaction (if the external
> ACL
> > matches). Again, in this sketch, we are using this rule for its
> > annotation side effect only.
> >
> >
> > > And this works like I need now....
> >
> > AFAICT, if the tests indicate that this configuration works, then the
> > tests are broken. IMHO, you should fix the tests (while you have a
> > broken configuration that can be used to test the tests) before
> > proceeding with the configuration fix.
> >
> >
> > HTH,
> >
> > Alex.
> > P.S. Please keep this email thread on squid-users instead of
> responding
> > to me personally.
> >
> >
> >
> >
> > > ср, 19 апр. 2023 г. в 21:01, Alexeyяр Gruzdov:
> > >
> > > so, ok - Lets limit just to one cache peer and one single
> > ACL (just
> > > to understand the logic):
> > >
> > > cache_peer peerG1.com parent 40001 0 no-query no-digest
> > name=peerG1
> > >
> > > external_acl_type ext_proxy_g1_type %LOGIN %DST
> > > /usr/local/bin/g1.py (this will answer "OK" or "ERR",
> > depends if
> > > user consists in DB)
> > >
> > > acl proxy_g1_ext_acl ext_proxy_g1_type annotate_transaction
> > > proxy=g1 (If I right understood here is a key point of how
> > to add
> > > the tag to transaction related with user)
> > > acl proxy_peerG1_acl note proxy g1 (here we create the ACL
> > based
> > > on the tag and this is fast ACL yet and we should to use it in
> > > cache_peer_access)
> > >
> > >
> > > http_access deny proxy_g1_ext_acl !all
> > > ......<others http access rules>
> > >
> > >
> > > cache_peer_access peerG1 allow proxy_peerG1_acl
> > > cache_peer_access peerG1 deny all
> > >
> > > Is that correct ?
> > >
> > > вт, 18 апр. 2023 г. в 23:44, Alex Rousskov
> > > <rousskov at measurement-factory.com
> > <mailto:rousskov at measurement-factory.com>
> > > <mailto:rousskov at measurement-factory.com
> > <mailto:rousskov at measurement-factory.com>>>:
> > >
> > > On 4/18/23 11:41, Alexeyяр Gruzdov wrote:
> > >
> > > > Could you explain me how the annotation transaction
> > works and
> > > how it
> > > > related to acl that I could to use with cache_peers
> > >
> > > Transactions have a (possibly empty) set of name=value
> > annotations.
> > >
> > > During Squid configuration time, Squid parses all ACL
> > > declarations in
> > > your configuration file. When Squid parses an
> > > annotation_transaction ACL
> > > declaration, Squid remembers what transaction annotation
> > to add
> > > in the
> > > future, [every time] when that ACL is evaluated (e.g.,
> > used in
> > > http_access rule that Squid reaches during transaction
> > processing).
> > >
> > > When evaluated, an "annotation_transaction" ACL simply
> > adds the
> > > previously configured annotation to the current
> > transaction and
> > > returns
> > > a "yes, this transaction matches" result.
> > >
> > > When evaluated, a "note" ACL returns a "yes, this
> transaction
> > > matches"
> > > result if and only if the current transaction already has
> the
> > > matching
> > > annotation. This ACL does not modify the set of
> transaction
> > > annotations.
> > >
> > > The combination of annotate_transaction and note ACLs
> > allows you to
> > > annotate a transaction at one time and check previously
> set
> > > transaction
> > > annotations at another time. The timing and meaning of
> those
> > > annotations
> > > are up to you.
> > >
> > >
> > > > ok! Lets look to my case example:
> > >
> > > > cache_peer peerG1.com parent 40001 0 no-query no-digest
> > > name=peerG1 round-robin
> > >
> > > > cache_peer_access peerG1 allow proxy_peerG1_acl
> > > > cache_peer_access peerG1 allow proxy_all_acl
> > > > cache_peer_access peerG1 deny all
> > >
> > > > acl proxy_peerG1_acl proxy_auth "../users.peerG1.txt"
> > > > acl proxy_all_acl proxy_auth "../users.all.txt"
> > >
> > > [ I added the missing "acl " directive to the above ACL
> > > declarations and
> > > stripped rules for two out of three cache_peers ]
> > >
> > > As you know, the above cache_peer_access configuration is
> not
> > > supported
> > > because it uses "slow" proxy_auth ACLs in
> cache_peer_access
> > > directives
> > > that only support "fast" ACLs. It does not matter (to me),
> > > whether the
> > > above appears to "work" in some environments. YMMV.
> > >
> > > To fix this problem, we can use http_access rules to
> > essentially
> > > remember proxy_auth evaluation results (at http_access
> > > evaluation time)
> > > as transaction annotations. Here is an untested sketch
> that
> > > omits other
> > > (important but irrelevant here) http_access rules and
> assumes
> > > that these
> > > sketched http_access rules _are_ evaluated:
> > >
> > > # if proxy_peerG1_acl matches, evaluate
> mark_for_peerG1
> > > http_access deny proxy_peerG1_acl mark_for_peerG1 !all
> > >
> > > # if proxy_all_acl matches, evaluate
> mark_for_all_peers
> > > http_access deny proxy_all_acl mark_for_all_peers !all
> > >
> > >
> > > Now we can use those remembered proxy_... acl evaluation
> > results
> > > (i.e.
> > > we can check for the matching annotations) in
> > cache_peer_access
> > > rules:
> > >
> > > cache_peer_access peerG1 allow marked_for_peerG1
> > > cache_peer_access peerG1 allow marked_for_all_peers
> > > cache_peer_access peerG1 deny all
> > >
> > >
> > > where the new ACLs mentioned above are declared along
> > these lines:
> > >
> > > acl mark_for_peerG1 annotate_transaction for_peer_=G1
> > > acl mark_for_all_peers annotate_transaction
> > for_all_peers_=true
> > >
> > > acl marked_for_peerG1 note for_peer_ G1
> > > acl marked_for_all_peers note for_all_peers_ true
> > >
> > > This can probably be simplified further by using
> > for_peer_=ALL
> > > instead
> > > of for_all_peers_=true annotation, but I wanted to
> > preserve the
> > > symmetry
> > > with your original configuration.
> > >
> > >
> > > > And these all works like I need, But - once I am
> > changing a
> > > list of
> > > > users (add or remove) - I need to use "squid -k
> > > reconfigure"...... but
> > > > of course better to go without this reconfigure
> > >
> > > One can avoid reconfiguration using an external ACL
> > script that
> > > gives
> > > Squid the right for_peer_=... annotations (instead of
> using
> > > "constant"
> > > or "hard-coded" annotate_transaction ACLs to store the
> same
> > > annotations).
> > >
> > > However, it may be better to make the above sketch to work
> > > _before_ you
> > > replace mark_for_peerG1 ACLs/rules with an external
> > > mark_for_the_right_peer ACL.
> > >
> > >
> > > HTH,
> > >
> > > Alex.
> > > P.S. This thread continues the discussion started at
> > > https://bugs.squid-cache.org/show_bug.cgi?id=5268
> > <https://bugs.squid-cache.org/show_bug.cgi?id=5268>
> > > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> > <https://bugs.squid-cache.org/show_bug.cgi?id=5268>>
> > >
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> > <mailto:squid-users at lists.squid-cache.org>
> > > <mailto:squid-users at lists.squid-cache.org
> > <mailto:squid-users at lists.squid-cache.org>>
> > > http://lists.squid-cache.org/listinfo/squid-users
> > <http://lists.squid-cache.org/listinfo/squid-users>
> > > <http://lists.squid-cache.org/listinfo/squid-users
> > <http://lists.squid-cache.org/listinfo/squid-users>>
> > >
> > >
> > >
> > > --
> > > С уважением к Вам
> > > Алексей
> > > +79043828661
> > > 620000 г.Екатеринбург 2022
> > >
> > >
> > >
> > > --
> > > С уважением к Вам
> > > Алексей
> > > +79043828661
> > > 620000 г.Екатеринбург 2022
> > >
> >
> >
> >
> > --
> > С уважением к Вам
> > Алексей
> > +79043828661
> > 620000 г.Екатеринбург 2022
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20230420/3233f5cf/attachment-0001.htm>
More information about the squid-users
mailing list