[squid-users] Fwd: cache_peer_access by dynamic ACL
Alex Rousskov
rousskov at measurement-factory.com
Thu Apr 20 18:31:11 UTC 2023
On 4/20/23 13:14, Alexeyяр Gruzdov wrote:
> Tell me please If I right understood I could to get answer like
> "name=value" from my ACL ext script, instead of "OK" or "ERR", right?
Not "instead", but in addition to either OK or ERR.
Unfortunately, our documentation of the external ACL helper protocol has
been hidden in a transition to GitHub Pages (or I cannot find it), but
you can use the old wiki sources at the URL below to learn about that
protocol:
https://github.com/squid-cache/squid-cache.github.io/blob/fa00274/Features/AddonHelpers.md#access-control-acl
General information about from-helper annotations:
https://github.com/squid-cache/squid-cache.github.io/blob/fa00274/Features/AddonHelpers.md#keyvalue-pairs-kv-pairs-format
> And does it means - I could to get answer depends from what users
> authorises in to proxy.
> For example:
>
> If user "Jon" - the my ACL will check the policy in the DB and send
> answer like "proxy=g1" to squid, or if user is "Jack" - answer will be
> like "proxy=all"
Yes, what annotation(s) the helper script sends to Squid (and why) is up
to you. Please note, however, that annotation _names_ not ending with an
underscore (_) are officially reserved for Squid use. I would be
especially worried about using single, simple, generic words like "proxy"!
> and I will have ACL for this
>
> acl proxy_g1_marked.acl note proxy g1
> acl proxy_all_marked.acl note proxy all
>
> will that correct ?
Yes. An no transaction_annotation ACLs.
HTH,
Alex.
> чт, 20 апр. 2023 г. в 19:50, Alex Rousskov
> <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>:
>
> On 4/20/23 10:54, Alexeyяр Gruzdov wrote:
> > The cpu is around 100% even no any requests is going....
>
> Then the problem is most likely in your script. Foe example, the script
> actively doing something while there are no Squid requests to work on
> (instead of blocking while waiting for the next Squid request).
>
>
> > For now I left just one cache_peer in configuration and got the
> new error:
> >
> > commBind Cannot bind socket FD 28 to [::]: (13) Permission denied
>
> Could be one of the misconfigurations discussed at
> https://wiki.squid-cache.org/Features/SmpScale#cannot-bind-socket-fd-nn-to--13-permission-denied <https://wiki.squid-cache.org/Features/SmpScale#cannot-bind-socket-fd-nn-to--13-permission-denied>
>
>
> HTH,
>
> Alex.
>
>
> > чт, 20 апр. 2023 г. в 17:21, Alex Rousskov:
> >
> > On 4/20/23 04:23, Alexeyяр Gruzdov wrote:
> >
> > > cache_peer peerG1.com parent 40001 0 no-query no-digest
> name=peerG1
> > >
> > > external_acl_type ext_proxy_g1_type %LOGIN %DST
> /usr/local/bin/g1.py
> > >
> > > acl proxy_g1_ext_mark_acl ext_proxy_g1_type
> > >
> > > acl proxy_g1_ext_marked_acl annotate_transaction proxy=g1
> > >
> > > acl proxy_peerG1_acl note proxy g1
> > >
> > > http_access deny
> proxy_g1_ext_mark_acl proxy_g1_ext_marked_acl !all
> > > .....
> > > others http_access rules
> > >
> > > And this above works.
> >
> > Glad to hear that. ( If others are going to use the above as
> a guiding
> > example, I would recommend naming these ACLs very
> differently, but that
> > is not important to Squid. )
> >
> >
> > > BUT
> > > I am worried about why this my external script for ACL type
> > loads the
> > > one of core of CPU to 100%.....???
> >
> > External ACL caching aside, the script will be contacted once
> for every
> > Squid transaction. Does your script CPU usage go down to zero
> when
> > there
> > is no traffic? If not, then there is a bug in the script itself.
> >
> > If you use the script from the command line, without Squid,
> does it
> > consume a lot of CPU and/or take a lot of time per fake
> query? You can
> > adjust the script to log the real query (when the script is
> used by
> > Squid), so that you can easily replicate that query when
> running the
> > script without Squid...
> >
> > The cache key in your case is (the expansion of) "%LOGIN
> %DST". It is
> > enabled by default IIRC. Look for "cache" related options at
> > http://www.squid-cache.org/Doc/config/external_acl_type/
> <http://www.squid-cache.org/Doc/config/external_acl_type/>
> > <http://www.squid-cache.org/Doc/config/external_acl_type/
> <http://www.squid-cache.org/Doc/config/external_acl_type/>>
> >
> >
> > > ( I used three of workers in config,
> > > but I can see a six process called like my external helper
> > script, looks
> > > like squid runs x2 process for external ACL )
> >
> > See external_acl_type children-* options:
> > http://www.squid-cache.org/Doc/config/external_acl_type/
> <http://www.squid-cache.org/Doc/config/external_acl_type/>
> > <http://www.squid-cache.org/Doc/config/external_acl_type/
> <http://www.squid-cache.org/Doc/config/external_acl_type/>>
> >
> > In most environments, I recommend setting all three of them
> to the same
> > value. Please note that these options are not SMP-aware
> (yet), so Squid
> > will _not_ divide their values by the number of workers and
> give each
> > worker as many children as you state in squid.conf.
> >
> >
> > > Because if I will put the one more group of users (that
> must to use
> > > another cache_peer ) - I will need to create one more external
> > script
> > > that will making to check an existed users from an
> other DB table
> >
> > Once you get the basic setup above working for one group to your
> > satisfaction, I would recommend migrating from (one script
> and one
> > matching annotate_transaction ACL) per group to a single
> script for all
> > groups. That single external ACL script will send the right
> > annotation(s) to Squid.
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> > > ср, 19 апр. 2023 г. в 22:39, Alex Rousskov:
> > >
> > > On 4/19/23 13:30, Alexeyяр Gruzdov wrote:
> > >
> > > > cache_peer peerG1.com parent 40001 0 no-query no-digest
> > name=peerG1
> > >
> > > > external_acl_type ext_proxy_g1_type %LOGIN %DST
> > /usr/local/bin/g1.py
> > >
> > > > acl proxy_g1_ext_acl ext_proxy_g1_type
> > >
> > > OK. I assume that /usr/local/bin/g1.py will only match
> users that
> > > should
> > > go to cache_peer called peerG1.
> > >
> > >
> > > > acl proxy_g1_ext_acl_mark annotate_transaction
> proxy=g1
> > >
> > > Please note that the name of this annotate_transaction
> ACL --
> > > "proxy_g1_ext_acl_mark" -- implies a relationship to the
> > external ACL
> > > named "proxy_g1_ext_acl", but there is no such
> relationship.
> > Squid does
> > > not care about ACL names, but this naming problem may
> indicate a
> > > misunderstanding. To follow your naming scheme, this ACL
> > should be
> > > called something like "proxy_g1_mark_acl" or
> > "mark_for_proxy_g1_acl".
> > >
> > >
> > > > acl proxy_peerG1_acl note proxy g1
> > >
> > > OK. FWIW, a more consistent ACL name would have been
> > > "proxy_g1_marked_acl" or "marked_for_proxy_g1_acl". Again,
> > Squid does
> > > not really care about these names, so use whatever you
> think is
> > > consistent/meaningful/etc.
> > >
> > >
> > > > http_access deny proxy_g1_ext_acl !all
> > >
> > > This line has no (positive) effect. Squid will
> evaluate the
> > external
> > > ACL, but since the rule, as a whole, will never match
> due to
> > "!all",
> > > and
> > > since the external ACL has no (relevant) side effects, you
> > can just
> > > delete this line from your configuration.
> > >
> > > Needless to say, if you delete this line, then
> > proxy_g1_ext_acl will be
> > > unused, which should tell you that this configuration
> is not
> > doing what
> > > you want. See below for a fix recommendation.
> > >
> > >
> > > > http_access deny proxy_g1_ext_acl_mark !all
> > >
> > > This line will mark _all_ transactions. You only want
> to mark
> > > transactions that also matched proxy_g1_ext_acl. That
> "b only
> > if a"
> > > logic is accomplished by using _both_ ACLs in the same
> rule:
> > >
> > > http_access deny proxy_g1_ext_acl
> proxy_g1_ext_acl_mark !all
> > >
> > > With the above http_access rule (instead of the
> earlier two),
> > Squid
> > > will
> > > evaluate the external ACL, and, if it matches, Squid will
> > also evaluate
> > > the annotation-setting ACL. The whole rule will then be
> > rejected due to
> > > "!all", but not until it annotates the transaction (if the
> > external ACL
> > > matches). Again, in this sketch, we are using this
> rule for its
> > > annotation side effect only.
> > >
> > >
> > > > And this works like I need now....
> > >
> > > AFAICT, if the tests indicate that this configuration
> works,
> > then the
> > > tests are broken. IMHO, you should fix the tests
> (while you
> > have a
> > > broken configuration that can be used to test the
> tests) before
> > > proceeding with the configuration fix.
> > >
> > >
> > > HTH,
> > >
> > > Alex.
> > > P.S. Please keep this email thread on squid-users
> instead of
> > responding
> > > to me personally.
> > >
> > >
> > >
> > >
> > > > ср, 19 апр. 2023 г. в 21:01, Alexeyяр Gruzdov:
> > > >
> > > > so, ok - Lets limit just to one cache peer and
> one single
> > > ACL (just
> > > > to understand the logic):
> > > >
> > > > cache_peer peerG1.com parent 40001 0 no-query
> no-digest
> > > name=peerG1
> > > >
> > > > external_acl_type ext_proxy_g1_type %LOGIN %DST
> > > > /usr/local/bin/g1.py (this will answer "OK"
> or "ERR",
> > > depends if
> > > > user consists in DB)
> > > >
> > > > acl proxy_g1_ext_acl
> > ext_proxy_g1_type annotate_transaction
> > > > proxy=g1 (If I right understood here is a key
> point
> > of how
> > > to add
> > > > the tag to transaction related with user)
> > > > acl proxy_peerG1_acl note proxy g1 (here we
> create
> > the ACL
> > > based
> > > > on the tag and this is fast ACL yet and we
> should to
> > use it in
> > > > cache_peer_access)
> > > >
> > > >
> > > > http_access deny proxy_g1_ext_acl !all
> > > > ......<others http access rules>
> > > >
> > > >
> > > > cache_peer_access peerG1 allow proxy_peerG1_acl
> > > > cache_peer_access peerG1 deny all
> > > >
> > > > Is that correct ?
> > > >
> > > > вт, 18 апр. 2023 г. в 23:44, Alex Rousskov
> > > > <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>
> > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>
> > > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>
> > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>>
> > > > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>
> > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>
> > > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>
> > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>>>>:
> > > >
> > > > On 4/18/23 11:41, Alexeyяр Gruzdov wrote:
> > > >
> > > > > Could you explain me how the annotation
> transaction
> > > works and
> > > > how it
> > > > > related to acl that I could to use with
> cache_peers
> > > >
> > > > Transactions have a (possibly empty) set of
> name=value
> > > annotations.
> > > >
> > > > During Squid configuration time, Squid
> parses all ACL
> > > > declarations in
> > > > your configuration file. When Squid parses an
> > > > annotation_transaction ACL
> > > > declaration, Squid remembers what transaction
> > annotation
> > > to add
> > > > in the
> > > > future, [every time] when that ACL is
> evaluated (e.g.,
> > > used in
> > > > http_access rule that Squid reaches during
> transaction
> > > processing).
> > > >
> > > > When evaluated, an "annotation_transaction"
> ACL simply
> > > adds the
> > > > previously configured annotation to the current
> > > transaction and
> > > > returns
> > > > a "yes, this transaction matches" result.
> > > >
> > > > When evaluated, a "note" ACL returns a
> "yes, this
> > transaction
> > > > matches"
> > > > result if and only if the current transaction
> > already has the
> > > > matching
> > > > annotation. This ACL does not modify the set of
> > transaction
> > > > annotations.
> > > >
> > > > The combination of annotate_transaction and
> note ACLs
> > > allows you to
> > > > annotate a transaction at one time and check
> > previously set
> > > > transaction
> > > > annotations at another time. The timing and
> > meaning of those
> > > > annotations
> > > > are up to you.
> > > >
> > > >
> > > > > ok! Lets look to my case example:
> > > >
> > > > > cache_peer peerG1.com parent 40001 0
> no-query
> > no-digest
> > > > name=peerG1 round-robin
> > > >
> > > > > cache_peer_access peerG1 allow
> proxy_peerG1_acl
> > > > > cache_peer_access peerG1 allow
> proxy_all_acl
> > > > > cache_peer_access peerG1 deny all
> > > >
> > > > > acl proxy_peerG1_acl proxy_auth
> > "../users.peerG1.txt"
> > > > > acl proxy_all_acl proxy_auth
> "../users.all.txt"
> > > >
> > > > [ I added the missing "acl " directive to the
> > above ACL
> > > > declarations and
> > > > stripped rules for two out of three
> cache_peers ]
> > > >
> > > > As you know, the above cache_peer_access
> > configuration is not
> > > > supported
> > > > because it uses "slow" proxy_auth ACLs in
> > cache_peer_access
> > > > directives
> > > > that only support "fast" ACLs. It does not
> matter
> > (to me),
> > > > whether the
> > > > above appears to "work" in some
> environments. YMMV.
> > > >
> > > > To fix this problem, we can use http_access
> rules to
> > > essentially
> > > > remember proxy_auth evaluation results (at
> http_access
> > > > evaluation time)
> > > > as transaction annotations. Here is an untested
> > sketch that
> > > > omits other
> > > > (important but irrelevant here) http_access
> rules
> > and assumes
> > > > that these
> > > > sketched http_access rules _are_ evaluated:
> > > >
> > > > # if proxy_peerG1_acl matches, evaluate
> > mark_for_peerG1
> > > > http_access deny proxy_peerG1_acl
> > mark_for_peerG1 !all
> > > >
> > > > # if proxy_all_acl matches, evaluate
> > mark_for_all_peers
> > > > http_access deny proxy_all_acl
> > mark_for_all_peers !all
> > > >
> > > >
> > > > Now we can use those remembered proxy_... acl
> > evaluation
> > > results
> > > > (i.e.
> > > > we can check for the matching annotations) in
> > > cache_peer_access
> > > > rules:
> > > >
> > > > cache_peer_access peerG1 allow
> marked_for_peerG1
> > > > cache_peer_access peerG1 allow
> > marked_for_all_peers
> > > > cache_peer_access peerG1 deny all
> > > >
> > > >
> > > > where the new ACLs mentioned above are
> declared along
> > > these lines:
> > > >
> > > > acl mark_for_peerG1 annotate_transaction
> > for_peer_=G1
> > > > acl mark_for_all_peers annotate_transaction
> > > for_all_peers_=true
> > > >
> > > > acl marked_for_peerG1 note for_peer_ G1
> > > > acl marked_for_all_peers note
> for_all_peers_ true
> > > >
> > > > This can probably be simplified further by
> using
> > > for_peer_=ALL
> > > > instead
> > > > of for_all_peers_=true annotation, but I
> wanted to
> > > preserve the
> > > > symmetry
> > > > with your original configuration.
> > > >
> > > >
> > > > > And these all works like I need, But -
> once I am
> > > changing a
> > > > list of
> > > > > users (add or remove) - I need to use
> "squid -k
> > > > reconfigure"...... but
> > > > > of course better to go without this
> reconfigure
> > > >
> > > > One can avoid reconfiguration using an
> external ACL
> > > script that
> > > > gives
> > > > Squid the right for_peer_=... annotations
> (instead
> > of using
> > > > "constant"
> > > > or "hard-coded" annotate_transaction ACLs
> to store
> > the same
> > > > annotations).
> > > >
> > > > However, it may be better to make the above
> sketch
> > to work
> > > > _before_ you
> > > > replace mark_for_peerG1 ACLs/rules with an
> external
> > > > mark_for_the_right_peer ACL.
> > > >
> > > >
> > > > HTH,
> > > >
> > > > Alex.
> > > > P.S. This thread continues the discussion
> started at
> > > > https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>
> > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>>
> > > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>
> > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>>>
> > > >
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>
> > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>>
> > > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>
> > <https://bugs.squid-cache.org/show_bug.cgi?id=5268
> <https://bugs.squid-cache.org/show_bug.cgi?id=5268>>>>
> > > >
> > > > _______________________________________________
> > > > squid-users mailing list
> > > > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>>
> > > > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>>>
> > > > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> > > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>>
> > > >
> <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> > > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>>>
> > > >
> > > >
> > > >
> > > > --
> > > > С уважением к Вам
> > > > Алексей
> > > > +79043828661
> > > > 620000 г.Екатеринбург 2022
> > > >
> > > >
> > > >
> > > > --
> > > > С уважением к Вам
> > > > Алексей
> > > > +79043828661
> > > > 620000 г.Екатеринбург 2022
> > > >
> > >
> > >
> > >
> > > --
> > > С уважением к Вам
> > > Алексей
> > > +79043828661
> > > 620000 г.Екатеринбург 2022
> > >
> > >
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
More information about the squid-users
mailing list