[squid-users] Help: squid restarts and squidGuard die

[Amos Jeffries]

On 21/09/18 3:46 AM, Marcus Kool wrote:
> 
> On 20/09/18 08:46, Amos Jeffries wrote:
>> On 19/09/18 11:49 PM, Marcus Kool wrote:
>>>
>>> On 18/09/18 23:03, Amos Jeffries wrote:
>>>> On 19/09/18 1:54 AM, neok wrote:
>>>>> Thank you very much Amos for putting me in the right direction.
>>>>> I successfully carried out the modifications you indicated to me.
>>>>> Regarding ufdbGuard, if I understood correctly, what you recommend is
>>>>> to use
>>>>> the ufdbConvertDB tool to convert my blacklists in plain text to the
>>>>> ufdbGuard database format? And then use that/those databases in
>>>>> normal squid
>>>>> ACL's?
>>>>
>>>> No, ufdbguard is a fork of SquidGuard that can be used as a drop-in
>>>> replacement which works better while you improve your config.
>>>>
>>>> You should work towards less complexity. Squid / squid.conf is where
>>>> HTTP access control takes place. The helper is about re-writing the URL
>>>> (only) - which is a complex and destructive process.
>>>
>>> ufdbGuard is a simple tool that has the same syntax in its configuration
>>> file as squidGuard has.
>>> It is far from complex, has a great Reference Manual, exmaple config
>>> file and a responsive support desk.
>>> Amos, I have never seen you calling a URL writer being a complex and
>>> destructive process.  What do you mean?
>>
>> Re-writing requires Squid to:
>>   * fork external helpers, and
>>   * maintain queues of lookups to those helpers, and
>>   * maintain cache of helper responses, and
>>   * maintain a whole extra copy of HTTP-request state, and
>>   * copy some (not all) of that state info between the two "client"
>> requests.
>>
>>   ... lots of complexity, memory, CPU time, traffic latency, etc.
> 
> Squid itself is complex and for any feature of Squid one can make a list
> like above to say that it is complex.
> The fact that one can make such a list does not mean much to me.
> One can make the same or a similar list for external acl helpers and
> even native acls.
> 
>> Also when used for access control (re-write to an "error" URL) the
>> re-write helper needs extra complexity in itself to act as the altered
>> origin server for error pages, or have some fourth-party web server.
> 
> Squid cannot do everything that a URL writer, and specifically
> ufdbGuard, can.
> For example, Squid must restart and break all open connections when a
> tiny detail of the configuration changes.  With ufdbGuard this does not
> happen.


Squid does not close or break any client connections when reconfigured.
Squid pauses active transactions, reconfigures then continues with the
new config.

Are you perhapse mistaking the fact that Squid shuts down the
*rewriters* on reconfigure for a full Squid shutdown?

(hmm, there is another downside to placing all the access control in a
helper - waiting for the helpers to restart on config changes. Though as
you say ufdbguard does it efficiently, others do not).



> ufdbGuard supports dynamic lists of users, domains and source ip
> addresses which are updated every X minutes without any service
> interruption.

So does Squid, via external ACL and/or authentication.


> When other parameters change, ufdbGuard resets itself with zero service
> interruption for Squid and its users.

This is not always true. If the helper pauses even for some milliseconds
it is holding up Squid and clients. Particularly if it is a bottleneck
process like URL-rewrite interface where the helper lookup queue limits
total traffic capacity of the entire proxy.

I think you mean that the helper has threading to do a load in the
background and swap in the config. Correct?

Squid is working (very slowly) towards that model and the SMP features
already reconfigure one worker at a time sequentially so effectively
there should always be a helper with either old or new config answering
incoming traffic while one "resets itself".


> ufdbGuard can decide to probe a site to make a decision, and hence
> detect Skype, Teamviewer and other types of sites that an admin might
> want to block.  Squid cannot.

Squid can, via external ACL. IIRC, Eliezer wrote an ICAP system that did
that too.

Also, the URL-rewrite helper cannot do anything if Squid cannot pass it
a URL. By nature of what the interface is designed to do.


> ufdbGuard can decide to do a lookup of a reverse IP lookup to make a
> decision.  Squid cannot.

Squid can via external ACL.

We have not had much (any?) requests for an ACL doing that. Patches welcome.


> ufdbGuard supports complex time restrictions for access. Squid support
> simple time restrictions.

Such as?

Squid supports complex time points and/or ranges. The time ACL is a
bitmap extending at 1 second intervals across an entire week. Further
extension is done with external ACL, note ACL and/or allof ACL.


> ufdbGuard supports flat file domain/url lists and a commercial URL
> database.  Squid does not.
> And the list goes on.

I am still looking for a feature Squid does not actually support in one
way or another.

As you can see from Flashdown posts "external ACL" can mean SquidGuard /
ufdbguard running on that other interface. So really *anything* they can
do so can Squid external ACL - if not one of the other mechanisms.
At no point is the URL-rewrite API _necessary_ for access control in a
modern Squid.


> 
> So when you state on the mailing list that users should unconditionally
> stop using a URL writer in favor of using Squid acls, you may be causing
> troubles for admins who do not know the implications of your advice.


Understood. I shall try harder to remember the disclaimers usually
added. Thank you for pointing out the omission.


> 
>>> URL rewriters have been used for decades for HTTP access control but you
>>> state "squid.conf is where HTTP access control takes place".
>>
>> Once upon a time, back at the dawn of the WWW (before the 1990s) Squid
>> lacked external_acl_type and modular ACLs.
>>
>> That persisted for the first decade or so of Squid's life, with only the
>> re-write API for admin to use for complicated permissions.
>>
>> Then one day about 2 decades or so ago, external ACL was added and the
>> ACLs were also made much easier to implement and plug in new checks.
>> Today we have hundreds of native ACLs and even a selection of custom ACL
>> helpers. Making the need for these abuses of the poor re-writers.
>>
>> Old habits and online tutorials however are hard to get rid of.
> 
> If you want to get rid of habits that in your view are old/obsolete,
> then why not start a discussion?

I have. This appears to be the latest one.


> And in the event that at the end of the discussion, the decision is made
> that a particular interface should be removed, why not phase it out ?

It still has uses as a URL-rewrite/redirect interface for actions not
related directly to access control.

> 
>>> Are you saying that you want it is the _only_ place for HTTP access
>>> control?
>>
>>
>> I'm saying the purpose of the url_rewrite_* API in Squid is to tell
>> Squid whether the URL (only) needs some mangling in order for the
>> server/origin to understand it.
>>   It can re-write transparently with all the problems that causes to
>> security scopes and URL sync between the endpoints. Or redirect the
>> client to the "correct" URL.
>>
>>
>> The Squid http_access and similar *access controls* are the place for
>> access control - hint is in the naming. With external ACL type for
>> anything Squid does not support natively or well. As Flashdown mentioned
>> even calls to SquidGuard etc. can be wrapped and used as external ACLs.
> 
> Wrapping and externals ACLs adds the same complexity, memory, CPU time,
> traffic latency, etc that you use as an argument against a URL writer.
> Is it only because of the name, 'external acl helper' vs 'url rewriter
> helper', that you dislike the url rewriter?


The external ACL usage is better because it is run at times when Squid
is making decisions about access to the proxy and/or features supplied
by the proxy. The admin can decide exactly when in Squid processing it
is tested - limiting the delays and reducing the lookups to only when
necessary.
 eg. you cannot decide whether a transaction is allowed to be sent to
ICAP or not with a re-writer. Or decide whether ssl_bump is going to
splice vs bump with a URL re-writer.


URL-rewrite API requires access control to *prevent* it being used, and
runs *after* access control - so Squid has to actively allow the traffic
through its own access controls first in order to use the URL-rewrite
API. This leads to a lot of the "http_access allow all" type config issues.


Amos