[squid-users] Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

[Amos Jeffries]

On 19/10/17 03:33, Francisco Amaro wrote:
> Hi all,
> 
> So, we've been (slowly) investigating this issue these past days, and 
> have reached some conclusions.
> 
> a) First, and sorry for not being clear on that, this only happens on 
> "squid reload", not on full stop/start cycles.
> A full start cycle does not give this error, that's why we are pretty 
> sure the issue is not a bungled ACL, per se.

That is unfortunately not a guarantee of success. squid.conf and files 
it refers to can be changed while Squid is running and have no effect 
until next reload happens.

Also what system infrastructure is running this "squid reload" command?

The correct commands to reload squid.conf are "squid -k reconfigure" 
[just reload] or "squid -k check" [validate then reload *if* it passes]. 
Any other system scripts or processes (eg systemd or openrc, or upstart) 
trying to manage Squid may be adding bugs of its own.


> 
> b) Besides that, it only happens when the proxy has some load on it, we 
> have been unable to reproduce this on
> our test machines, since they do not have a significant load. We've been 
> postponing our ACL's changes for
> early morning/lunch-hour these past days, and we didn't have had a FATAL 
> error since last week.
> 
> c) Although we don't have non US-ASCII on the ACL's themselves, we have 
> a lot of them on comments, so
> I've stripped down all the comments of the ACL's files and tried that on 
> our test bed, it didn't generare any
> kind of errors, but since they do not have some load on it, we are not 
> sure if it was the comments stripping
> or the lack of load on the server...

Filtering the config down to 7-bit / US-ASCII characters is a very good 
idea. We have done a bunch of work making Squid handle UTF-8, but the 
config file is one are where that is far from complete so the behaviour 
is unpredictable.

I do know that a Unicode character at the end of a line or Windows / 
MacOS line terminators have been known to cause admin a lot of headaches.


> 
> So, we are now searching for an "easy" way of generating traffic on a 
> squid server, so we can, at least, be
> able to reproduce the error, so we can try the different solutions....
> 


The Squid dev team tend to use Apache Bench ("ab" tool) to generate lots 
of simple transactions very fast if the same URL repeatedly is 
sufficient for the test or caching is disabled.
   Or a tool called Polygraph if the URLs have to be variable and 
stressing all sorts of cache and post-cache logic.

I believe there are also tools that can take a feed of web traffic and 
replay it. But I have not had to use any of those yet on Squid.

Amos