[squid-dev] Squid 4.13: too much memory used for ACL url_regex when big list file used

[Amos Jeffries]

On 17/08/21 5:45 am, Meridoff wrote:
> Hello, I have simplest squid config with such acl:
> 
> acl a1 url_regex "/tmp/urls.txt"
> 
> In /tmp/urls.txt there are about 220 000 URL regexps, most of them in 
> such form (example):
> ^(https?|ftp)://([a-z0-9.-]+\.)?nicebox\.pro(/.*)?$
> OR
> ^(https?|ftp)://order-yudobashi-com\.363q1\.bar/
> 
> There are a lot of memory used by squid for such configuration: about 
> 2GB. Without this command: about 30MB used.

Are those numbers without traffic going through?
ie. just Squid loading the configuration then does nothing.

> 
> So aprxm. 10KB for 1 regexp. Is it normal? I think it is too big..
> 

regex rules get aggregated into longer compound lines, then compiled 
into a binary form for the regex library to handle. It is not easy to 
tell how the compiled form will compare in size to the original strings.


> I think that such big consumption is because of regexps , for simple 
> strings (for example, domains or IPs) consumption will be less.
> 

Sure. "Simple strings" will be not much larger than the size of the file 
input - though there is some extra memory used for indexing etc.


> How I can decrease memory consumption for such big regexp lists in ACLs 
> ? May be some fix in squid or some fix in regexp lists ?
> 

First thing to do is figure out where the memory is being consumed.

This command:
  squidclient mgr:mem

Will produce a spreadhsheet in TSV format of the memory allocations your 
Squid has. Look through that for the big memory spenders, with an eye on 
the regex one(s) you are suspicious of.


Amos