[squid-users] Is Squid can shutdown unused idle redirector's children?

[Marcus Kool]

On 02/16/2015 08:00 PM, Eliezer Croitoru wrote:
> Hey Yuri,
>
> OK I have seen something...
> Now we might need also the virtual memory which might be vsz.
> And the cachemgr output is not from squidview..
> The last image I have seen from cachemgr was much helpful(with 10 helpers).
>
>  From what I have seen until now squidGuard uses about 13 MB of ram constantly.
> If this is what it costs to run a squidGuard helper you should consider that it has some peak usage time and lower bound usage time and you will need to calculate how much helpers you need to provide
> smooth surfing on peak usage time.
> You should consider this peak usage as the required memory from squidGuard for smooth operation.
> Since not you and not will be able to change the basic memory footprint(required) for squidGuard you will need to somehow decide if you require a much efficient software or to adjust your system
> accordingly.
>
> I would agree that 100 * 13 MB of ram means about 1.3 GB of ram usage and might not be acceptable for a "simple" filtering mechanism.

It is because squidGuard uses Berkeley DB which uses a default cache size of 256 KB for each URL table.
And each squidGuard process does this.  And of course the OS keeps a full copy in the file system buffer cache.
So with 100 redirectors and N URL tables, N * 100 * 256K is used by buffers for Berkeley DB alone.

> The main reasons for the unknown need for 100 helpers might be since it was not designed to be used this way.

squidGuard does not support the Squid feature 'concurrency' for url_rewrite_children.  ufdbGuard does.
With concurrency, latency goes down and the number of processes can also be reduced.

> I must say that squidGuard is a very good piece of software but it lacks couple things which might make it un-usable for some if not many systems and maybe including yours.
> Indeed in your case there is a chance that if you will even install a full blown DB that will be stored all in ram and you will write a helper that will mimic squidguard functionality\logic with
> concurrency support you will get much less memory consumption with much more efficient request handling.
>
> It is much smarter to write a helper which will have a overall efficiency mark on it then to add into squid something that might not be even needed in the first place.

ufdbGuard has all that you need.  It holds one copy of the database in memory without allocating extra memory.
The ufdbGuard URL redirectors are leightweight processes using very little memory and
the redirectors support concurrency so less are required.

Off list, Yuri asked help for compilation on Solaris and I made a fix for ufdbGuard which is availabe for Yuri.
If there are no further complaints from Yuri, I will release the patch in one or two days.

Marcus

> For now before writing any helper etc I will want to see the cachemgr interface output for the "redirector" option in two cases:
> - startup
> - after midnight when there are lots of helpers running under squid
>
> You can see that squid since version 3.2 squid offers access to the squid cache manager interface using a simple url such as:
> "http://filter:3128/squid-internal-mgr/redirector"
> (using the visible_hostname and the forward proxy port of squid)
>
> All The Bests,
> Eliezer
>
> On 16/02/2015 21:56, Yuri Voinov wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Now:
>>
>> 2015/02/16 23:10:23 kid1|   store_swap_size = 29826351.50 KB
>> 2015/02/16 23:10:24 kid1| storeLateRelease: released 0 objects
>> 2015/02/16 23:15:01 kid1| Starting new redirector helpers...
>> 2015/02/16 23:15:01 kid1| helperOpenServers: Starting 1/100
>> 'squidGuard' processes
>> 2015/02/17 01:40:15 kid1| Starting new redirector helpers...
>> 2015/02/17 01:40:15 kid1| helperOpenServers: Starting 1/100
>> 'squidGuard' processes
>> ^Croot @ cthulhu / # ps -e -o user,pid,rss,comm
>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users