[squid-users] SQUID memory error after vm.swappines changed from 60 to 10

Thu Nov 9 13:04:41 UTC 2017

On Wed, Nov 8, 2017 at 3:48 PM, Marcus Kool <marcus.kool at urlfilterdb.com> wrote:
>
> On 08/11/17 11:36, Bike dernikov1 wrote:
>>
>> Hi,
>>
>> We stumbled on ufdbGuard, but licence/price was problem, we didn't
>> read documentation carefully.
>
> yes, ufdbguard is free.
>
>> We will definitely try ufdbGuard, but we are now in process of moving
>> squid/squidguard to production, so we can't test on production (angry
>> users, Internet must work :)).
>>
>> Memory compsumption:squid use largest part of memory  (12GB now,
>> second proces use 300MB memory), 14GB used by all process. So squid
>> use over 80% of total used memory.
>> So no there are not any problematic process. But we changed swappiness
>> settings.
>
> Did you monitor Squid for growth (it can start with 12 GB and grow slowly) ?

Yes we are monitoring continuosly.
Now:
Output from free -m.

           total       used    free   shared  buff/cache  available
Mem:  24101     20507  256    146      3337         3034
Swap: 24561      5040   19521

vm.swappiness=40

Memory by process:
squid  Virt       RES   SHR  MEM%
           22,9G  18.7   8164   79,6
squidguard two process  300MB boths,.

CPU 0.33 0.37 0.43

> Squid cannot fork and higher swappiness increases the amount of memory that
> the OS can use to copy processes.
> It makes me think that you have the memory overcommit set to 2 (no
> overcommit).
> What is the output of the following command ?
>    sysctl  -a | grep overcommit

Command output:

vm.nr_overcommit_hugepages = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50

cat /proc/sys/vm/overcommit_memory
0

>> Advice for some settings:
>> We have absolute max peak of  2500 users which user squid (of 2800),
>> what are recomended settings for:
>> negotiate_kerberos_children start/idle
>> squidguard helpers.
>
>
> I have little experience with kerberos, but most likely this is not the
> issue.
> When Squid cannot fork the helpers, helper settings do not matter much.

> For 2500 users you probably need 32-64 squidguard helpers.

Can you confirm: For 2500 users:

url_rewrite children X (squidguard)  32-64 will be ok ? We have set
much larger number.

For  helper:
negotitate_kerberos_auth

auth_param negotiate children X startup Y idle Z. What X, Y, Z are
best for our user number ?

We disabled kerberos replay cache because of disk performance (4 SAS
DISK  15K, RAID 10) (iowait jumped high, and CPU load jumped to min
40 max 200).
We don't use disk caching.

Thanks for help,

> Marcus
>
>
>> Thanks for help,
>>
>> On Wed, Nov 8, 2017 at 10:53 AM, Marcus Kool
>> <marcus.kool at urlfilterdb.com> wrote:
>>>
>>> There is definitely a problem with available memory because Squid cannot
>>> fork.
>>> So start with looking at how much memory Squid and its helpers use.
>>> Do do have other processes on this system that consume a lot of memory ?
>>>
>>> Also note that ufdbGuard uses less memory that squidGuard.
>>> If there are 30 helpers squidguard uses 300% more memory than ufdbGuard.
>>>
>>> Look at the wiki for more information about memory usage:
>>> https://wiki.squid-cache.org/SquidFaq/SquidMemory   (currently has an
>>> expired certificate but it is safe to go ahead)
>>>
>>> Marcus
>>>
>>>
>>>
>>> On 08/11/17 07:26, Bike dernikov1 wrote:
>>>>
>>>>
>>>> Hi, I hope that someone can explain what happened, why squid stopped
>>>> working.
>>>> The problem is related to  memory/swap handling.
>>>>
>>>> After we changed vm.swappiness parameter from 60 to 10 (tuning
>>>> attempt, to lower a disk usage, because we have only 4 disks in a
>>>> RAID10, so disk subsystem  is a weak link), we got a lot of errors in
>>>> cache.log.
>>>> The problems started after scheduled logrotate after  2AM.
>>>> Squid ran out of memory, auth helpers stopped working.
>>>> It's weird because we didn't disable swap, but behavior is like we did.
>>>> After an error, we increased parameter from 10 to 40.
>>>>
>>>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT
>>>> cores).
>>>> We have 2800 users, using  kerberos authentication, squidguard for
>>>> filtering, ldap authorization.
>>>> When problem appeared memory was still 3GB free (free column), ram
>>>> (caching) was filled to 15GB, so 21 GB ram filled, 3GB free.
>>>>
>>>> Thanks for help,
>>>>
>>>>
>>>> errors from cache.log.
>>>>
>>>> 2017/11/08 02:55:27| Set Current Directory to /var/log/squid/
>>>> 2017/11/08 02:55:27 kid1| storeDirWriteCleanLogs: Starting...
>>>> 2017/11/08 02:55:27 kid1|   Finished.  Wrote 0 entries.
>>>> 2017/11/08 02:55:27 kid1|   Took 0.00 seconds (  0.00 entries/sec).
>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>> daemon:/var/log/squid/access.log
>>>> 2017/11/08 02:55:27 kid1| logfileRotate:
>>>> daemon:/var/log/squid/access.log
>>>> 2017/11/08 02:55:28 kid1| Pinger socket opened on FD 30
>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 1/1000
>>>> 'squidGuard' processes
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run '/usr/bin/squidGuard'
>>>> process.
>>>> 2017/11/08 02:55:28 kid1| helperOpenServers: Starting 300/3000
>>>> 'negotiate_kerberos_auth' processes
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>> 2017/11/08 02:55:28 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>>> 2017/11/08 02:55:28 kid1| WARNING: Cannot run
>>>> '/usr/lib/squid/negotiate_kerberos_auth' process.
>>>>
>>>> external ACL 'memberof' queue overload. Using stale result.
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users