[squid-users] tuning squid memory (aka avoiding the reaper)

[Aaron Turner]

One more update before I restart squid:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3188 squid     20   0 4821844 4.337g 1.008g R  53.0 30.4 349:31.24 squid
 3187 squid     20   0 3539696 3.153g 1.008g R  31.9 22.1 259:15.31 squid
 3190 squid     20   0 3198228 2.834g 1.008g S  29.2 19.8 230:23.30 squid
 3189 squid     20   0 3033460 2.680g 1.008g R  27.0 18.8 226:17.63 squid

https://synfin.net/misc/mgr_mem_1000.txt

--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune


On Fri, Sep 29, 2017 at 9:45 AM, Aaron Turner <synfinatic at gmail.com> wrote:
> So this is smelling like a mem leak to me.  First after running for a few hours:
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>  3188 squid     20   0 3586264 3.175g 1.007g R  63.4 22.2 162:59.38 squid
>  3187 squid     20   0 2941332 2.585g 1.005g S  45.5 18.1 129:36.40 squid
>  3190 squid     20   0 2641828 2.304g 1.001g R  41.4 16.1 109:49.08 squid
>  3189 squid     20   0 2524892 2.182g 0.987g S  42.1 15.3 110:30.96 squid
>
> I configured squid w/ 1GB mem cache, no disk cache, ssl bumping and 4
> workers.  Looks like they've all pretty much mapped the 1GB which is
> what I'd expect.  However, resident memory is clearly quite high
> considering my config.
>
> While this was running I was capturing the output of mgr:mem and
> started looking at the numbers.  Now I'm not 100% I understand the
> meanings of all the columns, but I also don't see any indication of
> what is using all that resident memory.
>
> I've grabbed a few of the mgr:mem output spanning the test and
> uploaded them here since I hate sending attachments to lists:
>
> https://synfin.net/misc/watch_share.tar.gz
> --
> Aaron Turner
> https://synfin.net/         Twitter: @synfinatic
> My father once told me that respect for the truth comes close to being
> the basis for all morality.  "Something cannot emerge from nothing,"
> he said.  This is profound thinking if you understand how unstable
> "the truth" can be.  -- Frank Herbert, Dune
>
>
> On Thu, Sep 28, 2017 at 10:05 PM, Amos Jeffries <squid3 at treenet.co.nz> wrote:
>> On 29/09/17 12:35, Aaron Turner wrote:
>>>
>>> Ok, i'll work on that.  One other thing, is that if I let it run long
>>> enough, squid will crash with errors like the following:
>>>
>>> FATAL: Received Bus Error...dying.
>>> 2017/09/28 23:28:09 kid4| Closing HTTP port 10.93.3.4:3128
>>> 2017/09/28 23:28:09 kid4| Closing HTTP port 127.0.0.1:3128
>>> 2017/09/28 23:28:09 kid4| storeDirWriteCleanLogs: Starting...
>>> 2017/09/28 23:28:09 kid4|   Finished.  Wrote 0 entries.
>>> 2017/09/28 23:28:09 kid4|   Took 0.00 seconds (  0.00 entries/sec).
>>> CPU Usage: 0.541 seconds = 0.466 user + 0.075 sys
>>> Maximum Resident Size: 121440 KB
>>> Page faults with physical i/o: 0
>>>
>>> At first I thought the bus error was hardware, but it's happened on
>>> two different EC2 instances now.
>>>
>>
>> Yes "Bus Error" is definitely hardware. The OS kernel had an error loading
>> some data from RAM into the CPU or something along those lines.
>>
>> The only things we can do about it is check that your Squid is up to date
>> with the system environment - eg fairly recent Squid version built with the
>> OS latest compiler version against its current libc or whatever the
>> equivalents of those are for your system. If there are binary level issues
>> with the libc interfaces weird things can happen.
>>
>> Amos