[squid-users] tuning squid memory (aka avoiding the reaper)

[Alex Rousskov]

On 10/25/2017 10:41 AM, Aaron Turner wrote:
> More testing.  This time with 4.0.21.  Disabled all caching, only
> enabled ssl bumping.  Same config as last time.  Still leaking memory.
> I took two snapshots of info & mem usage and honestly I don't see a
> smoking gun pointing to why my squid processes were getting as large
> as 1.4GB.
> 
> I've attached the two files incase someone with more experience can
> find something useful.

I do not have the time to study your snapshots, unfortunately, but I do
continue to recommend that you take a lot more than two snapshots (e.g.
one snapshot every 10 minutes over a few hours of steady load) and then
either find lines with an increasing counter of alive objects OR confirm
that those lines do not exist.

A Perl script at [1] implements the above analysis, but it has not been
updated for a few years, may not work "as is" with the current mgr:mem
output format, and may need additional tuning to work well in your
specific case, so you may better off starting from scratch.

  [1] http://www.measurement-factory.com/tmp/attachments/memdiff.pl


Cheers,

Alex.


> On Mon, Oct 9, 2017 at 5:04 PM, Aaron Turner <synfinatic at gmail.com> wrote:
>> So more testing.  I haven't found the line in the info:mem logs which
>> is the red flag, but additional testing proves that the memleak has
>> something to do with ssl bumping.  Once I turn that off, the memory
>> leaks stop.
>>
>> this was the ssl related config options:
>>
>> http_port 10.0.0.1:3128 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=400MB cert=/etc/squid/ssl_cert/myCA.pem
>> sslflags=NO_DEFAULT_CA
>> http_port localhost:3128
>> ssl_bump bump all
>>
>> sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
>> sslcrtd_children 32 startup=2 idle=2
>> sslproxy_session_cache_size 100 MB
>> sslproxy_cert_error allow all
>> sslproxy_flags DONT_VERIFY_PEER
>>
>> --
>> Aaron Turner
>> https://synfin.net/         Twitter: @synfinatic
>> My father once told me that respect for the truth comes close to being
>> the basis for all morality.  "Something cannot emerge from nothing,"
>> he said.  This is profound thinking if you understand how unstable
>> "the truth" can be.  -- Frank Herbert, Dune
>>
>>
>> On Wed, Oct 4, 2017 at 10:53 AM, Alex Rousskov
>> <rousskov at measurement-factory.com> wrote:
>>> On 10/02/2017 09:37 PM, Aaron Turner wrote:
>>>> So it's leaking memory and not tracking it?
>>>
>>>
>>> That combination (or, to be more precise, its implication) is possible
>>> but relatively unlikely in your specific case -- when GBs are leaked,
>>> there is usually something tracked related to those GBs. Please note
>>> that what Squid _tracks_ may not amount to GBs of RAM! For example,
>>> Squid can track a tiny object that is included in every large untracked
>>> leaked object.
>>>
>>> A frequent leak often manifests itself in mgr:mem snapshots as a nearly
>>> always increasing counter of alive associated objects. If you take one
>>> snapshot every 30 minutes or so, then you may be able to identify
>>> suspects by comparing same-object alive counters across 5-10 snapshots.
>>> Sorry, I do not have the time to do that for the snapshots you have
>>> shared (and you probably need a different collection of snapshots to
>>> make this search more productive).
>>>
>>> Alex.
>>>
>>>
>>>> On Mon, Oct 2, 2017 at 8:25 PM, Amos Jeffries <squid3 at treenet.co.nz> wrote:
>>>>> On 03/10/17 04:39, Aaron Turner wrote:
>>>>>>
>>>>>> Anyone see anything useful?
>>>>>
>>>>>
>>>>> The numbers in those reports all seem reasonable to me. Nothing is showing
>>>>> up with GB of RAM used.