[squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64

[Doug Sampson]

> On 26/11/2014 8:59 a.m., Doug Sampson wrote:
> >
> > Thanks, Amos, for your pointers.
> >
> > I've commented out all the fresh_patterns lines appearing above
> > the last two lines.
> >
> > I also have dropped diskd in favor of using aufs exclusively,
> > taking out the min-size parameter. I've commented out the
> > diskd_program support option. In the previous version of squid
> > (2.7) I had split the cache_dir into two types with great success
> > using coss and aufs. Previously I had only aufs and performance
> > wasn't where I wanted it. Apparently coss is no longer supported in
> > the 3.x version of squid atop FreeBSD.
> 
> COSS has been replaced with Rock storage type in Squid-3. They should
> be used in roughly similar ways in terms of traffic optimization.
> 
> >
> > The pathname for the cache swap logs have been fixed. Apparently
> > this came from a squid.conf example that I copied in parts. Would
> > this be the reason why we are seeing the error messages in
> > /var/log/messages regarding swapping mentioned in my original
> > post?
> 
> No. I think that is coming out of the OS kernel memory management. It
> uses the term "swap" as well in regards to disk backed virtual memory.
> 
> If your system is "swapping" (using that disk backed "swap memory")
> while running Squid then you will get terrible performance as a matter
> of course since the Squid cache index and cache_meme is often very
> large in RAM and accessed often.
> 
> >
> > The hierarchy_stoplist line has been stripped out as you say it is
> > deprecated.
> >
> > The mem .TSV file is attached herewith.
> >
> > Currently I have the cache_dir located on the OS disk and all of
> > the cache logging files on a second drive. Is this the optimal
> > setup of cache-dir and logs?
> 
> I would do it the other way around. Logs are appended with a small
> amount of data each transaction, whereas the main cache_dir has a
> relativey large % of the bandwidth throughput being written out to it
> constantly (less % in recent Squid, but still a lot). The dik most
> likely to die early is the one holding cache_dir.
> 
I'm still running into the issue of being out of available space in the swapfile on my system. I've attached another TSV file indicating the various type of memory being in use and whatnot. Is there anything in there that screams out?

Amos, you said earlier that it was the OS system that needed to be tuned. Are there any references to where I can fine-tune it for Squid usage? I looked here http://oss.org.cn/man/newsoft/squid/Squid_FAQ/FAQ-8.html#how-much-ram and I'm unable to figure out a way to decrease the amount of memory that I could use. I tried limiting cache_mem to 1344 MB from some higher value but that didn't work. What are some of the methods that FreeBSD 10.0 users using to limit the use of memory that Squid uses?

Sort of fixing the memory leaks, it looks like I need to consider the possibility of restarting the Squid service on a regular basis (i.e. at least once a week) in order to enable Squid to perform at an acceptable level and to avoid clogging /var/log/messages with such messages as follows:

<...>
+swap_pager_getswapspace(15): failed
+swap_pager_getswapspace(2): failed
+swap_pager_getswapspace(16): failed
+swap_pager_getswapspace(16): failed
<...>

Is this a common practice among squid admins to restart squid periodically?

Thank you!

~Doug
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: squid-internal-mgr_MEM_20141205.tsv
Type: text/tab-separated-values
Size: 15791 bytes
Desc: squid-internal-mgr_MEM_20141205.tsv
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20141205/89b5c3ee/attachment-0001.tsv>