<div dir="ltr"><div class="gmail_default" style="font-size:small">We are still chasing this one down but made a major breakthrough. The leak is related to squid in intercept mode + SSL decryption + origin with invalid certs. In our case, the majority of the cases were related to Windows Update and Windows Defender domains, so a stopgap solution is to bypass decryption for these sites (eg, .<a href="http://update.microsoft.com">update.microsoft.com</a>). If you do, don't use dstdomain ACL, as the domain is not available at the time of the checking. Use something like ssl::server_name[_regex].</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Hope this helps!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 27, 2023 at 2:28 PM Gustavo Carvalho <<a href="mailto:gustavocarv4872@gmail.com">gustavocarv4872@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Hamilton, thanks for helping!<br>
<br>
I wish I could provide this log while squid is crashing, but there<br>
have been no incidents since wednesday. From what I've heard, the RAM<br>
on that server's VM has been increased to 32GB.<br>
<br>
Anyway, here is the squidclient mgr:mem log output. I hope it can be helpful.<br>
<br>
On Thu, Jan 26, 2023 at 5:43 PM Hamilton Coutinho<br>
<<a href="mailto:hamilton.coutinho@gmail.com" target="_blank">hamilton.coutinho@gmail.com</a>> wrote:<br>
><br>
> Hi Gustavo,<br>
><br>
> I'm seeing the same thing. I could narrow down (but can't say with 100% confidence) to the code that does certificate verification when configured for SSL decryption. What is the output of squidclient mgr:mem for you? Do you see unexplainably high counts for in-use objects like HttpRequest, PeekingPeerConnector, Comm::Connection, Security::ErrorDetail?<br>
><br>
><br>
> On Thu, Jan 26, 2023 at 12:31 PM Gustavo Carvalho <<a href="mailto:gustavocarv4872@gmail.com" target="_blank">gustavocarv4872@gmail.com</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> I have Squid 5.6 on a FreeBSD 13.1 server with 16GB RAM<br>
>><br>
>> I noticed that squid starts to consume a lot of ram until it starts to<br>
>> consume swap space. When this happens, browsing becomes extremely<br>
>> slow.<br>
>><br>
>> This is happening at least once a week when I have to restart squid to<br>
>> get it back to normal.<br>
>><br>
>> Any ideas?<br>
>><br>
>> ############# Wed Jan 25 08:30:00 -03 2023 #############<br>
>><br>
>> HTTP/1.1 200 OK<br>
>> Server: squid<br>
>> Mime-Version: 1.0<br>
>> Date: Wed, 25 Jan 2023 11:30:00 GMT<br>
>> Content-Type: text/plain;charset=utf-8<br>
>> Expires: Wed, 25 Jan 2023 11:30:00 GMT<br>
>> Last-Modified: Wed, 25 Jan 2023 11:30:00 GMT<br>
>> X-Cache: MISS from xxxx.xxxx.xxxx<br>
>> X-Cache-Lookup: MISS from xxxx.xxxx.xxxx:3128<br>
>> Via: 1.1 xxxx.xxxx.xxxx (squid)<br>
>> Connection: close<br>
>><br>
>> Squid Object Cache: Version 5.6<br>
>> Build Info:<br>
>> Service Name: squid<br>
>> Start Time: Thu, 19 Jan 2023 20:25:17 GMT<br>
>> Current Time: Wed, 25 Jan 2023 11:30:00 GMT<br>
>> Connection information for squid:<br>
>> Number of clients accessing cache: 224<br>
>> Number of HTTP requests received: 7541590<br>
>> Number of ICP messages received: 0<br>
>> Number of ICP messages sent: 0<br>
>> Number of queued ICP replies: 0<br>
>> Number of HTCP messages received: 0<br>
>> Number of HTCP messages sent: 0<br>
>> Request failure ratio: 0.00<br>
>> Average HTTP requests per minute since start: 930.5<br>
>> Average ICP messages per minute since start: 0.0<br>
>> Select loop called: 78733524 times, 6.176 ms avg<br>
>> Cache information for squid:<br>
>> Hits as % of all requests: 5min: 8.4%, 60min: 12.1%<br>
>> Hits as % of bytes sent: 5min: 21.6%, 60min: 14.1%<br>
>> Memory hits as % of hit requests: 5min: 90.8%, 60min: 75.9%<br>
>> Disk hits as % of hit requests: 5min: 4.0%, 60min: 19.7%<br>
>> Storage Swap size: 2829956 KB<br>
>> Storage Swap capacity: 90.0% used, 10.0% free<br>
>> Storage Mem size: 16172 KB<br>
>> Storage Mem capacity: 98.7% used, 1.3% free<br>
>> Mean Object Size: 28.95 KB<br>
>> Requests given to unlinkd: 186982<br>
>> Median Service Times (seconds) 5 min 60 min:<br>
>> HTTP Requests (All): 0.00562 0.01847<br>
>> Cache Misses: 0.15048 0.23230<br>
>> Cache Hits: 0.00000 0.00000<br>
>> Near Hits: 0.14252 0.13498<br>
>> Not-Modified Replies: 0.00865 0.03066<br>
>> DNS Lookups: 0.00000 0.00372<br>
>> ICP Queries: 0.00000 0.00000<br>
>> Resource usage for squid:<br>
>> UP Time: 486282.612 seconds<br>
>> CPU Time: 65555.712 seconds<br>
>> CPU Usage: 13.48%<br>
>> CPU Usage, 5 minute avg: 26.89%<br>
>> CPU Usage, 60 minute avg: 68.00%<br>
>> Maximum Resident Size: 37896960 KB<br>
>> Page faults with physical i/o: 10843<br>
>> Memory accounted for:<br>
>> Total accounted: -1459461 KB<br>
>> memPoolAlloc calls: 11408<br>
>> memPoolFree calls: 1888969689<br>
>> File descriptor usage for squid:<br>
>> Maximum number of file descriptors: 4096<br>
>> Largest file desc currently in use: 2149<br>
>> Number of file desc currently in use: 679<br>
>> Files queued for open: 0<br>
>> Available number of file descriptors: 3417<br>
>> Reserved number of file descriptors: 100<br>
>> Store Disk files open: 0<br>
>> Internal Data Structures:<br>
>> 97906 StoreEntries<br>
>> 3002 StoreEntries with MemObjects<br>
>> 2838 Hot Object Cache Items<br>
>> 97742 on-disk objects<br>
>><br>
>> ------ pfctl -si ------<br>
>><br>
>> Status: Enabled for 25 days 22:58:24 Debug: Urgent<br>
>><br>
>> State Table Total Rate<br>
>> current entries 8085<br>
>> searches 6650475717 2965.4/s<br>
>> inserts 133521957 59.5/s<br>
>> removals 133552376 59.5/s<br>
>> Counters<br>
>> match 605960865 270.2/s<br>
>> bad-offset 0 0.0/s<br>
>> fragment 1 0.0/s<br>
>> short 54 0.0/s<br>
>> normalize 659 0.0/s<br>
>> memory 0 0.0/s<br>
>> bad-timestamp 0 0.0/s<br>
>> congestion 0 0.0/s<br>
>> ip-option 0 0.0/s<br>
>> proto-cksum 0 0.0/s<br>
>> state-mismatch 104674 0.0/s<br>
>> state-insert 38501 0.0/s<br>
>> state-limit 0 0.0/s<br>
>> src-limit 0 0.0/s<br>
>> synproxy 0 0.0/s<br>
>> map-failed 0 0.0/s<br>
>><br>
>> ------ sysctl -a | grep swap ------<br>
>><br>
>> swap_pager: out of swap space<br>
>> swp_pager_getswapspace(32): failed<br>
>> swap_pager: out of swap space<br>
>> swp_pager_getswapspace(31): failed<br>
>> swap_pager: out of swap space<br>
>> swp_pager_getswapspace(1): failed<br>
>> 1 PART da0p2 2147483648 512 i 2 o 544768 ty freebsd-swap xs GPT xt<br>
>> 516e7cb5-6ecf-11d6-8ff8-00022d09712b<br>
>> 0 MD md1 94371840 512 u 1 s 512 f 0 fs 0 l 94371840 t swap label<br>
>> 0 MD md0 62914560 512 u 0 s 512 f 0 fs 0 l 62914560 t swap label<br>
>> z0xfffff80003ec5800 [shape=box,label="SWAP\nswap\nr#3"];<br>
>> <name>swap</name><br>
>> <type>swap</type><br>
>> <type>swap</type><br>
>> <type>freebsd-swap</type><br>
>> vm.swap_enabled: 1<br>
>> vm.domain.0.stats.unswappable: 2044<br>
>> vm.swap_idle_threshold2: 10<br>
>> vm.swap_idle_threshold1: 2<br>
>> vm.swap_idle_enabled: 0<br>
>> vm.disable_swapspace_pageouts: 0<br>
>> vm.stats.vm.v_swappgsout: 3154299<br>
>> vm.stats.vm.v_swappgsin: 510404<br>
>> vm.stats.vm.v_swapout: 174446<br>
>> vm.stats.vm.v_swapin: 62590<br>
>> vm.stats.swap.free_completed: 54375<br>
>> vm.stats.swap.free_deferred: 56992<br>
>> vm.nswapdev: 1<br>
>> vm.swap_fragmentation:<br>
>> vm.swap_async_max: 4<br>
>> vm.swap_maxpages: 32572800<br>
>> vm.swap_total: 2147483648<br>
>> vm.swap_reserved: 384676114432<br>
>><br>
>> ------ /usr/sbin/swapinfo -h ------<br>
>><br>
>> Device Size Used Avail Capacity<br>
>> /dev/da0p2 2.0G 2.0G 8.0K 100%<br>
>><br>
>><br>
>> ############# squid.conf #############<br>
>><br>
>> http_port 3128 ssl-bump generate-host-certificates=on<br>
>> dynamic_cert_mem_cache_size=20MB cert=/xxxx/conf/certs/ca.crt<br>
>> key=/xxxx/conf/certs/ca.key<br>
>> http_port 3129 intercept<br>
>> https_port 3130 intercept ssl-bump generate-host-certificates=on<br>
>> dynamic_cert_mem_cache_size=20MB cert=/xxxx/conf/certs/ca.crt<br>
>> key=/xxxx/conf/certs/ca.key<br>
>> visible_hostname xxxx.xxxx.xxxx<br>
>> max_filedescriptors 4096<br>
>> maximum_object_size 4096 KB<br>
>> minimum_object_size 0 KB<br>
>> maximum_object_size_in_memory 256 KB<br>
>> fqdncache_size 1024<br>
>> cache_mgr xxxx@xxxx<br>
>> dns_nameservers 127.0.0.1<br>
>> cache_replacement_policy heap LFUDA<br>
>> memory_replacement_policy heap GDSF<br>
>> cache_mem 16 MB<br>
>> cache_dir ufs /xxxx/chroot/osproxy/cache 3072 32 256<br>
>> forwarded_for on<br>
>> memory_pools off<br>
>> logformat xxxx %ts|%6tr|%>a|%Ss|%03>Hs|%<st|%rm|%un|%mt|%ea|%ru<br>
>> logfile_rotate 0<br>
>> httpd_suppress_version_string on<br>
>> strip_query_terms off<br>
>> _______________________________________________<br>
>> squid-users mailing list<br>
>> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
>> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
><br>
><br>
><br>
> --<br>
> Hamilton<br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><font size="2">Hamilton</font></div></div></div></div></div>