<div dir="auto">Hey Ivan,<div dir="auto"><br></div><div dir="auto">From what i remember there is a calculation for how much k per conn should squid use.</div><div dir="auto">another thing is that squid is not returning memory once ot took it.</div><div dir="auto">Amos knows about this and might be able to respond.</div><div dir="auto"><br></div><div dir="auto">Eliezer</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 5, 2020, 20:34 Ivan Bulatovic <<a href="mailto:ivan.bulatovic@gmail.com">ivan.bulatovic@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Eliezer,<br>
<br>
In the original message I sent to the squid-users mail list, I<br>
attached listings from mgr:info and mgr:mem. The server is definitely<br>
using a lot of connections (close to 200K connections), which is why I<br>
increased the open files limits in linux as well as in squid.conf. And<br>
there are a lot of requests per second, when the server is running. I<br>
could understand that it needs a lot of memory for in-transit cache,<br>
but that memory should be later released back to OS, once the<br>
requests load goes down. However, that is not the case, even days<br>
after there is no load on the server, it stays at 11GB of RAM and 5.5<br>
GB of swap used. The second I restart the squid process, everything<br>
goes to normal, memory is released. That is why I suspect there is<br>
some memory leak somewhere.<br>
<br>
The server is a Ubuntu 18.04 LTS VM (running on Hyper-V 2019 server),<br>
with 8 virtual processors and 12GB of RAM (although I can increase<br>
that if that is the problem, but I thought that without caching this<br>
would be more than enough).<br>
<br>
I am not using dynamic memory on Hyper-V (it is turned off for this VM).<br>
<br>
Best regards,<br>
Ivan<br>
<br>
On Wed, Aug 5, 2020 at 7:14 PM NgTech LTD <<a href="mailto:ngtech1ltd@gmail.com" target="_blank" rel="noreferrer">ngtech1ltd@gmail.com</a>> wrote:<br>
><br>
> I think that the mgr:info or another page there contains the amount of requests per second etc.<br>
> also netstat or ss -ntp might give some basic understanding about this server size.<br>
><br>
> are you using dynamic memory on the hyper-v hypervisor?<br>
><br>
> Eliezer<br>
><br>
> On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <<a href="mailto:ivan.bulatovic@gmail.com" target="_blank" rel="noreferrer">ivan.bulatovic@gmail.com</a>> wrote:<br>
>><br>
>> Hi Alex,<br>
>><br>
>> Thank you very much for your help.<br>
>><br>
>> I opened a bug on <a href="http://bugs.squid-cache.org" rel="noreferrer noreferrer" target="_blank">bugs.squid-cache.org</a><br>
>> (<a href="https://bugs.squid-cache.org/show_bug.cgi?id=5071" rel="noreferrer noreferrer" target="_blank">https://bugs.squid-cache.org/show_bug.cgi?id=5071</a>).<br>
>><br>
>> Best regards,<br>
>> Ivan<br>
>><br>
>> On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov<br>
>> <<a href="mailto:rousskov@measurement-factory.com" target="_blank" rel="noreferrer">rousskov@measurement-factory.com</a>> wrote:<br>
>> ><br>
>> > On 8/3/20 9:11 AM, Ivan Bulatovic wrote:<br>
>> ><br>
>> > > Looks like squid has some serious memory issues when under heavy load<br>
>> > > (90 servers that crawl Internet sites).<br>
>> ><br>
>> > > Maximum Resident Size: 41500720 KB<br>
>> ><br>
>> > If the above (unreliable) report matches your observations using system<br>
>> > tools like "top", then it is indeed likely that your Squid is suffering<br>
>> > from a memory leak -- 41GB is usually too much for most non-caching<br>
>> > Squid instances.<br>
>> ><br>
>> > Identifying the leak may take some time, and I am not volunteering to do<br>
>> > the necessary legwork personally, but the Squid Project does fix<br>
>> > virtually all runtime leaks that we know about. If you want to speed up<br>
>> > the process, one of the best things you can do is to run Squid under<br>
>> > valgrind with a good suppression file. This requires building Squid with<br>
>> > a special ./configure option. Several testing iterations may be<br>
>> > necessary. If you are willing to do this, please file a bug report and<br>
>> > somebody will guide you through the steps.<br>
>> ><br>
>> ><br>
>> > > It just eats up memory, and<br>
>> > > does not free it up even days after it is being used (with no load on<br>
>> > > the proxy for days).<br>
>> ><br>
>> > Some memory retention is expected by default. See<br>
>> > <a href="http://www.squid-cache.org/Doc/config/memory_pools/" rel="noreferrer noreferrer" target="_blank">http://www.squid-cache.org/Doc/config/memory_pools/</a><br>
>> ><br>
>> > Unfortunately, AFAICT, your mgr:mem output does not show any obvious<br>
>> > leaks -- all numbers are very small. If something is leaking a lot, then<br>
>> > it is probably not pooled by Squid.<br>
>> ><br>
>> ><br>
>> > HTH,<br>
>> ><br>
>> > Alex.<br>
>> ><br>
>> ><br>
>> > > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:<br>
>> > >><br>
>> > >> Hi all,<br>
>> > >><br>
>> > >> I am trying to configure squid to run as a forward proxy with no<br>
>> > >> caching (cache deny all) with an option to choose the outgoing IP<br>
>> > >> address based on the username. So all squid has to do is to use a<br>
>> > >> certain outgoing IP address for a certain user, return the data from<br>
>> > >> the server to that user and cache nothing.<br>
>> > >><br>
>> > >> For that I created a special authentication helper and used the ACLs<br>
>> > >> and tcp_outgoing_address to create a lot of users and outgoing IP<br>
>> > >> addresses (about 260 at the moment). Example (not the real IP I use,<br>
>> > >> of course):<br>
>> > >><br>
>> > >> acl use_IP1 proxy_auth user1<br>
>> > >> tcp_outgoing_address 1.2.3.4 use_IP1<br>
>> > >><br>
>> > >> I also configured the squid to use 4 workers, but this happens even<br>
>> > >> when I use only one worker (default)<br>
>> > >><br>
>> > >> And this works. However, under heavy load, Squid eats all of the RAM<br>
>> > >> and then starts going to swap. And the memory usage does not drop when<br>
>> > >> I remove all the load from squid (I shut down all clients).<br>
>> > >><br>
>> > >> I left it to see if the memory will be freed but even after leaving it<br>
>> > >> for an hour the info page reports this:<br>
>> > >> Cache information for squid:<br>
>> > >> Hits as % of all requests: 5min: 0.0%, 60min: 0.0%<br>
>> > >> Hits as % of bytes sent: 5min: 0.0%, 60min: 1.1%<br>
>> > >> Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%<br>
>> > >> Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%<br>
>> > >> Storage Swap size: 0 KB<br>
>> > >> Storage Swap capacity: 0.0% used, 100.0% free<br>
>> > >> Storage Mem size: 0 KB<br>
>> > >> Storage Mem capacity: 0.0% used, 100.0% free<br>
>> > >> Mean Object Size: 0.00 KB<br>
>> > >> Requests given to unlinkd: 0<br>
>> > >><br>
>> > >> Resource usage for squid:<br>
>> > >> UP Time: 255334.875 seconds<br>
>> > >> CPU Time: 7122.436 seconds<br>
>> > >> CPU Usage: 2.79%<br>
>> > >> CPU Usage, 5 minute avg: 0.05%<br>
>> > >> CPU Usage, 60 minute avg: 37.66%<br>
>> > >> Maximum Resident Size: 41500720 KB<br>
>> > >> Page faults with physical i/o: 1003410<br>
>> > >><br>
>> > >> And here is the listing of free and top commands (with no load on the server):<br>
>> > >><br>
>> > >> # free -h<br>
>> > >> total used free shared buff/cache available<br>
>> > >> Mem: 11G 10G 791M 676K 491M 1.0G<br>
>> > >> Swap: 11G 5.5G 6.5G<br>
>> > >><br>
>> > >> # top<br>
>> > >> top - 14:12:32 up 3 days, 1:30, 1 user, load average: 0.00, 0.00, 0.00<br>
>> > >> Tasks: 177 total, 1 running, 102 sleeping, 0 stopped, 0 zombie<br>
>> > >> %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu2 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
>> > >> KiB Mem : 91.2/12251688<br>
>> > >> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
>> > >> ]<br>
>> > >> KiB Swap: 45.8/12582904<br>
>> > >> [||||||||||||||||||||||||||||||||||||||||||||||<br>
>> > >> ]<br>
>> > >><br>
>> > >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
>> > >> 7851 proxy 20 0 6946872 2.514g 8084 S 0.0 21.5 29:43.74 squid<br>
>> > >> 7832 proxy 20 0 6711480 2.464g 8040 S 0.0 21.1 29:58.17 squid<br>
>> > >> 7814 proxy 20 0 6834928 2.454g 10024 S 0.0 21.0 29:47.56 squid<br>
>> > >> 7843 proxy 20 0 6906252 2.436g 8208 S 0.0 20.8 29:15.60 squid<br>
>> > >> 1329 root 20 0 2416672 167272 12680 S 0.0 1.4 136:18.57 metricbeat<br>
>> > >> 1321 root 20 0 1831804 48364 11648 S 0.0 0.4 14:32.10 filebeat<br>
>> > >> 474 root 19 -1 127796 17576 17144 S 0.0 0.1 0:27.01<br>
>> > >> systemd-journal<br>
>> > >> 7811 proxy 20 0 549384 14168 8372 S 0.0 0.1 0:20.87 squid<br>
>> > >> 1166 root 20 0 1749724 10596 4468 S 0.0 0.1 0:31.83 snapd<br>
>> > >> 43940 proxy 20 0 28884 9608 5384 S 0.0 0.1 0:00.14 python3<br>
>> > >> 43941 proxy 20 0 28884 9552 5328 S 0.0 0.1 0:00.10 python3<br>
>> > >> 43939 proxy 20 0 28884 9524 5308 S 0.0 0.1 0:00.12 python3<br>
>> > >> 43938 proxy 20 0 28884 9452 5232 S 0.0 0.1 0:00.16 python3<br>
>> > >> 48848 root 20 0 105688 6960 5968 S 0.0 0.1 0:00.02 sshd<br>
>> > >> 48974 janitor 20 0 108120 5380 4372 S 0.0 0.0 0:00.00 sshd<br>
>> > >> 1 root 20 0 86360 4364 2488 S 0.0 0.0 32:46.22 systemd<br>
>> > >> ...<br>
>> > >> ... lines ommited<br>
>> > >> ...<br>
>> > >><br>
>> > >> In the attachment you can find the printout from squidclient mgr:info<br>
>> > >> and squidclient mgr:mem. These are both taken at the moment when there<br>
>> > >> is no more load on the proxy. I also included my squid.conf file<br>
>> > >> (minus the two files where acls are defined and outgoing IP addresses,<br>
>> > >> these two contain only acl and tcp_outgoing_address lines as in the<br>
>> > >> example above).<br>
>> > >><br>
>> > >> Machine info:<br>
>> > >> OS: Ubuntu 18.04 with latest updates<br>
>> > >> Squid version 4.12 (from diladele repository)<br>
>> > >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM<br>
>> > >><br>
>> > >> I can not understand what is eating all of the memory, if I disabled the cache.<br>
>> > >><br>
>> > >> Maybe I configured something wrong but I can not find what.<br>
>> > >><br>
>> > >> Thank you for any help you can provide.<br>
>> > >><br>
>> > >> Best regards,<br>
>> > >> Ivan<br>
>> > > _______________________________________________<br>
>> > > squid-users mailing list<br>
>> > > <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br>
>> > > <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
>> > ><br>
>> ><br>
>> _______________________________________________<br>
>> squid-users mailing list<br>
>> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br>
>> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div>