[squid-users] squid in container aborted on low memory server

[Alex Rousskov]

On 3/6/19 11:25 AM, George Xie wrote:

>> So disabling swap entirely on the server is not a great idea. It just
>> moves the error and shutdown to happen at peak traffic times when it is
>> least wanted.

> but I guess this is common in the "cloud" era, eg: Google Compute Engine.

Moreover, in many production environments "swapping during peak traffic
times" is worse than "shutting down during peak traffic times". YMMV.

Alex.


> Xie Shi
> On Tue, Mar 5, 2019 at 4:13 PM Amos Jeffries <squid3 at treenet.co.nz> wrote:
>>
>> On 4/03/19 9:45 pm, George Xie wrote:
>>>     > On 4/03/19 5:39 pm, George Xie wrote:
>>>     > > hi all:
>>>     > >
>>>     > > Squid version: 3.5.23-5+deb9u1
>>>     > > Docker version 18.09.3, build 774a1f4
>>>     > > Linux instance-4 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27)
>>>     > > x86_64 GNU/Linux
>>>     > >
>>>     > > I have the following squid config:
>>>     > >
>>>     > >
>>>     > >     http_port 127.0.0.1:3128 <http://127.0.0.1:3128>
>>>     > >     cache deny all
>>>     > >     access_log none
>>>     > >
>>>     > What is it exactly that you think this is doing in regards to Squid
>>>     > memory needs?
>>>     >
>>>
>>>
>>> sorry, I don't get your quest.
>>>
>>
>> I was asking to see what you were thinking was going on with those settings.
>>
>> As Alex already pointed out the "cache deny all" does not reduce memory
>> needs of Squid in any way. It just makes 256MB of that RAM become
>> pointless allocating.
>>
>> So, if you actually do not want the proxy caching anything, then
>> disabling the cache_mem (set it to 0 as per Alex response) would be the
>> best choice of action before you go any further.
>>
>> Or if you *do* want caching, and were trying to disable it for testing
>> the memory issue. Then your test was wrong, and produces incorrect
>> conclusion. Just reducing cache_mem would be best for this case - set it
>> to a value that should reasonably fit this container and see if the
>> proxy runs okay.
>>
>>
>> ...
>>>     > >
>>>     > > it appears that squid (or glibc) tries to allocate 392m memory,
>>>     which is
>>>     > > larger than host free memory 370m.
>>>     > > but I guess squid don't need that much memory, I have another
>>>     running
>>>     > > squid instance, which only uses < 200m memory.
>>>     > No doubt it is configured to use less memory. For example by reducing
>>>     > the default memory cache size.
>>>     >
>>>
>>>
>>> that running squid instance has the same config.
>>>
>>
>> Then something odd is going on between the two. They should indeed have
>> had the same behaviour (either work or same error).
>>
>> Whatever the issue is it is being triggered by the large blocks of RAM
>> allocated by a default Squid. The easiest to modify is the cache_mem.
>>
>>
>>>
>>>     > > the oddest thing is if I run squid on the host (also Debian 9)
>>>     directly,
>>>     > > not in the container, squid could start and run as normal.
>>>     > >
>>>     > Linux typically allows RAM over-allocation. Which works okay so
>>>     long as
>>>     > there is sufficient swap space and there is time between memory
>>>     usage to
>>>     > do the swap in/out process.
>>>     > Amos
>>>
>>>
>>> swap is disabled in the host server, so do in the container.
>>>
>>> after all, I wonder why squid would try to claim 392m memory if don't
>>> need that much.
>>>
>>
>> Squid thinks it does. All client traffic is denied being cached by that
>> "deny all". BUT ... there are internally generated items which also use
>> cache. So there is 256MB default RAM cache allocated and only those few
>> small things being put in it.
>>
>> You could set it to '0' or to some small value and the allocation size
>> should go down accordingly.
>>
>>
>> That said, every bit of client traffic headed towards the proxy uses
>> memory of volatile amount and at peak times it may need to allocate
>> large blocks.
>>
>> So disabling swap entirely on the server is not a great idea. It just
>> moves the error and shutdown to happen at peak traffic times when it is
>> least wanted.
>>
>>
>> Amos
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>