[squid-users] SQUID memory error after vm.swappines changed from 60 to 10

[Bike dernikov1]

>>On 11/08/2017 02:26 AM, Bike dernikov1 wrote:

>> I hope that someone can explain what happened, why squid stopped working.

>I can suggest a working theory: You did not have enough RAM before
>vm.swappiness changes and the same insufficient RAM problem led to
>failed system calls after you told the OS to free RAM (into swap) less
>aggressively. The changes effectively decreased the amount of
>immediately available RAM.

>From "free -m output" (available column)  i thought that we have
enough memory, because  as I can remember, available never dropped
under 2GB.
As I understood  that system with  vm.swappines set to 10, will start
draining ram to swap after 90% memory used.
So why didn't swap hop in ? That we set vm.swappines to 0 i would
understand error.
Because errors started with  logrotate, i suspect that  problem
exploded because of logrotate and swap started to write to a disk at
same time,
and because of slow disks (small IOPS) it could not drain memory fast
enough, so available memory vanished and hell lose from there.


>If Squid is the primary service on your server, then consider:

It isn't only service but use over 90-95 percent memory resources.

>1. Removing RAID (for Squid disk cache).

We don't use disk cache because of slow disks, compared to  "high"
bandwith (we have 500Mbit/s)

>2. Removing swap (or at least not counting on it and treating
>  any non-trivial swap usage as a system misconfiguration).

What to search ? Key words, vm.swappines got us in problem, as we
tried to tune  ?

>3. Identify the primary RAM consumers and
>  the factors that influence their memory usage.

Squid, we suspect on kerberos negotiator, squid helpers, we will tune
up down setting over two weeks.
We have problems with squid.conf reconfiguration, because users lose
connection (they got squid error screen for moment) so it is nightmare
if we changing configuration in work time.

>The above actions may not solve your "insufficient RAM" problem (unless
>RAID was the primary culprit) but they will give you a better starting
>point for finding the solution. Others on the list can help you get
>there, especially if you have specific questions.
>
>
>Good luck,
>
>Alex.

Thanks for theory, and help.


On Wed, Nov 8, 2017 at 4:08 PM, Alex Rousskov
<rousskov at measurement-factory.com> wrote:
> On 11/08/2017 02:26 AM, Bike dernikov1 wrote:
>
>> I hope that someone can explain what happened, why squid stopped working.
>
> I can suggest a working theory: You did not have enough RAM before
> vm.swappiness changes and the same insufficient RAM problem led to
> failed system calls after you told the OS to free RAM (into swap) less
> aggressively. The changes effectively decreased the amount of
> immediately available RAM.
>
> If Squid is the primary service on your server, then consider:
>
> 1. Removing RAID (for Squid disk cache).
>
> 2. Removing swap (or at least not counting on it and treating
>    any non-trivial swap usage as a system misconfiguration).
>
> 3. Identify the primary RAM consumers and
>    the factors that influence their memory usage.
>
> The above actions may not solve your "insufficient RAM" problem (unless
> RAID was the primary culprit) but they will give you a better starting
> point for finding the solution. Others on the list can help you get
> there, especially if you have specific questions.
>
>
> Good luck,
>
> Alex.
>
>> The server has 24GB DDR3 memory,  disk swap set to 24GB, 12 CPU (24HT cores).
>> We have 2800 users, using  kerberos authentication, squidguard for
>> filtering, ldap authorization.