[squid-users] External helper consumes too many DB connections

[roee klinger]

Hello,

I am running multiple instances of Squid in a K8S environment, each Squid
instance has a helper that authenticates users based on their username and
password, the scripts are written in Python.

I have been facing an issue, that when under load, the helpers (even with
3600 sec TTL) swamp the MariaDB instance, causing it to reach 100% CPU,
basically I believe because each helper opens up its own connection to
MariaDB, which ends up as a lot of connections.

My initial idea was to create a Redis DB next to each Squid instance and
connect each Squid to its own dedicated Redis. I will sync Redis with
MariaDB every minute, thus decreasing the connections count from a few 100s
to just 1 every minute. This will also improve speeds since Redis is much
faster than MariaDB.

The problem is, however, that there will still be many connections from
Squid to Redis, and I probably that will consume a lot of DB resources as
well, which I don't actually know how to optimize, since it seems that
Squid opens many processes, and there is no way to get them to talk to each
other (expect TTL values, which seems not to help in my case, which I also
don't understand why that is).

What is the best practice to handle this? considering I have the following
requirements:

1. Fast
2. Refresh data every minute
3. Consume as least amount of DB resources as possible

Thanks,
Roee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220208/ab7b4b57/attachment.htm>