[squid-users] Setting Squid to work with a remote DB?

Sat Aug 28 20:15:53 UTC 2021

On 8/28/21 4:00 PM, roee klinger wrote:

> If I can convenience the team to increase the "auth_param basic
> credentialsttl" to around 60
> minutes, we should have absolutely no problem, please correct me if I am
> wrong.

This is not my area of expertise, so I hope that others on the list will
respond, but I suspect that the correct answer depends on what exactly
is being authenticated in your environment. For example:

* If a single user computer requires at most one 1s authentication
during any 60 minute period, then you are probably right.

* However, if 100 transactions from that single user computer require 1s
authentication every hour (e.g., because they all arrived within 1
second from each other using 100 independent concurrent connections, and
each connection has to be authenticated from scratch, incurring that 1s
delay), then you might get some complaints from users (and their
automatons) that the internet "feels a bit sluggish" "right after I
resume working"...

Alex.

> As per my understanding, even if the DB server is all the way across the
> world, and we have an added
> response time of 1 second that occurs only once every 60 minutes, that
> would have almost 0 impact
> on the performance and resources used by Squid.
> 
> My only concern is for 407s, do users that get 407 are also cached?
> In case that they do get cached, that would mean that if I send one request
> using some username, it gets 407, and then I add the username to the DB,
> it will still not work for 60 minutes.
> 
> However, if it is not cached, that could potentially cause big resources
> usage as well,
> since every 407 request takes around 1 second to resolve, and we have a
> lot of those.
> 
> Anyway, it seems that no matter if 407 are cached or not, it will cause
> some issues
> if the DB response takes around 1 second unless I am overestimating the
> resource 
> use of these requests.
> 
> I would love to hear your thoughts on that solution,
> Thanks,
> Roee
> 
> 
> On Sat, Aug 28, 2021 at 10:33 PM Alex Rousskov
> <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>> wrote:
> 
>     On 8/28/21 2:59 PM, roee klinger wrote:
> 
>     > I have multiple Squid servers installed in different data centers
>     across
>     > different cloud providers, and they all need to authenticate users
>     using
>     > a single database (MySQL) on a separate server, which is also on a
>     > different cloud provider on a different data center.
>     >
>     > I have already written an external authentication script that
>     reads from
>     > MySQL and everything is working fine, however, I have some
>     > performance concerns, since the DB is located externally and in a
>     > different region of the world from the Squid server.
>     >
>     > I made some speed tests to see how long it takes to query the DB as
>     > Squid would:
>     >
>     > if the database is located on the same machine as Squid:
>     > 1.067-millisecond average query
>     >
>     > if the database is located on the same datacenter as Squid:
>     > 2.67-millisecond average query
>     >
>     > if the database is located on a different datacenter than Squid
>     > (different country as well): 38.9-millisecond average query
>     >
>     >
>     > Now I am wondering, is 36-millisecond average added query time a big
>     > deal when dealing with HTTP/S traffic? how significant is this added
>     > time to Squid and will performance get hurt drastically?
> 
>     * When you look at a single HTTP transaction, adding 36ms is usually not
>     a big deal: Humans are incapable of discerning such delays and
>     automatons that need true real-time performance are unlikely to go
>     through your Squids.
> 
>     * Additional 36ms added to mean transaction response time create
>     significant perceived delays for sites/services that load lots of
>     resources _sequentially,_ especially if such a site/service has lighting
>     fast response times without those extra delays. For example, without
>     authentication caching and similar optimizations, a cumulative extra
>     delay of 100 sequential transactions (that would normally take, say,
>     300ms total) would be 3.6 seconds -- something many humans will be
>     annoyed with!
> 
>     * Additional 36ms added to mean transaction response time can kill
>     performance of a Squid instance that is operating at the limit of some
>     resource capacity.
> 
>     For example, imagine that, without authentication delays, your Squid
>     transactions have 10ms mean response time, and your Squid instance is
>     dealing with 10'000 requests per second load. That combination results
>     in just 100 concurrent requests (10'000r/s * 0.01s = 100r). If you add
>     36ms to that 10ms response time, your Squid would have to deal with 460
>     concurrent requests (10'000r/s * 0.046s = 460r) -- a 360% increase in
>     concurrency levels and associated resource consumption. Such an increase
>     is likely to maim any Squid that was operating without significant spare
>     resources.
> 
>     For an opposite example, imagine that, without authentication delays,
>     your Squid transactions have 1 second mean response time, and your Squid
>     instance is dealing with 10'000 requests per second load. That
>     combination results in 10'000 concurrent requests. If you add 36ms to
>     that 1s response time, your Squid would have to deal with 10'360
>     concurrent requests -- a mere 3.6% increase in concurrency levels and
>     associated resource consumption.
> 
>     Keep in mind that as Squid approaches resource limits, things usually
>     get _exponentially_ worse.
> 
> 
>     The impact of additional authentication delays should be fairly easy to
>     model/test.
> 
> 
>     HTH,
> 
>     Alex.
> 
> 
>     > I know there is some caching going on the Squid side, but I had to set
>     > the caching to really low values (around 15s), as per the
>     requirement I
>     > was given.
>     >
>     > If I will have no other choice, I will simply replicate the DB table
>     > from the DB server to the Squid server, but I prefer not to do
>     that, as
>     > it will require installing MySQL on all the Squid servers (or some
>     other
>     > DB, but then I have to do replication from different DBs).
>     >
>     > Thanks.
>     >
>     >
>     > _______________________________________________
>     > squid-users mailing list
>     > squid-users at lists.squid-cache.org
>     <mailto:squid-users at lists.squid-cache.org>
>     > http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
>     >
> 
>     _______________________________________________
>     squid-users mailing list
>     squid-users at lists.squid-cache.org
>     <mailto:squid-users at lists.squid-cache.org>
>     http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
> 