[squid-users] Setting Squid to work with a remote DB?

[roee klinger]

Thank you Alex, for the detailed response, I will keep that info in mind
while testing.

If I can convenience the team to increase the "auth_param basic
credentialsttl" to around 60
minutes, we should have absolutely no problem, please correct me if I am
wrong.

As per my understanding, even if the DB server is all the way across the
world, and we have an added
response time of 1 second that occurs only once every 60 minutes, that
would have almost 0 impact
on the performance and resources used by Squid.

My only concern is for 407s, do users that get 407 are also cached?
In case that they do get cached, that would mean that if I send one request
using some username, it gets 407, and then I add the username to the DB,
it will still not work for 60 minutes.

However, if it is not cached, that could potentially cause big resources
usage as well,
since every 407 request takes around 1 second to resolve, and we have a lot
of those.

Anyway, it seems that no matter if 407 are cached or not, it will cause
some issues
if the DB response takes around 1 second unless I am overestimating the
resource
use of these requests.

I would love to hear your thoughts on that solution,
Thanks,
Roee


On Sat, Aug 28, 2021 at 10:33 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 8/28/21 2:59 PM, roee klinger wrote:
>
> > I have multiple Squid servers installed in different data centers across
> > different cloud providers, and they all need to authenticate users using
> > a single database (MySQL) on a separate server, which is also on a
> > different cloud provider on a different data center.
> >
> > I have already written an external authentication script that reads from
> > MySQL and everything is working fine, however, I have some
> > performance concerns, since the DB is located externally and in a
> > different region of the world from the Squid server.
> >
> > I made some speed tests to see how long it takes to query the DB as
> > Squid would:
> >
> > if the database is located on the same machine as Squid:
> > 1.067-millisecond average query
> >
> > if the database is located on the same datacenter as Squid:
> > 2.67-millisecond average query
> >
> > if the database is located on a different datacenter than Squid
> > (different country as well): 38.9-millisecond average query
> >
> >
> > Now I am wondering, is 36-millisecond average added query time a big
> > deal when dealing with HTTP/S traffic? how significant is this added
> > time to Squid and will performance get hurt drastically?
>
> * When you look at a single HTTP transaction, adding 36ms is usually not
> a big deal: Humans are incapable of discerning such delays and
> automatons that need true real-time performance are unlikely to go
> through your Squids.
>
> * Additional 36ms added to mean transaction response time create
> significant perceived delays for sites/services that load lots of
> resources _sequentially,_ especially if such a site/service has lighting
> fast response times without those extra delays. For example, without
> authentication caching and similar optimizations, a cumulative extra
> delay of 100 sequential transactions (that would normally take, say,
> 300ms total) would be 3.6 seconds -- something many humans will be
> annoyed with!
>
> * Additional 36ms added to mean transaction response time can kill
> performance of a Squid instance that is operating at the limit of some
> resource capacity.
>
> For example, imagine that, without authentication delays, your Squid
> transactions have 10ms mean response time, and your Squid instance is
> dealing with 10'000 requests per second load. That combination results
> in just 100 concurrent requests (10'000r/s * 0.01s = 100r). If you add
> 36ms to that 10ms response time, your Squid would have to deal with 460
> concurrent requests (10'000r/s * 0.046s = 460r) -- a 360% increase in
> concurrency levels and associated resource consumption. Such an increase
> is likely to maim any Squid that was operating without significant spare
> resources.
>
> For an opposite example, imagine that, without authentication delays,
> your Squid transactions have 1 second mean response time, and your Squid
> instance is dealing with 10'000 requests per second load. That
> combination results in 10'000 concurrent requests. If you add 36ms to
> that 1s response time, your Squid would have to deal with 10'360
> concurrent requests -- a mere 3.6% increase in concurrency levels and
> associated resource consumption.
>
> Keep in mind that as Squid approaches resource limits, things usually
> get _exponentially_ worse.
>
>
> The impact of additional authentication delays should be fairly easy to
> model/test.
>
>
> HTH,
>
> Alex.
>
>
> > I know there is some caching going on the Squid side, but I had to set
> > the caching to really low values (around 15s), as per the requirement I
> > was given.
> >
> > If I will have no other choice, I will simply replicate the DB table
> > from the DB server to the Squid server, but I prefer not to do that, as
> > it will require installing MySQL on all the Squid servers (or some other
> > DB, but then I have to do replication from different DBs).
> >
> > Thanks.
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
> >
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20210828/2b2c7d34/attachment-0001.htm>