[squid-users] Scaling concurrent TCP sessions beyond ephemeral port range
Praveen Ponakanti
pponakanti at roblox.com
Wed Nov 23 23:07:01 UTC 2022
Hi Alex,
Thanks for all the details. I will build a new squid image off master/v6 to
check for the memory leak with test traffic.
Regarding the TLS session cache, I will try setting it to 100MB. Are there
any stats exposed from the tls session cache that can be monitored to study
the session cache usage with our traffic patterns? Would those be
accessible via the squid-internal-mgr endpoint?
Thanks
Praveen
On Tue, Nov 22, 2022 at 7:58 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:
> On 11/22/22 21:06, Praveen Ponakanti wrote:
>
> > Do we have a recent squid ver 6 snapshot build available for testing?
>
> Sorry, I do not know the exact answer to your question. One can
> certainly build master/v6 from git sources, of course.
>
>
> > The following config knobs were tried and did not make much of a
> > difference with respect to concurrent outbound TLS sessions across the
> > workers.
>
> I can only speculate that there is a Squid bug that prevents workers
> from using (or sharing) the TLS session cache OR your traffic patterns
> do not allow (much) sharing OR your TLS session cache is too small for
> those traffic patterns. Lot's of possibilities here!
>
>
> > The docs say the default sslproxy_session_cache_size is 2 MB,
> > how high can we go?
>
> I have not tested this, but, bugs notwithstanding, I would expect you to
> be able to go as high as the maximum shared memory segment size in your
> environment (e.g., see sysctl kernel.shmax). The entire cache must fit
> into a single shared memory segment IIRC. It certainly does not hurt to
> try 100 MB if you already tried 10 MB. Squid should complain if it
> cannot allocate a segment of the specified size.
>
>
> > Are there any other knobs we can try to improve
> > session reuse for HTTPS reqs ? (without enabling squid cache)
>
> FWIW, I would try to understand _why_ sessions are not shared (enough)
> in the first place, especially if increasing sslproxy_session_cache_size
> value does not result in session cache hit ratio increase while there
> are still many false cache misses. I do not know whether anybody is
> paying a lot of attention to that cache in production, but the
> corresponding Squid code is not tested by the Squid Project CI (yet?).
>
> Also, there are different kinds of TLS session reuse. It is possible
> that the sessions you want to reuse are not supported by the Squid
> session cache. Unfortunately, I do not remember enough details to tell
> you more, but one can set up a controlled test and see what is actually
> going on.
>
>
> HTH,
>
> Alex.
>
>
> > sslproxy_session_cache_size 10 MB
> > tls_outgoing_options options=NO_TICKET
> >
> >
> > > Agree, it might not make sense to increase the complexity with
> sharing
> > > socket among the workers. Was thinking more on the lines of a
> hashmap
> > > that the coordinator could use to pick workers that already have a
> TCP
> > > connection to the destination being requested, instead of having
> the
> > > workers themselves share connection details.
> >
> >
> > Coordinator does not receive/see regular HTTP traffic. If we start
> > routing HTTP transactions through that process, it may become the
> > bottleneck itself _and_ will introduce additional overheads for
> passing
> > descriptors to workers. From performance point of view, the model
> with
> > one "routing" task doling work to workers works best (and is commonly
> > used) in threaded applications, but Squid is not threaded at that
> level.
> >
> >
> > > Most of the TCP connections are for HTTPS reqs, w/o TLS
> > termination at
> > > the squid. Does squid currently support a TLS session cache ?
> >
> > Yes, there is some support for worker-specific TLS session caching,
> > with
> > directives like sslproxy_session_cache_size, tls_outgoing_options
> > options=NO_TICKET (for outgoing sessions IIRC) and https_port
> > sslflags=NO_SESSION_REUSE and https_port sslcontext (for incoming
> > sessions).
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> > > If you are dealing with TLS sessions as well, then we should
> > add a
> > > shared memory TLS session cache that all workers can tap into.
> > >
> > >
> > > Cheers,
> > >
> > > Alex.
> > >
> > > > On Tue, Jun 21, 2022 at 2:11 PM Alex Rousskov wrote:
> > > >
> > > > On 6/19/22 12:48, Praveen Ponakanti wrote:
> > > >
> > > > > What is the process to have this code patch
> > upstreamed for
> > > future
> > > > squid
> > > > > versions?
> > > >
> > > > In short, just post a quality pull request on GitHub
> > (or find
> > > somebody
> > > > who can guide your code towards official acceptance
> > for you). For
> > > > details, please see
> > > https://wiki.squid-cache.org/MergeProcedure
> > <https://wiki.squid-cache.org/MergeProcedure>
> > > <https://wiki.squid-cache.org/MergeProcedure
> > <https://wiki.squid-cache.org/MergeProcedure>>
> > > > <https://wiki.squid-cache.org/MergeProcedure
> > <https://wiki.squid-cache.org/MergeProcedure>
> > > <https://wiki.squid-cache.org/MergeProcedure
> > <https://wiki.squid-cache.org/MergeProcedure>>>
> > > >
> > > >
> > > > Thank you,
> > > >
> > > > Alex.
> > > >
> > > >
> > > > > On Fri, May 20, 2022 at 9:31 PM Amos Jeffries
> > > > <squid3 at treenet.co.nz <mailto:squid3 at treenet.co.nz>
> > <mailto:squid3 at treenet.co.nz <mailto:squid3 at treenet.co.nz>>
> > > <mailto:squid3 at treenet.co.nz <mailto:squid3 at treenet.co.nz>
> > <mailto:squid3 at treenet.co.nz <mailto:squid3 at treenet.co.nz>>>
> > > > > wrote:
> > > > >
> > > > > On 20/05/22 19:44, Praveen Ponakanti wrote:
> > > > > > Hi Alex,
> > > > > >
> > > > > > Thanks for going through several steps to
> > help mitigate
> > > > src port
> > > > > > exhaustion. We are looking to achieve
> > 400-500% more
> > > > > > concurrent connections if we could :) as
> > there is a
> > > > > significant buffer
> > > > > > on the available CPU.
> > > > >
> > > > > Then you require at least 4, maybe 5, IP
> > addresses to
> > > handle
> > > > that many
> > > > > concurrent connections with Squid.
> > > > >
> > > > >
> > > > > We would like to investigate going beyond the
> > ephemeral port
> > > > range for
> > > > > some specific destination IP:PORT addresses. For
> > that it
> > > appears
> > > > squid
> > > > > does not round-robin requests if we use multiple
> > > > tcp_outgoing_addresses.
> > > > > We could use ACL’s to pick a different outbound IP
> > based
> > > on the
> > > > clients
> > > > > source IP, however that is not very ideal in our
> > > environment as our
> > > > > clients aren’t always equally split by subnet.
> > However, if
> > > we could
> > > > > split by the client’s source port that might help
> > achieve
> > > this. For
> > > > > example something like:
> > > > >
> > > > >
> > > > > acl pool1 clientport 0-32768
> > > > >
> > > > > acl pool2 clientport 32769-65536
> > > > >
> > > > >
> > > > > tcp_outgoing_address 10.1.0.1 pool1
> > > > >
> > > > > tcp_outgoing_address 10.1.0.2 pool2
> > > > >
> > > > >
> > > > > Squid's ACLs currently do not allow filtering by the
> > > client's source
> > > > > port. We could look into a separate patch to add
> this
> > > > functionality to
> > > > > squid’s ACL code if that makes sense. Or is there a
> > better
> > > way to
> > > > > achieve this?
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Praveen
> > > > >
> > > > >
> > > > > > The option to use multiple
> tcp_outoing_addresses
> > > appears to be
> > > > > promising
> > > > > > along with some tweaks to the TCP timeouts.
> > I guess we
> > > > could use
> > > > > ACLs to
> > > > > > pick a different outbound IP based on the
> > > requesting client's
> > > > > prefix. We
> > > > > > had not considered that option as the
> ephemeral
> > > ports were
> > > > no longer
> > > > > > available to other applications when squid
> > uses most of
> > > > them with a
> > > > > > single outbound IP configured. We are also
> > looking to
> > > > modify the
> > > > > code to
> > > > > > use the IP_BIND_ADDRESS_NO_PORT sockopt as
> that
> > > could help
> > > > delay
> > > > > port
> > > > > > assignment with the bind() call on the
> > outbound TCP
> > > > sessions (to
> > > > > > hopefully allow access to the 4-tuple on the
> > socket).
> > > > >
> > > > > Patches welcome.
> > > > >
> > > > > However, please be aware that use of the
> 4-tuple is
> > > often no
> > > > different
> > > > > from the 3-tuple since the dst-port is
> > typically identical
> > > > for all
> > > > > outgoing traffic to a given dst-IP.
> > > > >
> > > > >
> > > > > Cheers
> > > > > Amos
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20221123/7d1bc55e/attachment-0001.htm>
More information about the squid-users
mailing list