[squid-users] Scaling concurrent TCP sessions beyond ephemeral port range

Fri May 20 07:44:28 UTC 2022

Hi Alex,

Thanks for going through several steps to help mitigate src port
exhaustion. We are looking to achieve 400-500% more concurrent connections
if we could :) as there is a significant buffer on the available CPU.
The option to use multiple tcp_outoing_addresses appears to be promising
along with some tweaks to the TCP timeouts. I guess we could use ACLs to
pick a different outbound IP based on the requesting client's prefix. We
had not considered that option as the ephemeral ports were no longer
available to other applications when squid uses most of them with a single
outbound IP configured. We are also looking to modify the code to use the
IP_BIND_ADDRESS_NO_PORT sockopt as that could help delay port assignment
with the bind() call on the outbound TCP sessions (to hopefully allow
access to the 4-tuple on the socket).
Thanks
Praveen

On Thu, May 19, 2022 at 7:18 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 5/19/22 20:22, Praveen Ponakanti wrote:
>
> > Does anyone have recommendations on scaling concurrent connections
> > through the squid proxy to above the ephemeral port range?
>
> I know of several solutions, but not all of them are probably applicable
> to your specific situation:
>
> 1. Decrease the amount of time closed TCP connections occupy the port.
> For example, if you have many connections in TIME_WAIT state, and can
> afford to lower that state duration, it may help free ports faster.
>
> 2. If outgoing connections are closed faster (i.e. after fewer requests)
> than they should be, then fix that problem to increase connection reuse
> (and, hence, decrease port pressure). This solution is usually
> applicable to environments where you control both ends of the connection
> and see some premature closures.
>
> 3. Use more outgoing IP addresses. Without modifications, Squid would
> not automatically pick the next outgoing IP address after using up most
> of the ports on the previous one, but perhaps the OS would do the right
> thing _for_ Squid? Not sure. You can use tcp_outgoing_address with
> random ACLs to force-spread the load across multiple IPs (and, hence,
> multiple port banks). This does not work if you must use a single
> outgoing IP for some reason.
>
> 4. Modify Squid to retry port binding errors. This is easy to do but
> (without #5 below) it will not help much once ephemeral ports become
> scarce (in my experience; I have not checked what the latest kernels are
> capable of in this area).
>
> 5. If you need, say, 20-30% more concurrency (rather than 100+%) and
> cannot use multiple outbound IP addresses, then would be possible to
> modify Squid to implement a manual port allocation algorithm that
> usually works a lot more reliable under load than ephemeral ports
> administered by the OS (last time I checked, which was a few years ago).
> You will still be bound by the TCP limit of 64K ports (minus whatever
> you want to leave for other applications that open outgoing connections)
> and various TCP-level timeouts, of course, but the number of cases where
> Squid cannot open a port because of OS port mismanagement will go down.
>
> FWIW, we successfully use solutions 3, 4, and 5 in Web Polygraph
> benchmark (that can be configured to create lots of outgoing connections).
>
>
> > I have squid v5.5 on Ubuntu with about 48K ephemeral ports available
> > with the ip_local_port_range. The squid is bound to listen on port 3128
> > and has a single tcp_outgoing_address configured. We notice that after
> > about 40-45k concurrent connections through the proxy it is unable to
> > reuse ports and it severely limits local ports available to other
> > applications running on the system. The squid is setup to run 30
> > workers; total CPU is still under 10% during peak connection rates.
> >
> >
> > Is any build config flag required to enable SO_REUSEPORT or SO_REUSEADDR
> > on the outbound TCP sessions opened by squid?
>
> Squid can be configured to use SO_REUSEPORT on _incoming_ connections
> (see *_port worker-queues), but that is not what you are asking about.
> Outside of that worker-queues feature, Squid will not set SO_REUSEPORT
> AFAICT.
>
> Squid does set SO_REUSEADDR unless you use the -R command line option
> AFAICT.
>
>
> > It does not appear that there is an option to use the
> > IP_BIND_ADDRESS_NO_PORT sockopt flag which can help with ephemeral port
> > reuse.
>
> No.
>
>
> > We have tried enabling tcp_tw_reuse, ip_autobind_reuse and
> > ip_nonlocal_bind flags, but unable to get the system reuse the ephemeral
> > ports. The fs.file-max is set to 4M. Pasted some errors below. Any
> > suggestions are appreciated!
>
>
> HTH,
>
> Alex.
>
>
>
> > 2022/05/19 23:35:00 kid12| commBind Cannot bind socket FD 3075 to
> > </IP/>: (99) Cannot assign requested address
> >
> > current master transaction: master48536607
> >
> > 2022/05/19 23:35:00 kid23| commBind Cannot bind socket FD 1320 to
> > </IP/>: (99) Cannot assign requested address
> >
> > current master transaction: master26662366
> >
> >
> > 2022/05/19 23:37:30 kid13| commBind Cannot bind socket FD 3346 to
> > </IP/>: (98) Address already in use
> >
> > current master transaction: master11976056
> >
> > 2022/05/19 23:37:30 kid12| commBind Cannot bind socket FD 6459 to
> > </IP/>: (98) Address already in use
> >
> > current master transaction: master48561031
> >
> >
> > While the system is in this state, local curl’s to another endpoint on
> > the same node are not able to obtain a TCP socket.
> >
> >
> > curl: (7) Couldn't connect to server
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220520/804da239/attachment.htm>