<div dir="ltr"><div>Hi,</div><div><br></div><div>Moving this thread to the squid-dev list as the PR below has been opened to propose a fix to the ephemeral port exhaustion problem we had experienced with ver 5.5. </div><div><a href="https://github.com/squid-cache/squid/pull/1115">https://github.com/squid-cache/squid/pull/1115</a><br></div><div><br></div><div>Thanks to Alex and others for providing pointers in this regards. Apologies if I have missed out any requirements on the PR, please let me know and I can update as needed. The fix proposed is based on the post below from Cloudflare that had experienced this issue with their own applications. </div><div><a href="https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/">https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/</a><br></div><div><br></div><div>We were hitting the errors stated on the PR as soon as the system exhausted 1x <span style="color:rgb(36,41,47);font-family:-apple-system,"system-ui","Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px">ip_local_port_range concurrent TCP sessions. </span>With this fix, we have been able to scale well beyond that ( > 100k concurrent outbound TCP sessions). This patch has been running in our production environment for close to 2 months without any new issues.</div><div><br></div><div>Please let me know if you have any questions, and again apologize if I forgot to include something before or with the PR.</div><div><br></div><div>Thanks</div><div>Praveen</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <strong class="gmail_sendername" dir="auto">Alex Rousskov</strong> <span dir="auto"><<a href="mailto:rousskov@measurement-factory.com">rousskov@measurement-factory.com</a>></span><br>Date: Tue, Jun 21, 2022 at 2:11 PM<br>Subject: Re: [squid-users] Scaling concurrent TCP sessions beyond ephemeral port range<br>To: <<a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a>><br></div><br><br>On 6/19/22 12:48, Praveen Ponakanti wrote:<br>
<br>
> What is the process to have this code patch upstreamed for future squid <br>
> versions?<br>
<br>
In short, just post a quality pull request on GitHub (or find somebody <br>
who can guide your code towards official acceptance for you). For <br>
details, please see <a href="https://wiki.squid-cache.org/MergeProcedure" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/MergeProcedure</a><br>
<br>
<br>
Thank you,<br>
<br>
Alex.<br>
<br>
<br>
> On Fri, May 20, 2022 at 9:31 PM Amos Jeffries <<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a> <br>
> wrote:<br>
> <br>
> On 20/05/22 19:44, Praveen Ponakanti wrote:<br>
> > Hi Alex,<br>
> ><br>
> > Thanks for going through several steps to help mitigate src port<br>
> > exhaustion. We are looking to achieve 400-500% more<br>
> > concurrent connections if we could :) as there is a<br>
> significant buffer<br>
> > on the available CPU.<br>
> <br>
> Then you require at least 4, maybe 5, IP addresses to handle that many<br>
> concurrent connections with Squid.<br>
> <br>
> <br>
> We would like to investigate going beyond the ephemeral port range for <br>
> some specific destination IP:PORT addresses. For that it appears squid <br>
> does not round-robin requests if we use multiple tcp_outgoing_addresses. <br>
> We could use ACL’s to pick a different outbound IP based on the clients <br>
> source IP, however that is not very ideal in our environment as our <br>
> clients aren’t always equally split by subnet. However, if we could <br>
> split by the client’s source port that might help achieve this. For <br>
> example something like:<br>
> <br>
> <br>
> acl pool1 clientport 0-32768<br>
> <br>
> acl pool2 clientport 32769-65536<br>
> <br>
> <br>
> tcp_outgoing_address 10.1.0.1 pool1<br>
> <br>
> tcp_outgoing_address 10.1.0.2 pool2<br>
> <br>
> <br>
> Squid's ACLs currently do not allow filtering by the client's source <br>
> port. We could look into a separate patch to add this functionality to <br>
> squid’s ACL code if that makes sense. Or is there a better way to <br>
> achieve this?<br>
> <br>
> <br>
> Thanks<br>
> <br>
> Praveen<br>
> <br>
> <br>
> > The option to use multiple tcp_outoing_addresses appears to be<br>
> promising<br>
> > along with some tweaks to the TCP timeouts. I guess we could use<br>
> ACLs to<br>
> > pick a different outbound IP based on the requesting client's<br>
> prefix. We<br>
> > had not considered that option as the ephemeral ports were no longer<br>
> > available to other applications when squid uses most of them with a<br>
> > single outbound IP configured. We are also looking to modify the<br>
> code to<br>
> > use the IP_BIND_ADDRESS_NO_PORT sockopt as that could help delay<br>
> port<br>
> > assignment with the bind() call on the outbound TCP sessions (to<br>
> > hopefully allow access to the 4-tuple on the socket).<br>
> <br>
> Patches welcome.<br>
> <br>
> However, please be aware that use of the 4-tuple is often no different<br>
> from the 3-tuple since the dst-port is typically identical for all<br>
> outgoing traffic to a given dst-IP.<br>
> <br>
> <br>
> Cheers<br>
> Amos<br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <mailto:<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a>><br>
> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
> <<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a>><br>
> <br>
> <br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</div></div></div>