<div dir="ltr"><div>Hi Alex,</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Do we have a recent squid ver 6 snapshot build available for testing? Looking for something that includes the patch from the PR to introduce the ip_bind_address_noport socket option on outbound TCP connections, I dont see any new builds after Sep 6th.</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><a href="http://www.squid-cache.org/Versions/v6/">http://www.squid-cache.org/Versions/v6/</a><br></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><br></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">We have been using the last commit (prior to merge) from my branch on a few canary instances and have observed a memory leak with squid 6.0.0. The memory usage by squid with that code has grown by almost 20G over the last 2+ months, while other v5.5 squids (with a local patch for adding that socket option) have not had that significant of an increase. </p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><br></p></div>Thanks for your insights below, apologies for not being able to try your suggestions until recently. I hadn't looked at the coordinator in depth and forgotten that squid workers are not threads. <div>The following config knobs were tried and did not make much of a difference with respect to concurrent outbound TLS sessions across the workers. The docs say the default sslproxy_session_cache_size is 2 MB, how high can we go? Are there any other knobs we can try to improve session reuse for HTTPS reqs ? (without enabling squid cache)</div><div><br></div><div>sslproxy_session_cache_size 10 MB<br>tls_outgoing_options options=NO_TICKET</div><div><br><div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(80,0,80)">> Agree, it might not make sense to increase the complexity with sharing</span><br style="color:rgb(80,0,80)"><span style="color:rgb(80,0,80)">> socket among the workers. Was thinking more on the lines of a hashmap</span><br style="color:rgb(80,0,80)"><span style="color:rgb(80,0,80)">> that the coordinator could use to pick workers that already have a TCP</span><br style="color:rgb(80,0,80)"><span style="color:rgb(80,0,80)">> connection to the destination being requested, instead of having the</span><br style="color:rgb(80,0,80)"><span style="color:rgb(80,0,80)">> workers themselves share connection details.</span></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>Coordinator does not receive/see regular HTTP traffic. If we start <br>
routing HTTP transactions through that process, it may become the <br>
bottleneck itself _and_ will introduce additional overheads for passing <br>
descriptors to workers. From performance point of view, the model with <br>
one "routing" task doling work to workers works best (and is commonly <br>
used) in threaded applications, but Squid is not threaded at that level.<br> <br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
> Most of the TCP connections are for HTTPS reqs, w/o TLS termination at <br>
> the squid. Does squid currently support a TLS session cache ?<br>
<br>
Yes, there is some support for worker-specific TLS session caching, with <br>
directives like sslproxy_session_cache_size, tls_outgoing_options <br>
options=NO_TICKET (for outgoing sessions IIRC) and https_port <br>
sslflags=NO_SESSION_REUSE and https_port sslcontext (for incoming sessions).<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
HTH,<br>
<br>
Alex.<br>
<br>
<br>
> If you are dealing with TLS sessions as well, then we should add a<br>
> shared memory TLS session cache that all workers can tap into.<br>
> <br>
> <br>
> Cheers,<br>
> <br>
> Alex.<br>
> <br>
> > On Tue, Jun 21, 2022 at 2:11 PM Alex Rousskov wrote:<br>
> ><br>
> > On 6/19/22 12:48, Praveen Ponakanti wrote:<br>
> ><br>
> > > What is the process to have this code patch upstreamed for<br>
> future<br>
> > squid<br>
> > > versions?<br>
> ><br>
> > In short, just post a quality pull request on GitHub (or find<br>
> somebody<br>
> > who can guide your code towards official acceptance for you). For<br>
> > details, please see<br>
> <a href="https://wiki.squid-cache.org/MergeProcedure" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/MergeProcedure</a><br>
> <<a href="https://wiki.squid-cache.org/MergeProcedure" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/MergeProcedure</a>><br>
> > <<a href="https://wiki.squid-cache.org/MergeProcedure" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/MergeProcedure</a><br>
> <<a href="https://wiki.squid-cache.org/MergeProcedure" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/MergeProcedure</a>>><br>
> ><br>
> ><br>
> > Thank you,<br>
> ><br>
> > Alex.<br>
> ><br>
> ><br>
> > > On Fri, May 20, 2022 at 9:31 PM Amos Jeffries<br>
> > <<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a> <mailto:<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>><br>
> <mailto:<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a> <mailto:<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>>><br>
> > > wrote:<br>
> > ><br>
> > > On 20/05/22 19:44, Praveen Ponakanti wrote:<br>
> > > > Hi Alex,<br>
> > > ><br>
> > > > Thanks for going through several steps to help mitigate<br>
> > src port<br>
> > > > exhaustion. We are looking to achieve 400-500% more<br>
> > > > concurrent connections if we could :) as there is a<br>
> > > significant buffer<br>
> > > > on the available CPU.<br>
> > ><br>
> > > Then you require at least 4, maybe 5, IP addresses to<br>
> handle<br>
> > that many<br>
> > > concurrent connections with Squid.<br>
> > ><br>
> > ><br>
> > > We would like to investigate going beyond the ephemeral port<br>
> > range for<br>
> > > some specific destination IP:PORT addresses. For that it<br>
> appears<br>
> > squid<br>
> > > does not round-robin requests if we use multiple<br>
> > tcp_outgoing_addresses.<br>
> > > We could use ACL’s to pick a different outbound IP based<br>
> on the<br>
> > clients<br>
> > > source IP, however that is not very ideal in our<br>
> environment as our<br>
> > > clients aren’t always equally split by subnet. However, if<br>
> we could<br>
> > > split by the client’s source port that might help achieve<br>
> this. For<br>
> > > example something like:<br>
> > ><br>
> > ><br>
> > > acl pool1 clientport 0-32768<br>
> > ><br>
> > > acl pool2 clientport 32769-65536<br>
> > ><br>
> > ><br>
> > > tcp_outgoing_address 10.1.0.1 pool1<br>
> > ><br>
> > > tcp_outgoing_address 10.1.0.2 pool2<br>
> > ><br>
> > ><br>
> > > Squid's ACLs currently do not allow filtering by the<br>
> client's source<br>
> > > port. We could look into a separate patch to add this<br>
> > functionality to<br>
> > > squid’s ACL code if that makes sense. Or is there a better<br>
> way to<br>
> > > achieve this?<br>
> > ><br>
> > ><br>
> > > Thanks<br>
> > ><br>
> > > Praveen<br>
> > ><br>
> > ><br>
> > > > The option to use multiple tcp_outoing_addresses<br>
> appears to be<br>
> > > promising<br>
> > > > along with some tweaks to the TCP timeouts. I guess we<br>
> > could use<br>
> > > ACLs to<br>
> > > > pick a different outbound IP based on the<br>
> requesting client's<br>
> > > prefix. We<br>
> > > > had not considered that option as the ephemeral<br>
> ports were<br>
> > no longer<br>
> > > > available to other applications when squid uses most of<br>
> > them with a<br>
> > > > single outbound IP configured. We are also looking to<br>
> > modify the<br>
> > > code to<br>
> > > > use the IP_BIND_ADDRESS_NO_PORT sockopt as that<br>
> could help<br>
> > delay<br>
> > > port<br>
> > > > assignment with the bind() call on the outbound TCP<br>
> > sessions (to<br>
> > > > hopefully allow access to the 4-tuple on the socket).<br>
> > ><br>
> > > Patches welcome.<br>
> > ><br>
> > > However, please be aware that use of the 4-tuple is<br>
> often no<br>
> > different<br>
> > > from the 3-tuple since the dst-port is typically identical<br>
> > for all<br>
> > > outgoing traffic to a given dst-IP.<br>
> > ><br>
> > ><br>
> > > Cheers<br>
> > > Amos<br>
<br>
</blockquote></div></div></div></div>