[squid-users] How to enable squid to use more server resources when using 500+ http_ports?

[Amos Jeffries]

TL;DR this behaviour is expected when expanding beyond the default limit 
on http_port lines permitted in squid.conf. YMMV as to how much you can 
expand, but it will appear at some point.



On 29/08/22 18:06, Marcelo wrote:
> Hello,
> 
> I have a server with large resources ( RAM, CPU, etc) running Ubuntu 20.04.
> 
> When I put squid 4.13 to manage more than 500 http_ports, it performs 
> very slowly.
> 
> Question is, how can I setup Squid or Ubuntu to use more hw resources in 
> order to perform better?

If the behaviour only occurs once you expand the listening ports the 
solution is to reduce the number of listening ports and spread over more 
Squid (on more hardware/machines).


The hard coded default is small for three reasons that I am aware of:

  1) these ports need checking frequently even when no I/O is expected 
to happen. So they are checked every 10ms in addition to the regular I/O 
traffic.

  2) it was set long ago when hardware was more restricted. So YMMV and 
testing like you are doing is recommended when touching it.

  3) most Squid users do not need anywhere even close to the limit set. 
So we are not inclined to change the default value - instead we allow 
custom builds to set themselves if necessary.


That said the speed situation you are experiencing may be more related 
to how many new connections (and thus new HTTP messages to process) can 
be opened per second with X listening ports vs X+Y listening ports. 
Squid does have capacity limits at many points of the traffic processing.


> 
> Or does it a matter of software architecture limitation?
> 
> Perform well with at least 500 http_ports would be very nice so we can 
> use smaller numbers of VPSs.
> 

On a good modern machine this is know to be within the realms of 
reasonable, but it does impact the overall performance per (1) above.


> Its hard to manage so many of it.

Indeed. Another reason not to.

That said the "include" directive and a folder of per-port config files 
was created exactly to help with this type of management problem. 
Automated management of a set of per-port config files can easily be 
setup with popular network management tools and/or scripts.


> It would be a dream to have like 10k http_ports in a single server.

Very unlikely to be doable. Each port used this way reduces the number 
of ports available for client and server transactional traffic.


> 
> If it is not possible, is there any planning about do it in a future?
> 

Not specifically. Due to reason (3) above.


Amos