[squid-users] Running SMP workers on multi-core systems

[Alex Rousskov]

On 10/1/21 2:37 AM, Praveen Ponakanti wrote:

> when I attempt to use the following config with 5.1 the cache log
> says that I need to run 'squid -z' to create the swap directories. Is
> that something I would need to add to my Docker build steps so that it
> is set up before the squid server is run? 

The exact solution is up to you, but Squid cannot be configured to
automatically initialize cache_dirs for you (yet) -- something must run
"squid -z" before a new cache_dir directory can be used by a regular
Squid instance.

> "cache_dir ufs /squid/var/cache/squid 400000 16 256"

Please note that "ufs" cache_dirs are not compatible with SMP Squids.
SMP Squid behavior with UFS-based cache_dirs is undefined. If you need
disk caching, use "rock" cache_dirs instead.

Alex.


> 2021/09/30 22:27:39 kid4| FATAL: Failed to verify one of the swap
> directories, Check cache.log
> for details.  Run 'squid -z' to create swap directories
> if needed, or if running Squid for the first time.


> On Thu, Sep 30, 2021 at 6:30 AM Alex Rousskov wrote:
> 
>     On 9/30/21 2:49 AM, Praveen Ponakanti wrote:
> 
>     > I am trying to use the squid cache as an outbound HTTP/S proxy service
>     > for our production network and would like to scale up the request per
>     > sec by running it on multi-core systems. 
> 
>     > I have searched up the archives and config docs, but couldn't find
>     > anything more recent than the below on setting up SMP workers. 
>     > https://wiki.squid-cache.org/ConfigExamples/SmpCarpCluster
>     <https://wiki.squid-cache.org/ConfigExamples/SmpCarpCluster>
>     > https://wiki.squid-cache.org/ConfigExamples/MultiCpuSystem
>     <https://wiki.squid-cache.org/ConfigExamples/MultiCpuSystem>
> 
>     The two old exotic configurations above are not what you should be
>     starting with when thinking about SMP support in modern Squids. The best
>     starting points I know about are
> 
>     * https://wiki.squid-cache.org/Features/SmpScale
>     <https://wiki.squid-cache.org/Features/SmpScale>
>     * http://www.squid-cache.org/Doc/config/workers/
>     <http://www.squid-cache.org/Doc/config/workers/>
>     * http://www.squid-cache.org/Doc/config/cpu_affinity_map/
>     <http://www.squid-cache.org/Doc/config/cpu_affinity_map/>
>     * http(s)_port worker-queues option
> 
>     That Feature wiki page does not get many updates these days; if there is
>     a conflict between information sources, the directive documentation in
>     your squid.conf.documented may have more recent information.
> 
> 
>     > * Can someone please let me know if the config examples in those docs
>     > still apply to the latest version 5.1.
> 
>     I bet those use cases themselves do not apply well to your situation.
> 
> 
>     > * Do we still need the combination of squid, frontend, backend.conf
>     > files, or is it enabled simply with the "workers" line in the main
>     > config file? 
> 
>     The workers directive does not require multiple Squid instances (with
>     multiple configuration files, etc.).
> 
> 
>     > * When I attempt to run the squid server with "workers 4" in the
>     > squid.conf, I get the following error (one for each kid) and the main
>     > process does not bind to the "http_port 3128". I see several UDP ports
>     > opened up for each kid process.
> 
>     > 2021/09/29 00:44:10 kid5| commBind Cannot bind socket FD 11 to
>     [::]: (2) No such file or directory
> 
>     Unfortunately, I cannot tell exactly what went wrong based on that
>     low-level message alone, but it could be a variant of [1]. If you cannot
>     figure it out after checking [1] suggestions, consider sharing "squid
>     -X" startup cache.log for analysis. There is
> 
>     [1]
>     https://wiki.squid-cache.org/Features/SmpScale#Cannot_bind_socket_FD_NN_to_.5B::.5D:_.2813.29_Permission_denied
>     <https://wiki.squid-cache.org/Features/SmpScale#Cannot_bind_socket_FD_NN_to_.5B::.5D:_.2813.29_Permission_denied>
> 
>     The UDP ports you are seeing are probably for the internal DNS resolver
>     which is not SMP-aware.
> 
> 
>     > * Does anyone have recommendations on the maximum number of workers to
>     > use on a 64 core host (assuming no other CPU intensive apps are
>     running
>     > on the same host). Caching is not a must for our initial
>     deployment, so
>     > we are fine disabling caching.
> 
>     28-30. See the following wiki section for the corresponding rules of
>     thumb:
>     https://wiki.squid-cache.org/Features/SmpScale#How_to_configure_SMP_Squid_for_top_performance.3F
>     <https://wiki.squid-cache.org/Features/SmpScale#How_to_configure_SMP_Squid_for_top_performance.3F>
> 
> 
>     HTH,
> 
>     Alex.
>