[squid-users] Cache Peers and traffic handling

[Alex Rousskov]

On 4/14/21 2:29 AM, koshik moshik wrote:
> Thank you! Yes, it works fine with 5 peers. So, what would be the best
> solution to handle 5000 peers? 

As you can tell by other responses, you might be asking the wrong
question. However, I will still try to answer your question. The best
option may be to add support for a new Squid configuration parameter
that tells Squid to limit cache_peer candidate accumulation to N peers,
effectively making all those linear searches fast.

https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F

Alex.


> On Mon, Apr 12, 2021 at 6:03 PM Alex Rousskov wrote:
> 
>     On 4/10/21 5:03 PM, koshik moshik wrote:
> 
>     > I am trying to run a Squid proxy Server witth about 5000 cache
>     peers. I
>     > am running a dedicated server with 6 cores and 32GB RAM on Ubuntu 16. 
>     >
>     >
>     > Could you tell me what else is needed / not needed in my
>     squid.config? I
>     > am encountering a high CPU usage and would like to create a very
>     > efficient proxy server.
> 
>     IIRC, Squid code is not optimized for handling a large number of
>     cache_peers: Several cache peer selection steps involve linear searches.
> 
>     I do not know what exactly causes high CPU usage in your environment but
>     it could be those linear searches. You can test that (indirectly) by
>     decreasing the number of cache_peers from 5000 to, say, 5. That is a
>     weak test, of course, because other cache_peer-related overheads could
>     be to blame, but I would start there.
> 
> 
>     HTH,
> 
>     Alex.
> 
> 
> 
>     > Down below you can find my squid.config(I deleted the other cache_peer
>     > lines):
>     >
>     > -----------
>     >
>     > http_port 3128
>     >
>     > dns_v4_first on
>     >
>     > acl SSL_ports port 1-65535
>     >
>     > acl Safe_ports port 1-65535
>     >
>     > acl CONNECT method CONNECT
>     >
>     > http_access deny !Safe_ports
>     >
>     > http_access deny CONNECT !SSL_ports
>     >
>     > auth_param basic program /usr/lib/squid/basic_ncsa_auth
>     /etc/squid/.htpasswd
>     >
>     > auth_param basic children 5
>     >
>     > auth_param basic realm Squid Basic Authentication
>     >
>     > auth_param basic credentialsttl 5 hours
>     >
>     > acl password proxy_auth REQUIRED
>     >
>     > http_access allow password
>     >
>     > #http_access deny all
>     >
>     > cache allow all
>     >
>     > never_direct allow all
>     >
>     > ident_access deny all
>     >
>     >
>     >
>     >
>     >
>     > cache_mem 1 GB
>     >
>     > maximum_object_size_in_memory 16 MB
>     >
>     >
>     >
>     >
>     >
>     > # Leave coredumps in the first cache dir
>     >
>     > coredump_dir /var/spool/squid
>     >
>     >
>     > #Rules to anonymize http headers
>     >
>     > forwarded_for off
>     >
>     > request_header_access Allow allow all
>     >
>     > request_header_access Authorization allow all
>     >
>     > request_header_access WWW-Authenticate allow all
>     >
>     > request_header_access Proxy-Authorization allow all
>     >
>     > request_header_access Proxy-Authenticate allow all
>     >
>     > request_header_access Cache-Control allow all
>     >
>     > request_header_access Content-Encoding allow all
>     >
>     > request_header_access Content-Length allow all
>     >
>     > request_header_access Content-Type allow all
>     >
>     > request_header_access Date allow all
>     >
>     > request_header_access Expires allow all
>     >
>     > request_header_access Host allow all
>     >
>     > request_header_access If-Modified-Since allow all
>     >
>     > request_header_access Last-Modified allow all
>     >
>     > request_header_access Location allow all
>     >
>     > request_header_access Pragma allow all
>     >
>     > request_header_access Accept allow all
>     >
>     > request_header_access Accept-Charset allow all
>     >
>     > request_header_access Accept-Encoding allow all
>     >
>     > request_header_access Accept-Language allow all
>     >
>     > request_header_access Content-Language allow all
>     >
>     > request_header_access Mime-Version allow all
>     >
>     > request_header_access Retry-After allow all
>     >
>     > request_header_access Title allow all
>     >
>     > request_header_access Connection allow all
>     >
>     > request_header_access Proxy-Connection allow all
>     >
>     > request_header_access User-Agent allow all
>     >
>     > request_header_access Cookie allow all
>     >
>     > request_header_access All deny all
>     >
>     >
>     >
>     >
>     >
>     > #
>     >
>     > # Add any of your own refresh_pattern entries above these.
>     >
>     > #
>     >
>     > #refresh_pattern ^ftp:           1440    20%     10080
>     >
>     > #refresh_pattern ^gopher:        1440    0%      1440
>     >
>     > #refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
>     >
>     > #refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
>     >
>     > #refresh_pattern .               0       20%     4320
>     >
>     >
>     > ################################
>     >
>     > acl me proxy_auth ye-1
>     >
>     > cache_peer my.proxy.com <http://my.proxy.com>
>     <http://my.proxy.com/ <http://my.proxy.com/>> parent 31280
>     > login=user1:password1 no-query name=a1
>     >
>     > cache_peer_access a1 allow me
>     >
>     > cache_peer_access a1 deny all
>     >
>     >
>     > _______________________________________________
>     > squid-users mailing list
>     > squid-users at lists.squid-cache.org
>     <mailto:squid-users at lists.squid-cache.org>
>     > http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
>     >
>