[squid-users] Delay_pools problem in Squid 3.5.20

Wed Jul 26 05:35:50 UTC 2017

On 26/07/17 13:49, Alex Tang wrote:
> Dear All,
> 
> I had installed a squid 3.5.20 on Centos7. I'm also setting up a 
> delay_pools and cache_peer function on this squid server .
> 
> but I don't know why delay_pools function not working now. before I 
> tested is OK and I can limit user download less then 512 kb. but now, 
> all user download is over 512kb and can't control. I tried reboot the 
> squid or use the other delay_pools command to try to test which part has 
> problem to cause the delay_pools problem. I've checked my config file 
> seem OK (this is my first time to install and use squid proxy server) , 
> would you mind tell me is it my config problem or squid bug (maybe)
> 

What do you mean exactly by "before I tested is OK". The same Squid 
version? the same config file?

> here is my full config file.
> 
> ====================================================
> #
> # Recommended minimum configuration:
> #
> 
> # Example rule allowing access from your local networks.
> # Adapt to list your (internal) IP networks from where browsing
> # should be allowed
> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8># RFC1918 possible 
> internal network
> acl localnet src 111.11.0.0/12 <http://111.11.0.0/12># RFC1918 possible 
> internal network
> acl localnet src 11.123.0.0/16 <http://11.123.0.0/16># RFC1918 possible 
> internal network

Ah, 111/8 and 11/8 are not RFC 1918 network ranges. If you have been 
assigned those ranges and are using them internally that is fine, but 
remove the RFC1918 comment to avoid confusing anyone about them.

> acl localnet src fc00::/7       # RFC 4193 local private network range
> acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) 
> machines
> 
> acl SSL_ports port 443 563
> acl Safe_ports port 80# http
> acl Safe_ports port 21# ftp
> #acl Safe_ports port 443# https
> acl Safe_ports port 70# gopher
> acl Safe_ports port 210# wais
> #acl Safe_ports port 1025-65535# unregistered ports
> acl Safe_ports port 280# http-mgmt
> acl Safe_ports port 488# gss-http
> acl Safe_ports port 591# filemaker
> acl Safe_ports port 777# multiling http
> #acl CONNECT method CONNECT
> 
> 
> 
> #
> # Recommended minimum Access Permission configuration:
> #
> # Deny requests to certain unsafe ports
> #http_access allow !Safe_ports
> 
> # Deny CONNECT to other than secure SSL ports
> #http_access allow CONNECT !SSL_ports
> 

Do not do the above commenting-out of those lines. They are the basic 
security protection against certain types of DoS and attacks on your 
proxy. Which is also why they should always be first.

> # Only allow cachemgr access from localhost
> http_access allow localhost manager
> http_access deny manager
> 
> # We strongly recommend the following be uncommented to protect innocent
> # web applications running on the proxy server who think the only
> # one who can access services on "localhost" is a local user
> #http_access deny to_localhost
> 
> #
> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
> #
> 
> # Example rule allowing access from your local networks.
> # Adapt localnet in the ACL section to list your (internal) IP networks
> # from where browsing should be allowed
> http_access allow localnet
> http_access allow localhost
> 
> # And finally deny all other access to this proxy
> http_access deny all
> 
> icp_port 3130
 >
> # Leave coredumps in the first cache dir
> coredump_dir /var/spool/squid
> 
> #
> # Add any of your own refresh_pattern entries above these.
> #
> #refresh_pattern ^ftp:144020%10080
> #refresh_pattern ^gopher:14400%1440
> #refresh_pattern -i (/cgi-bin/|\?) 00%0
> #refresh_pattern .020%4320

The above refresh_pattern lines are tuned specifically for HTTP/1.1 
required behaviour with dynamic content. If you are caching I highly 
recommend leaving them active. If you are not using caching they are 
irrelevant.

> 
> http_port 8000 name=port_8000
> http_port 8001 name=port_8001
> 
> nonhierarchical_direct off
> 
> acl port_8000_acl myportname port_8000
> acl port_8001_acl myportname port_8001
> 
> always_direct deny port_8000_acl
> always_direct deny port_8001_acl
> 
> never_direct allow port_8000_acl
> never_direct allow port_8001_acl
> 

The above are redundant. always_direct overrides never_direct. The 
normal practice is to use only never_direct to forbid use of DNS 
identified origin servers and leave the cache_peer as an available routing.

> # 8000
> cache_peer xxx.xxxx.com parent 8000 3130 weight=20 
> no-digest no-query name=proxy8000
> cache_peer_access proxy8000 allow port_8000_acl
> cache_peer_access proxy8000 deny all
> 
> # 8001
> cache_peer xxx.xxxx.com parent 8001 3130 weight=20 
> no-digest no-query name=proxy8001
> cache_peer_access proxy8001 allow port_8001_acl
> cache_peer_access proxy8001 deny all
> 
> cache_mem 100 MB
> cache_swap_low 90
> cache_swap_high 95
> 
> 
> acl work_day time MTWHFAS 09:00-18:30
> acl BBHK src 11.123.0.0/16
> delay_pools 1
> delay_class 1 2
> delay_parameters 1 -1/-1 512000/512000
> delay_access 1 allow work_day
> delay_access 1 allow BBHK

Three things to be careful about. I order them here based on the 
likelihood that they are your problem:

  1) the work_day times are in 'local time', according to whatever the 
local wall-clock of your Squid machine is set to.

   This local time detail can catch people out if the production proxy 
is set to use UTC as its local time, or is hosted in another timezone 
(ie cloud service).

  2) delay pools operate by limiting Squid->server connection traffic. 
That means that cache HITs are not delayed. Also, HTTP/1.1 revalidations 
use very much less server connection bytes compared to client connection 
bytes. So the delay may have the appearance of not working, even when 
working perfectly.
   For proper bandwidth shaping it is best to use your system QoS 
functionality, not Squid delay pools. Especially given your criteria 
does not actually use any HTTP layer details to classify the traffic for 
delay.

  3) these lines form an OR condition. The BBHK are *always* delayed, 
and so is every client during the work_day times.

Check carefully where (from what client IPs to what proxy IPs), how, and 
when your previous working tests were done compared to the current 
non-working situation.

  4) the delay pools measure traffic in Bytes (KB or kB) not bits (Kb or 
kb). KB/sec limits are 8x larger than kb/sec.

I assume your "kb" word was just sloppy typing. But if you did mean bits 
this could be affecting your measurement results.

> 
> acl QUERY urlpath_regex cgi-bin \?
> no_cache deny QUERY

The above is a misconfiguration in Squid-3. The refresh_pattern which 
you commented out are the correct way to prevent broken dynamic content 
caching.

> maximum_object_size 2048 KB
> ipcache_size 1024
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 1024
> logformat squid      %ts.%03tu %6tr %>A %Ss/%03>Hs %<st %rm %ru %[un 
> %Sh/%<a %mt

Please do not re-define the "squid" format name. Current Squid releases 
will ignore your definition, and the log will not end up with what you 
want in it (the real built-in format does not have the exact output in 
some cases as those %-codes perform).
  Either use the built-in format (removing those above line), or use a 
different format name for your custom format.

> cache_access_log /var/log/squid/access.log

cache_access_log is long ago deprecated.

Use access_log instead, like so:
  access_log stdio:/var/log/access.log

or for better performance:
  access_log daemon:/var/log/access.log

> cache_log /var/log/squid/cache.log
> cache_store_log none
> pid_filename /var/run/squid.pid
> auth_param ntlm program /usr/bin/ntlm_auth 
> --helper-protocol=squid-2.5-ntlmssp
> auth_param ntlm children 30
> auth_param basic program /usr/bin/ntlm_auth 
> --helper-protocol=squid-2.5-basic
> auth_param basic children 30
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours
> request_body_max_size 15 MB
> 

NTLM plus delay pools. Ouch.

> 
> acl AuthorizedUsers proxy_auth REQUIRED
> acl shockwaveplayer browser Shockwave
> acl Java browser Java/1.4 Java/1.5 Java/1.6
> acl BBhknet src "/etc/squid/ACL/allow_net"
> acl allow_ip src "/etc/squid/ACL/allow_ip"
> #acl deny_ip src "/etc/squid/ACL/deny_ip"
> #acl DenyUsers proxy_auth "/etc/squid/ACL/deny_users"
> acl allow_pattern dstdom_regex "/etc/squid/ACL/allow_domain"
> 
> acl allow_sites url_regex "/etc/squid/ACL/allow_url"
> #http_access allow allow_sites
> 
> acl deny_site url_regex "/etc/squid/ACL/deny_url"
> #http_access deny deny_site
> 
> acl deny_pattern dstdom_regex "/etc/squid/ACL/deny_domain"
> #http_access deny deny_pattern
> 
> acl deny_domain dstdom_regex "/etc/squid/ACL/deny_domain"
> acl deny_url url_regex "/etc/squid/ACL/deny_url"
> 
> acl fwdurl dstdomain .salesforce.com <http://salesforce.com> .force.com 
> <http://force.com>
> never_direct allow fwdurl
> 
> acl fwdhkpurl dstdomain "/etc/squid/ACL/fwdhkpdomain"
> never_direct allow fwdhkpurl
> 
> 
> http_access allow fwdurl
> http_access allow fwdhkpurl

Note: The fwdurl and fwdhkpurl ACL are the same type, and used together 
for exactly the same things. You can simplify your config by merging 
them into one ACL name like so:
  acl fwdurl dstdomain .salesforce.com .force.com
  acl fwdurl dstdomain "/etc/squid/ACL/fwdhkpdomain"

then removing the *_access lines using fwdhkpurl.

> http_access allow shockwaveplayer
> http_access allow Java
> http_access allow allow_ip
> http_access allow allow_pattern
> http_access allow allow_sites
> http_access deny deny_domain
> http_access allow BBhknet
> http_access allow BBHK
> http_access deny all
> 
> 
> http_reply_access allow all
> icp_access allow all
> miss_access allow all
> cache_mgr BBHK Network Admin Email > cache_effective_user squid
> cache_effective_group squid
> visible_hostname proxy6.hkg.xerox.com
> max_filedesc 4096
> never_direct allow all
> error_directory /usr/share/squid/errors/English
> coredump_dir /var/spool/squid

Most of the above are default config settings. You can simplify your 
config by removing the default lines.

squid -k parse should give you a lot of hints about improvements as 
well. That is always being improved, so a quick -k parse after upgrades 
is recommended best practice.

Amos