[squid-users] Squid downloading huge amounts of un-requested data

Tue Apr 21 12:01:12 UTC 2015

On 17/04/2015 12:51 p.m., iridium191 wrote:
> Thanks for your response Amos, it is much appreciated. 
> The config is below, with comments excluded - we've done tests in the past
> to confirm it is not an open proxy and don't believe it is. Any commnts you
> may have would also be appreciated.
> The past excessive download events correlated with Microsoft patch Tuesdays
> or in the most recent case deploying a new Windows server and then manually
> updating it, which made us suspect that our refresh rules attempting to
> cache Windows updates was the cause of the problem.
> 
> In the config squidguard should be bypassed for Windows updates and
> squidclamav uses its own whitelist to bypass Windows update sites.

Okay. Noted, and confirmed by the below config.

> 
> Our traffic monitoring so far has been aggregated, so we could see that
> 103GB of http traffic was directed to the squid server from the firewall,
> and of that 15GB came from Microsoft, 12GB from akamai server 1 etc.. You're
> right we didn't consider that something other than squid on the server may
> be causing the requests.
> 

Now that you mention clamav ... I had some issues on my own proxies a
while back where the freshclam auto-update daemon had partially crashed
and on resume was unable to validate the AV updates properly. That
pushed it into a loop of re-downloading the entire virus signatures file
every few minutes - while the file was only a few dozen KB the constant
repeating grew to many GBs over the course of the month before it was
caught.

> The cache utilization report looks interesting in that we may be able to
> script it for more real-time notification of excessive traffic rather than
> relying on the morning firewall report. Are there any definitions of the
> various counters, eg client_http.kbytes_in, client_http.kbytes_in ?

Not that I'm aware of. They should be self-explanatory from the naming
though.

client_http.kbytes_in -> KB received in to Squid from all clients using
HTTP protocol.

The section headers explain how long a time period the counters below it
cover (5min, 60min, totals since last restart, etc).

> 
> Thanks again,
> 
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443         # https
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280                # http-mgmt
> acl Safe_ports port 488                # gss-http
> acl Safe_ports port 591                # filemaker
> acl Safe_ports port 777                # multiling http
> 
> acl CONNECT method CONNECT
> acl ftp proto FTP
> 
> acl manager url_regex -i ^cache_object:// /squid-internal-mgr/
> acl Purge method PURGE
> 
> acl Local_Networks src 10.250.111.0/24 10.250.112.0/24
> acl BypassCache dst 10.250.111.0/24 10.250.112.0/24
> acl BypassCache dst 146.178.211.0/24
> 
> acl BypassCacheDomains dstdomain "/etc/squid3/BypassCacheDomains"
> acl RestrictedUsers proxy_auth "/etc/squid3/RestrictedUsers"
> 
> # ACLs for Windows Updates & other exceptions
> acl WindowsUpdate dstdomain "/etc/squid3/WindowsUpdate"
> acl Whitelist_Domains dstdomain "/etc/squid3/Whitelist_Domains"
> 
> # ACL to allow monitoring of entire proxy chain from 10.250.111.124 without
> authentication 
> acl MonitorProxy src 10.250.111.124/32
> 
> acl Get_Username proxy_auth REQUIRED

The above ACL is unused and does nothing.

> 
> # Bypass squidguard for whitelisted domains
> redirector_access deny Whitelist_Domains
> redirector_access deny WindowsUpdate
> # Bypass squidguard for local sites 
> redirector_access deny BypassCache
> redirector_access deny BypassCacheDomains
> 
> # Bypass connections to local network and TLS
> always_direct allow BypassCache

always_direct does not seem to do what you think it does. All it does is
prevent Squid using a cache_peer to service those requests. They are
still proxied by *this* Squid.

> cache deny BypassCache
> always_direct allow BypassCacheDomains
> cache deny BypassCacheDomains
> 
> http_access allow manager localhost
> http_access allow localhost Purge
> http_access deny manager
> http_access deny Purge
> http_access deny to_localhost
> http_access deny !Local_Networks
> http_access allow Whitelist_Domains
> http_access allow WindowsUpdate
> http_access allow MonitorProxy
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> # Allow connection to HTTPS sites from the local network
> http_access allow CONNECT SSL_ports Local_Networks
> http_access allow ftp
> http_access allow !RestrictedUsers
> 
> http_access deny all
> 
> http_port 8080
> visible_hostname Squid3
> hierarchy_stoplist cgi-bin ?
> 
> # Log file locations
> access_log daemon:/var/log/squid3/access.log squid
> cache_store_log none
> cache_log /var/log/squid3/cache.log
> 
> # Disk cache directory.
> cache_dir aufs /squid_cache/Squid3Cache 25000 16 256
> cache_mem 2000 MB
> maximum_object_size_in_memory 1 MB
> 
> # Windows Update
> #range_offset_limit 200 MB WindowsUpdate 
> maximum_object_size 1 GB
> #quick_abort_min -1
> 
> dns_nameservers 127.0.0.1
> 
> icap_enable on
> icap_send_client_ip on
> icap_send_client_username on
> icap_client_username_encode off
> icap_client_username_header X-Authenticated-User
> icap_preview_enable on
> icap_preview_size 1024
> icap_service service_req reqmod_precache bypass=0
> icap://127.0.0.1:1344/squidclamav
> adaptation_access service_req allow all
> icap_service service_resp respmod_precache bypass=0
> icap://127.0.0.1:1344/squidclamav
> adaptation_access service_resp allow all
> 
> url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
> url_rewrite_children 20 startup=0 idle=1 concurrency=0
> 
> #Do not show client IP address
> via off
> forwarded_for off
> 
> #Rules to anonymize http headers
> request_header_access Allow allow all
> request_header_access Authorization allow all
> request_header_access WWW-Authenticate allow all
> request_header_access Proxy-Authorization allow all
> request_header_access Proxy-Authenticate allow all
> request_header_access Content-Encoding allow all
> request_header_access Content-Length allow all
> request_header_access Content-Type allow all
> request_header_access Date allow all
> request_header_access Expires allow all
> request_header_access Host allow all
> request_header_access If-Modified-Since allow all
> request_header_access Last-Modified allow all
> request_header_access Location allow all
> request_header_access Pragma allow all
> request_header_access Accept allow all
> request_header_access Accept-Charset allow all
> request_header_access Accept-Encoding allow all
> request_header_access Accept-Language allow all
> request_header_access Content-Language allow all
> request_header_access Mime-Version allow all
> request_header_access Retry-After allow all
> request_header_access Title allow all
> request_header_access Connection allow all
> request_header_access Proxy-Connection allow all
> request_header_access Cookie allow all
> ###request_header_access All deny all

These ones are response-only headers and you can remove from the list:
 Last-Modified, Location, Retry-After, Date, WWW-Authenticate,
Proxy-Authenticate, Expires

I recommend adding the Expect, ETag, TE, Transfer-Encoding, If-Match,
If-None-Match, If-Unmodified-Since, Range and If-Range headers to the
above allow lists. That will allow HTTP/1.1 persistent connections and
revalidations to work a lot better.

Come to think of it the Range/If-Range and ETag not being allowed is
probably related to your problem.

You can remove Proxy-Connection. Its an obsolete header Squid does not
emit. Mime-Version and Title are also pretty useless unless you have
WebDAV clients.

Accept-Charset and Accept-Language are not commonly useful and have a
large impact on anonymity. Removing them from your allow list could be
beneficial.

Amos