[squid-users] Squid downloading huge amounts of un-requested data

Fri Apr 17 00:51:39 UTC 2015

Thanks for your response Amos, it is much appreciated. 
The config is below, with comments excluded - we've done tests in the past
to confirm it is not an open proxy and don't believe it is. Any commnts you
may have would also be appreciated.
The past excessive download events correlated with Microsoft patch Tuesdays
or in the most recent case deploying a new Windows server and then manually
updating it, which made us suspect that our refresh rules attempting to
cache Windows updates was the cause of the problem.

In the config squidguard should be bypassed for Windows updates and
squidclamav uses its own whitelist to bypass Windows update sites.

Our traffic monitoring so far has been aggregated, so we could see that
103GB of http traffic was directed to the squid server from the firewall,
and of that 15GB came from Microsoft, 12GB from akamai server 1 etc.. You're
right we didn't consider that something other than squid on the server may
be causing the requests.

The cache utilization report looks interesting in that we may be able to
script it for more real-time notification of excessive traffic rather than
relying on the morning firewall report. Are there any definitions of the
various counters, eg client_http.kbytes_in, client_http.kbytes_in ?

Thanks again,

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280                # http-mgmt
acl Safe_ports port 488                # gss-http
acl Safe_ports port 591                # filemaker
acl Safe_ports port 777                # multiling http

acl CONNECT method CONNECT
acl ftp proto FTP

acl manager url_regex -i ^cache_object:// /squid-internal-mgr/
acl Purge method PURGE

acl Local_Networks src 10.250.111.0/24 10.250.112.0/24
acl BypassCache dst 10.250.111.0/24 10.250.112.0/24
acl BypassCache dst 146.178.211.0/24

acl BypassCacheDomains dstdomain "/etc/squid3/BypassCacheDomains"
acl RestrictedUsers proxy_auth "/etc/squid3/RestrictedUsers"

# ACLs for Windows Updates & other exceptions
acl WindowsUpdate dstdomain "/etc/squid3/WindowsUpdate"
acl Whitelist_Domains dstdomain "/etc/squid3/Whitelist_Domains"

# ACL to allow monitoring of entire proxy chain from 10.250.111.124 without
authentication 
acl MonitorProxy src 10.250.111.124/32

acl Get_Username proxy_auth REQUIRED

# Bypass squidguard for whitelisted domains
redirector_access deny Whitelist_Domains
redirector_access deny WindowsUpdate
# Bypass squidguard for local sites 
redirector_access deny BypassCache
redirector_access deny BypassCacheDomains

# Bypass connections to local network and TLS
always_direct allow BypassCache
cache deny BypassCache
always_direct allow BypassCacheDomains
cache deny BypassCacheDomains

http_access allow manager localhost
http_access allow localhost Purge
http_access deny manager
http_access deny Purge
http_access deny to_localhost
http_access deny !Local_Networks
http_access allow Whitelist_Domains
http_access allow WindowsUpdate
http_access allow MonitorProxy
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
# Allow connection to HTTPS sites from the local network
http_access allow CONNECT SSL_ports Local_Networks
http_access allow ftp
http_access allow !RestrictedUsers

http_access deny all

http_port 8080
visible_hostname Squid3
hierarchy_stoplist cgi-bin ?

# Log file locations
access_log daemon:/var/log/squid3/access.log squid
cache_store_log none
cache_log /var/log/squid3/cache.log

# Disk cache directory.
cache_dir aufs /squid_cache/Squid3Cache 25000 16 256
cache_mem 2000 MB
maximum_object_size_in_memory 1 MB

# Windows Update
#range_offset_limit 200 MB WindowsUpdate 
maximum_object_size 1 GB
#quick_abort_min -1

dns_nameservers 127.0.0.1

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024
icap_service service_req reqmod_precache bypass=0
icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/squidclamav
adaptation_access service_resp allow all

url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
url_rewrite_children 20 startup=0 idle=1 concurrency=0

#Do not show client IP address
via off
forwarded_for off

#Rules to anonymize http headers
request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access Cookie allow all
###request_header_access All deny all

--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-downloading-huge-amounts-of-un-requested-data-tp4670770p4670786.html
Sent from the Squid - Users mailing list archive at Nabble.com.