[squid-users] Squid for windows Very slow downloads of large files through squid with normal uploads

Thu Mar 22 17:10:15 UTC 2018

I am using squid 3.5 for windows as a transparent proxy to provide internet access to 7 servers in a secure environment that otherwise does not have internet access. I have two squids running behind a load balancer, each one is running server 2016 core with 2 Xeon processors that is either haswell generation with 1:1 physical processor to virtual processor mapping or a hyper-threading Broadwell generation processor that is 1:1 logical processor to virtual processor mapping, depending on how they are provisioned when they get started.

Doing a bandwidth test directly in the VM I am able to get internet throughput of 800-1200 Mbps.

Doing a file copy to and from the VM I am able to get 1200 Mbps lan throughput.

In proxied uploads I have observed speeds as high as 120 Mbps, which is more than enough for what I need and the bottleneck is likely in the backup software rather than squid. Uploads performance I am not worried about where they are at now - even if I only got 20-30 Mbps it would be adequate for what I need it for.

Downloads however are very slow. Small files do not seem to be impacted. Using the test a thinkbroadband.com/download, files up to 20 Mb will download at a reasonable 20-30 Mbps, but when I get to 50, it slows down to about 17 Mbps, and when I download AD Connect from Microsoft, which is about 80 Mb, I can see it start at about 30 Mbps, but eventually goes down to about 115 kbps and levels off. When I put an IP on the server I am using for testing that proxies through squid, I am able to download the file at several hundred mbps.  When I download the same file on the squid server - I can't tell exactly what throughput I was getting, but the 80 Mb file downloaded within 5 seconds.

In both squid servers, other than when the servers were booting, processor activity has not exceeded 9% in the last 7 days but usually sits below 2%. Memory usage has not exceeded 2 Gb, leaving 2 Gb free.

I am using OpenDNS for a DNS source, and have tried changing DNS to level3 but it made no performance difference.

I think that this may be squid trying to cache something, but had tried to turn all caching off.

My cache.log doesn't really have anything interesting in it that I can see. It's the same ~30 or so log entries each time the service starts, and that is about it. Here it is:

2018/03/22 09:47:27 kid1| Set Current Directory to /var/cache/squid
2018/03/22 09:47:27 kid1| Starting Squid Cache version 3.5.27 for x86_64-unknown-cygwin...
2018/03/22 09:47:27 kid1| Service Name: squid
2018/03/22 09:47:27 kid1| Process ID 1164
2018/03/22 09:47:27 kid1| Process Roles: worker
2018/03/22 09:47:27 kid1| With 3200 file descriptors available
2018/03/22 09:47:27 kid1| Initializing IP Cache...
2018/03/22 09:47:27 kid1| parseEtcHosts: /etc/hosts: (2) No such file or directory
2018/03/22 09:47:27 kid1| DNS Socket created at [::], FD 5
2018/03/22 09:47:27 kid1| DNS Socket created at 0.0.0.0, FD 6
2018/03/22 09:47:27 kid1| Adding nameserver 208.67.222.222 from squid.conf
2018/03/22 09:47:27 kid1| Adding nameserver 208.67.220.220 from squid.conf
2018/03/22 09:47:27 kid1| Logfile: opening log daemon:/var/log/squid/access.log
2018/03/22 09:47:27 kid1| Logfile Daemon: opening log /var/log/squid/access.log
2018/03/22 09:47:27 kid1| WARNING: no_suid: setuid(0): (22) Invalid argument
2018/03/22 09:47:27 kid1| Store logging disabled
2018/03/22 09:47:27 kid1| Swap maxSize 0 + 262144 KB, estimated 20164 objects
2018/03/22 09:47:27 kid1| Target number of buckets: 1008
2018/03/22 09:47:27 kid1| Using 8192 Store buckets
2018/03/22 09:47:27 kid1| Max Mem  size: 262144 KB
2018/03/22 09:47:27 kid1| Max Swap size: 0 KB
2018/03/22 09:47:27 kid1| Using Least Load store dir selection
2018/03/22 09:47:27 kid1| Set Current Directory to /var/cache/squid
2018/03/22 09:47:27 kid1| Finished loading MIME types and icons.
2018/03/22 09:47:27 kid1| HTCP Disabled.
2018/03/22 09:47:27 kid1| Squid plugin modules loaded: 0
2018/03/22 09:47:27 kid1| Adaptation support is off.
2018/03/22 09:47:27 kid1| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 10 flags=9
2018/03/22 09:47:28 kid1| storeLateRelease: released 0 objects

And this is my squid.conf:

#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed

#acl localnet src 10.0.0.0/8           # RFC1918 possible internal network
#acl localnet src 172.16.0.0/12    # RFC1918 possible internal network
#acl localnet src 192.168.0.0/16  # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
acl WSUS src 192.168.225.4/32
acl BACKUP src 192.168.225.11/32
acl ADFS src 192.168.224.7/32
acl ADFS src 192.168.228.8/32
acl DEVWEB src 192.168.226.6/32
acl UATWEB src 192.168.226.13/32
acl PRDWEB src 192.168.226.8/32
acl PRDWEB src 192.168.226.9/32

acl SSL_ports port 443
acl Safe_ports port 80                    # http
#acl Safe_ports port 21                  # ftp
acl Safe_ports port 443                  # https
#acl Safe_ports port 70                  # gopher
#acl Safe_ports port 210                                # wais
#acl Safe_ports port 1025-65535                # unregistered ports
#acl Safe_ports port 280                                # http-mgmt
#acl Safe_ports port 488                                # gss-http
#acl Safe_ports port 591                                # filemaker
#acl Safe_ports port 777                                # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#

# Only allow cachemgr access from localhost
#http_access allow localhost manager
#http_access deny manager

# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
http_access allow WSUS
http_access allow ADFS
http_access allow BACKUP
http_access allow DEVWEB
http_access allow UATWEB
http_access allow PRDWEB

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment the line below to enable disk caching - path format is /cygdrive/<full path to cache folder>, i.e.
#cache_dir aufs /cygdrive/d/squid/cache 3000 16 256
cache deny all

# Leave coredumps in the first cache dir
coredump_dir /var/cache/squid

# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:                     1440       20%        10080
refresh_pattern ^gopher:            1440       0%          1440
refresh_pattern -i (/cgi-bin/|\?) 0             0%          0
refresh_pattern .                             0              20%        4320

dns_nameservers 208.67.222.222 208.67.220.220

max_filedescriptors 3200

Does anyone see anything I am missing here?

My access.log doesn't really have anything interesting in it either, it just looks like it is working normally. I can attach that too if anyone wants to look at it after I redact some of the hosts.

Keith Hartley
Network Engineer II
MCSE: Productivity, MCSA: Server 2008, 2012, Office 365 |
Certified Meraki Network Associate, Security+
Geocent, LLC
o: 504-405-3578
a: 2219 Lakeshore drive Ste 300, New Orleans, LA 70122
w: www.geocent.com<http://www.geocent.com/>| e: khartley at geocent.com<mailto:khartley at geocent.com>

Confidentiality Notice:
This email communication may contain confidential information, may be legally privileged, and is intended only for the use of the intended recipients(s) identified. Any unauthorized review, use, distribution, downloading, or copying of this communication is strictly prohibited. If you are not the intended recipient and have received this message in error, immediately notify the sender by reply email, delete the communication, and destroy all copies. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180322/8d460298/attachment-0001.html>