[squid-users] Squid for windows Very slow downloads of large files through squid with normal uploads
Yuri
yvoinov at gmail.com
Thu Mar 22 22:47:13 UTC 2018
And also:
your configuration is not transparent proxy.
a) Squid 3.5 for windows does not built as transparent proxy (i.e. with
NAT support).
b) You do not have keyword*intercept* in your configuration.
This is simple forwarding proxy.
23.03.2018 04:38, Yuri пишет:
>
>
>
> 22.03.2018 23:10, Keith Hartley пишет:
>>
>> I am using squid 3.5 for windows as a transparent proxy to provide
>> internet access to 7 servers in a secure environment that otherwise
>> does not have internet access. I have two squids running behind a
>> load balancer, each one is running server 2016 core with 2 Xeon
>> processors that is either haswell generation with 1:1 physical
>> processor to virtual processor mapping or a hyper-threading Broadwell
>> generation processor that is 1:1 logical processor to virtual
>> processor mapping, depending on how they are provisioned when they
>> get started.
>>
>>
>>
>> Doing a bandwidth test directly in the VM I am able to get internet
>> throughput of 800-1200 Mbps.
>>
>>
>>
>> Doing a file copy to and from the VM I am able to get 1200 Mbps lan
>> throughput.
>>
>>
>>
>> In proxied uploads I have observed speeds as high as 120 Mbps, which
>> is more than enough for what I need and the bottleneck is likely in
>> the backup software rather than squid. Uploads performance I am not
>> worried about where they are at now – even if I only got 20-30 Mbps
>> it would be adequate for what I need it for.
>>
>>
>>
>> Downloads however are very slow. Small files do not seem to be
>> impacted. Using the test a thinkbroadband.com/download, files up to
>> 20 Mb will download at a reasonable 20-30 Mbps, but when I get to 50,
>> it slows down to about 17 Mbps, and when I download AD Connect from
>> Microsoft, which is about 80 Mb, I can see it start at about 30 Mbps,
>> but eventually goes down to about 115 kbps and levels off. When I put
>> an IP on the server I am using for testing that proxies through
>> squid, I am able to download the file at several hundred mbps. When
>> I download the same file on the squid server – I can’t tell exactly
>> what throughput I was getting, but the 80 Mb file downloaded within 5
>> seconds.
>>
>>
>>
>> In both squid servers, other than when the servers were booting,
>> processor activity has not exceeded 9% in the last 7 days but usually
>> sits below 2%. Memory usage has not exceeded 2 Gb, leaving 2 Gb free.
>>
>>
>>
>> I am using OpenDNS for a DNS source, and have tried changing DNS to
>> level3 but it made no performance difference.
>>
>>
>>
>> I think that this may be squid trying to cache something, but had
>> tried to turn all caching off.
>>
>>
>>
>> My cache.log doesn’t really have anything interesting in it that I
>> can see. It’s the same ~30 or so log entries each time the service
>> starts, and that is about it. Here it is:
>>
>>
>>
>> 2018/03/22 09:47:27 kid1| Set Current Directory to /var/cache/squid
>>
>> 2018/03/22 09:47:27 kid1| Starting Squid Cache version 3.5.27 for
>> x86_64-unknown-cygwin...
>>
>> 2018/03/22 09:47:27 kid1| Service Name: squid
>>
>> 2018/03/22 09:47:27 kid1| Process ID 1164
>>
>> 2018/03/22 09:47:27 kid1| Process Roles: worker
>>
>> 2018/03/22 09:47:27 kid1| With 3200 file descriptors available
>>
>> 2018/03/22 09:47:27 kid1| Initializing IP Cache...
>>
>> 2018/03/22 09:47:27 kid1| parseEtcHosts: /etc/hosts: (2) No such file
>> or directory
>>
>> 2018/03/22 09:47:27 kid1| DNS Socket created at [::], FD 5
>>
>> 2018/03/22 09:47:27 kid1| DNS Socket created at 0.0.0.0, FD 6
>>
>> 2018/03/22 09:47:27 kid1| Adding nameserver 208.67.222.222 from
>> squid.conf
>>
>> 2018/03/22 09:47:27 kid1| Adding nameserver 208.67.220.220 from
>> squid.conf
>>
>> 2018/03/22 09:47:27 kid1| Logfile: opening log
>> daemon:/var/log/squid/access.log
>>
>> 2018/03/22 09:47:27 kid1| Logfile Daemon: opening log
>> /var/log/squid/access.log
>>
>> 2018/03/22 09:47:27 kid1| WARNING: no_suid: setuid(0): (22) Invalid
>> argument
>>
>> 2018/03/22 09:47:27 kid1| Store logging disabled
>>
>> 2018/03/22 09:47:27 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
>> objects
>>
>> 2018/03/22 09:47:27 kid1| Target number of buckets: 1008
>>
>> 2018/03/22 09:47:27 kid1| Using 8192 Store buckets
>>
>> 2018/03/22 09:47:27 kid1| Max Mem size: 262144 KB
>>
>> 2018/03/22 09:47:27 kid1| Max Swap size: 0 KB
>>
>> 2018/03/22 09:47:27 kid1| Using Least Load store dir selection
>>
>> 2018/03/22 09:47:27 kid1| Set Current Directory to /var/cache/squid
>>
>> 2018/03/22 09:47:27 kid1| Finished loading MIME types and icons.
>>
>> 2018/03/22 09:47:27 kid1| HTCP Disabled.
>>
>> 2018/03/22 09:47:27 kid1| Squid plugin modules loaded: 0
>>
>> 2018/03/22 09:47:27 kid1| Adaptation support is off.
>>
>> 2018/03/22 09:47:27 kid1| Accepting HTTP Socket connections at
>> local=[::]:3128 remote=[::] FD 10 flags=9
>>
>> 2018/03/22 09:47:28 kid1| storeLateRelease: released 0 objects
>>
>>
>>
>>
>>
>> And this is my squid.conf:
>>
>>
>>
>> #
>>
>> # Recommended minimum configuration:
>>
>> #
>>
>>
>>
>> # Example rule allowing access from your local networks.
>>
>> # Adapt to list your (internal) IP networks from where browsing
>>
>> # should be allowed
>>
>>
>>
>> #acl localnet src 10.0.0.0/8 # RFC1918 possible internal
>> network
>>
>> #acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
>>
>> #acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
>>
>> acl localnet src fc00::/7 # RFC 4193 local private network range
>>
>> acl localnet src fe80::/10 # RFC 4291 link-local (directly
>> plugged) machines
>>
>> acl WSUS src 192.168.225.4/32
>>
>> acl BACKUP src 192.168.225.11/32
>>
>> acl ADFS src 192.168.224.7/32
>>
>> acl ADFS src 192.168.228.8/32
>>
>> acl DEVWEB src 192.168.226.6/32
>>
>> acl UATWEB src 192.168.226.13/32
>>
>> acl PRDWEB src 192.168.226.8/32
>>
>> acl PRDWEB src 192.168.226.9/32
>>
>>
>>
>>
>>
>>
>>
>> acl SSL_ports port 443
>>
>> acl Safe_ports port 80 # http
>>
>> #acl Safe_ports port 21 # ftp
>>
>> acl Safe_ports port 443 # https
>>
>> #acl Safe_ports port 70 # gopher
>>
>> #acl Safe_ports port 210 # wais
>>
>> #acl Safe_ports port 1025-65535 # unregistered ports
>>
>> #acl Safe_ports port 280 # http-mgmt
>>
>> #acl Safe_ports port 488 # gss-http
>>
>> #acl Safe_ports port 591 # filemaker
>>
>> #acl Safe_ports port 777 # multiling http
>>
>> acl CONNECT method CONNECT
>>
>>
>>
>> #
>>
>> # Recommended minimum Access Permission configuration:
>>
>> #
>>
>>
>>
>> # Only allow cachemgr access from localhost
>>
>> #http_access allow localhost manager
>>
>> #http_access deny manager
>>
>>
>>
>> # Deny requests to certain unsafe ports
>>
>> http_access deny !Safe_ports
>>
>>
>>
>> # Deny CONNECT to other than secure SSL ports
>>
>> http_access deny CONNECT !SSL_ports
>>
>>
>>
>> # We strongly recommend the following be uncommented to protect innocent
>>
>> # web applications running on the proxy server who think the only
>>
>> # one who can access services on "localhost" is a local user
>>
>> #http_access deny to_localhost
>>
>>
>>
>> #
>>
>> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
>>
>> #
>>
>>
>>
>> # Example rule allowing access from your local networks.
>>
>> # Adapt localnet in the ACL section to list your (internal) IP networks
>>
>> # from where browsing should be allowed
>>
>> http_access allow localnet
>>
>> http_access allow localhost
>>
>> http_access allow WSUS
>>
>> http_access allow ADFS
>>
>> http_access allow BACKUP
>>
>> http_access allow DEVWEB
>>
>> http_access allow UATWEB
>>
>> http_access allow PRDWEB
>>
>>
>>
>> # And finally deny all other access to this proxy
>>
>> http_access deny all
>>
>>
>>
>> # Squid normally listens to port 3128
>>
>> http_port 3128
>>
>>
>>
>> # Uncomment the line below to enable disk caching - path format is
>> /cygdrive/<full path to cache folder>, i.e.
>>
>> #cache_dir aufs /cygdrive/d/squid/cache 3000 16 256
>>
>> cache deny all
>>
>>
>>
>>
>>
>> # Leave coredumps in the first cache dir
>>
>> coredump_dir /var/cache/squid
>>
>>
>>
>> # Add any of your own refresh_pattern entries above these.
>>
>> refresh_pattern ^ftp: 1440 20% 10080
>>
>> refresh_pattern ^gopher: 1440 0% 1440
>>
>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>
>> refresh_pattern . 0
>> 20% 4320
>>
>>
>>
>> dns_nameservers 208.67.222.222 208.67.220.220
>>
>>
>>
>> max_filedescriptors 3200
>>
>>
>>
>>
>>
>>
>>
>> Does anyone see anything I am missing here?
>>
> Yes. In your almost default configuration (it is complete squid.conf?)
> obvious thing is:
>
> a) You do not use on-disk cache.
> b) You use memory cache by default - i.e. 256 Mb.
> c) You cache nothing due to deny all cache. So, it makes useless
> cache_mem default.
> d) Your configuration technically useless. I see neither proxying
> parameters, nor caching. Your squid now only additional hop for files.
> No more.
>
> So, squid nothing to do here. It simple should retransmit GET (GET?)
> request to server, and, without any caching/storing, retransmit it to
> user.
>
> Still correct?
>
> This put us directly to raw network IO. Without any buffering (which
> can be - but don't - your squid).
>
> On your place, I can start playing around with cache_mem parameter; of
> course, only after removing cache deny all.
>
> And after some experiments, may be, will make decision about drop out
> useless Squid's box.
>
> Seriously, what role of squid's here? Just setup border firewall to
> your servers to access it to Internet. It will be enough.
>
>>
>>
>>
>>
>> My access.log doesn’t really have anything interesting in it either,
>> it just looks like it is working normally. I can attach that too if
>> anyone wants to look at it after I redact some of the hosts.
>>
>>
>>
>>
>>
>> *Keith Hartley*
>>
>> /Network Engineer II/
>>
>> /MCSE: Productivity, MCSA: Server 2008, 2012, Office 365 / |
>>
>> /Certified Meraki Network Associate, Security+/
>>
>> *Geocent, LLC*
>>
>> *o:*504-405-3578
>>
>> *a:*2219 Lakeshore drive Ste 300, New Orleans, LA 70122
>>
>> *w:*www.geocent.com
>> <http://www.geocent.com/>|*e:*khartley at geocent.com
>> <mailto:khartley at geocent.com>
>>
>>
>>
>>
>>
>>
>>
>>
>> Confidentiality Notice:
>> This email communication may contain confidential information, may be
>> legally privileged, and is intended only for the use of the intended
>> recipients(s) identified. Any unauthorized review, use, distribution,
>> downloading, or copying of this communication is strictly prohibited.
>> If you are not the intended recipient and have received this message
>> in error, immediately notify the sender by reply email, delete the
>> communication, and destroy all copies. Thank you.
>>
>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>
> --
> "C++ seems like a language suitable for firing other people's legs."
>
> *****************************
> * C++20 : Bug to the future *
> *****************************
--
"C++ seems like a language suitable for firing other people's legs."
*****************************
* C++20 : Bug to the future *
*****************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180323/c297146b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180323/c297146b/attachment-0001.sig>
More information about the squid-users
mailing list