[squid-users] squid 3.4.8 Eating all of the Bandwidth

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/10/2014 7:07 a.m., saleh madi wrote:
> Hello,
> 
> Why squid 3.4.8 Eating all of the Bandwidth:
> 
> iptraf -d eth0 in the squid box
> 
> Incoming rates:  435007.2 kbits/sec
>  50974.2 packets/sec
> 
> 
> Outgoing rates:  276229.9 kbits/sec
>  45568.4 packets/sec
> 
> 
> squid config:
> 
> range_offset_limit 0 quick_abort_min 0 quick_abort_max 0
> 
> You see the difference between the Incoming rates and Outgoing
> rates is: 200M and more.

Please avoid relying on the terms "incoming" and "outgoing". When
proxying they get a bit obscure. Like telling someone to turn left or
right you have to know which way they are facing first.


Since you think there is a problem I am going to assume "incoming" is
traffic from the Internet? given that it has a higher number than
outgoing.

Bandwidth differences like this can occur for several reasons...

* Squid talking HTTP/1.1 to clients and 1.0 to servers. HTTP/1.1
contains a number of bandwidth optimization features such as REFRESH
(INM, IMS) requests which do not exist in HTTP/1.0.
 The client may be using those 1.1 features to efficiently update its
content, but the 1.0 server delivering only inefficient full-content
(200) responses to Squid.

* Squid does not cache range replies. Given your range_offset_limit it
is quite possible Squid is translating between 200 and 206 responses.
 The clients sending many small byte range requests can cause Squid to
fetch new content (200 response) from the server, then translate it to
a smaller 206 response.
 Use range_offset_limit sparingly.

* Squid-3.4 has much improved caching of objects. If you are using
ignore-auth or ignore-private options on the refresh_pattern directive
you can actively *prevent* Squid from storing many objects which
otherwise would cache nicely. Which makes the above problems worse.
 The popular "override-expires" can also cause the same issues if the
web server emits an Expires header indicating more storage time
allowed than refresh_pattern "min" time field. I find it quite common
for admin to think they are forcing things to cache when in fact they
are *reducing* the storage time with override-expires.

* Squid-3.x older than 3.5 are missing the "collapsed forwarding"
feature enjoyed by Squid-2.6/2.7 users. As a result many clients
requesting the same non-cached object at the same time can cause it to
be downloaded multiple times. Combined with the Range request issue
above this can cause a lot of extra bandwidth usage on the
Internet/server connections.


* It is also possible you have configured an open proxy. If so there
can be a lot of traffic to/from the Internet without matching LAN traffic.

* Squid is only using a portion of the bandwidth measured by iptraf.
For certain there are TCP protocol overheads, and likely also UDP
protocols such as DNS packets going over the connections.

Amos

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUTtzKAAoJELJo5wb/XPRjA+cIAMPUXydfHNKyL6LEhPb5lS0Y
d7VWS4F0Z8VKKibxtrdc8y9fh7LpnGQyU1hxkuJRsXZSI1EzkyXCNfnaqAK6IkIL
q+JEbsenIWO9U53rwN704Wd/O3ASAT9HO5q6QmBcZ8PMVlZYCLEqCdwW9pCoij47
ZrQ6Zb6wPcdR4vJ+LtsSeFXJiZahWMoKuvqEbZld2tJASfF3H4WWqlxcxSzT1hjn
Rfb1XU6GpGikNbqg3ExlwXS9wV43/tbyeUj0d+8SWhXI8b+Vx3DiWHh7BILsMkuD
+NQNnZ6cLGPi2ccK+4MOazuHjVHyC5iK1LoWLBudKp5sncRmCuXtYAAg9FWTASU=
=B7ez
-----END PGP SIGNATURE-----