[squid-users] squid 3.1 with https traffic and delay pools is flooding network with hundreds of thousands 65-70 bytes packets (and killing the routers, anyway)

[Horváth Szabolcs]

Hello!

We're having serious problems with a squid proxy server. 

The good news is the problem can be reproduced at any time in our production squid system.

Environment:
- CentOS release 6.5 (Final) with Linux kernel 2.6.32-431.29.2.el6.x86_64
- squid-3.1.10-22.el6_5.x86_64 (a bit old, CentOS ships this version)

Problem description:
- if we have a few mbytes/sec https traffic AND
- delay_classes are in place AND
- delay pools are full (I mean the available bandwidth for the customer are used)

-> then squid is trickling https traffic down to the clients in 65-70 byte packets.

Our WAN routers are not designed to handle thousands of 65-70 byte packets per seconds and therefore we have some network stability issues.

I tracked down the following:
- if delay_pools are commented out (clients can go with full speed as they like) -> the problem eliminates, https traffic flows with ~1500 byte packets
- if we use only http traffic, there is no problem: http traffic flows with ~1500 byte packets even if the delay pools are full

Our test URL is www.opengroup.org/infosrv/DCE/dce122.tar.gz, which is available both on http and https protocol.

Resources can be found at http://support.iqsys.hu/logs/

1. squid.conf -> squid configuration file
2. http-delaypool.pcap: 
	- wget -c http://www.opengroup.org/infosrv/DCE/dce122.tar.gz, 
	- delay pools are active
	- http flows with 1500 byte packets
3. http-nodelaypool.pcap: 
	- wget -c http://www.opengroup.org/infosrv/DCE/dce122.tar.gz, 
	- delay pools are INACTIVE
	- http flows with 1500 byte packets
4. https-delaypool.pcap:
	- wget -c https://www.opengroup.org/infosrv/DCE/dce122.tar.gz, 
	- delay pools are active
	- http flows with 69 byte packets -> this is extremely bad
5. https-nodelaypool.pcap:
	- wget -c https://www.opengroup.org/infosrv/DCE/dce122.tar.gz, 
	- delay pools are INACTIVE
	- http flows with 1500 byte packets

My question is: is it a known bug?

If yes, which version(s) contain the fix (I read through the changelog several times, but I can't found the exact bug)
	- if 3.1 branch fixes this, upgrade is easy
	- upgrading major versions (3.4, 3.5) is not so trivial due to the complex environment (telling you the truth, installing a new squid server and migrating to it would be much easier than in-place upgrade)	

If not, how can I track down the issue?
	- as far as I understand squid configuration, it's not too complex
	- although ICAP is enabled (squidclamav is used), it's not the root of the problem -> when ICAP is commented out, the problem remains

Any ideas are appreciated. 

Thanks for reading this.

Best regards,
  Szabolcs Horvath