[squid-users] Squid slow down after awhile

[Amos Jeffries]

On 15/10/17 05:20, masoud mazarei wrote:
> i sent you an pcap file which shows the problem.
> my client ip is 172.22.127.1 and target host is  94.182.227.21.
> squid machine mac is e4:11:5b:ea:30:c2.
> filter pcap file in wireshark by filter "ip.host==94.182.227.21" you 
> will see that the first SYN packet arrived in No.304 and relative time 
> 6.637173 but first packet which goes out from cache machine as client by 
> squid happend in No.371 and relative time 45.013691

That kind of indicates the problem is either in how long the client 
takes to deliver the HTTP request to Squid, or DNS lookups to find the 
destination(s).

> what is happend in (6.637173 - 45.013691) duration.?

For a transparent proxy these things have happened between SYN on 
client<->Squid and SYN on Squid<->server:


* NAT/TPROXY record lookups for client connection state

* wait for the client to send its HTTP request.
  - with happy eyeballs there may be a large wait between the SYN and 
first data sent by client for ~50% of connections.

* parsing of that HTTP request message.

* DNS lookup(s) for Host header verification

* http_access checks

* Adaptation hooks (ICAP / eCAP), if any

* URL re-writer lookups, if any

* HTTP 'cache' directive ACL checks

* HTTP cache lookup

* DNS lookups to find destination, if any
  - this should be very fast since the Host verify results should be 
cached. But if any of the above took longer than DNS TTL new lookups may 
be required - naturally increasing the delay further.

* Destination selection

* TCP server connection(s) setup
  - if you are only looking at IPv4 packets you may be missing multiple 
SYN packets for IPv6 servers before the first IPv4 SYN packet appears.


The points above with sub-notes are the ones most likely to be delayed 
for seconds. You may be seeing one particular source of the problem, or 
multiple adding together. 45 sec seems an unusual number. Most of the 
timeouts in Squid and networking are multiples of 30 sec.



> which debug level will help me to know what is happend in background?
> i enabled
> "debug_options 5,3 6,3 46,3 11,3 19,3 55,3 58,3"
> BUT there is no valuable data to solve this problem.
> 

You may need an ALL,6 trace then to see if there are any clues in odd 
places. As verbose as it is the debugging in Squid is far from complete 
so for some of these delay issues there no specific lines to look for 
mention and we have to go by relative timing of things.

The durations between actions on the list of points above should narrow 
down a bit better what to look at.

Amos