[squid-users] Sudden but sustained high bandwidth usage

Tue Mar 1 21:57:37 UTC 2016

Hey guys.

For the third time, we got a sudden high bandwidth usage, almost 
saturating our link, and it won't stop until squid is restarted.
I'm totally SURE this inbound traffic comes from squid. It's like it's 
download stuff itself....

Look that after squid was restarted near 10:45, the network usage drops 
immediately and won't increase as high as before anymore..

This pattern started to happen when I changed from ROCK+AUFS to 
ROCK+ROCK, squid 3.5.14 x64.

Here's the most important conf settings.. I appreciate all comments 
about it.

/acl windowsupdate dstdomain .ws.microsoft.com 
.windowsupdate.microsoft.com .update.microsoft.com .windowsupdate.com//
//http_access allow windowsupdate//
//range_offset_limit none windowsupdate//
//cache_mem 4 GB//
//maximum_object_size_in_memory 5 MB//
//memory_replacement_policy heap GDSF//
//cache_replacement_policy heap LFUDA//
//maximum_object_size 10 GB//
//cpu_affinity_map process_numbers=1,2,3,4,5,6 cores=1,2,3,4,5,6//
/*/workers 2/**/
/**/cache_dir rock /cache2/rock1 90000 min-size=0 max-size=32768/**/
/**/cache_dir rock /cache/rock1 300000 min-size=32768 
max-size=10737418240/*/
//store_dir_select_algorithm round-robin//
//read_ahead_gap 4096 KB//
//client_request_buffer_max_size 2048 KB//
//dns_v4_first on//
//ipcache_size 80000//
//fqdncache_size 40000//
//memory_pools on//
//memory_pools_limit 150 MB//
//reload_into_ims on//
//connect_retries 3//
//cache_swap_low 98//
//cache_swap_high 99//
//store_avg_object_size 92 KB//
//client_idle_pconn_timeout 30 seconds//
//client_persistent_connections off//
//server_persistent_connections off/

error.log right in this moment:
08:55:29 kid1| local=10.1.10.9:3080 remote=10.107.0.71:54515 FD 3665 
flags=1: read/write failure: (32) Broken pipe
09:00:02 kid2| snmpHandleUdp: FD 55 recvfrom: (11) Resource temporarily 
unavailable
09:00:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:02:14 kid2| WARNING: Closing client connection due to lifetime timeout
09:02:14 kid2| 
http://prod.video.msn.com/tenant/amp/entityid/BBq7uZY?blobrefkey=103&$blob=1
09:03:34 kid1| WARNING: HTTP: Invalid Response: Bad header encountered 
from http://sable.madmimi.com/view?id=24371.4971993.01561ff3
e8e7c09ac362ded25f80a76b AKA 
sable.madmimi.com/view?id=24371.4971993.01561ff3e8e7c09ac362ded25f80a76b
09:03:38 kid1| WARNING: Closing client connection due to lifetime timeout
09:03:38 kid1| 
http://download.windowsupdate.com/d/msdownload/update/software/defu/2016/02/am_delta_patch_1.213.7305.0_59c57a
caccbdfa7fa9dd5574f0a7ded60de11963.exe
09:04:15 kid1| WARNING: HTTP: Invalid Response: Bad header encountered 
from http://sable.madmimi.com/view?id=24371.4931972.b5133065
c861d91790f59bf39ef1abf3 AKA 
sable.madmimi.com/view?id=24371.4931972.b5133065c861d91790f59bf39ef1abf3
09:04:28 kid2| WARNING: Closing client connection due to lifetime timeout
09:04:28 kid2| 
http://www.ingressocerto.com/facet-search.json?f=/p-data-Offset:2
09:04:42 kid2| Could not parse headers from on disk object
09:05:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:05:02 kid2| snmpHandleUdp: FD 55 recvfrom: (11) Resource temporarily 
unavailable
09:05:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:05:18 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:06:32 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:06:32 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:06:36 kid2| clientIfRangeMatch: Weak ETags are not allowed in 
If-Range: "141811-1456747291000" ? "141811-1456747291000"
09:08:14 kid2| urlParse: URL too large (8360 bytes)
09:09:40 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:10:03 kid1| clientIfRangeMatch: Weak ETags are not allowed in 
If-Range: "94779-1455643401000" ? "94779-1455643401000"
09:14:01 kid1| local=10.1.10.9:3080 remote=10.101.2.60:1350 FD 2713 
flags=1: read/write failure: (32) Broken pipe
09:14:14 kid2| Could not parse headers from on disk object
09:14:18 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:15:25 kid2| WARNING: Closing client connection due to lifetime timeout
09:15:25 kid2| 
http://prod.video.msn.com/tenant/amp/entityid/BBq8oCY?blobrefkey=103&$blob=1
09:18:01 kid2| Could not parse headers from on disk object
09:19:42 kid1| clientIfRangeMatch: Weak ETags are not allowed in 
If-Range: "10819-1456747291000" ? "10819-1456747291000"
09:20:03 kid2| snmpHandleUdp: FD 55 recvfrom: (11) Resource temporarily 
unavailable
09:20:27 kid2| WARNING: HTTP: Invalid Response: Bad header encountered 
from http://sable.madmimi.com/view?id=24371.4894720.98d08b18
9bb8855e833cecd4ea9e7e7e AKA 
sable.madmimi.com/view?id=24371.4894720.98d08b189bb8855e833cecd4ea9e7e7e
09:21:06 kid2| local=10.1.10.9:3080 remote=10.42.0.43:50042 FD 1878 
flags=1: read/write failure: (110) Connection timed out
09:21:23 kid2| WARNING: HTTP: Invalid Response: Bad header encountered 
from http://sable.madmimi.com/view?id=24371.4870485.934d4afb
07878189bf41826766b1db26 AKA 
sable.madmimi.com/view?id=24371.4870485.934d4afb07878189bf41826766b1db26
09:23:04 kid1| local=10.1.10.9:3080 remote=10.102.2.19:49536 FD 1770 
flags=1: read/write failure: (110) Connection timed out
09:23:33 kid1| urlParse: URL too large (8268 bytes)
09:23:56 kid2| local=10.1.10.9:4816 remote=216.58.222.35:443 FD 1046 
flags=1: read/write failure: (32) Broken pipe
09:25:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:25:32 kid1| local=10.1.10.9:3080 remote=10.12.0.138:49951 FD 3338 
flags=1: read/write failure: (32) Broken pipe
09:25:49 kid1| urlParse: URL too large (8231 bytes)
09:27:24 kid1| urlParse: URL too large (8231 bytes)
09:27:46 kid2| urlParse: URL too large (10742 bytes)
09:28:16 kid1| local=10.1.10.9:36318 remote=216.58.222.35:443 FD 2169 
flags=1: read/write failure: (32) Broken pipe
09:28:58 kid1| Could not parse headers from on disk object
09:29:11 kid1| local=10.1.10.9:44504 remote=216.58.222.46:443 FD 4783 
flags=1: read/write failure: (32) Broken pipe
09:29:20 kid1| Could not parse headers from on disk object
09:30:03 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:31:13 kid1| local=10.1.10.9:3080 remote=10.12.0.147:49815 FD 4230 
flags=1: read/write failure: (32) Broken pipe
09:31:25 kid2| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:31:26 kid1| SECURITY ALERT: Missing hostname in URL 'http://'. see 
access.log for details.
09:31:46 kid1| Starting new redirector helpers...
09:31:46 kid1| helperOpenServers: Starting 20/70 'ufdbgclient' processes
09:31:50 kid2| local=10.1.10.9:63724 remote=74.207.230.168:443 FD 2461 
flags=1: read/write failure: (32) Broken pipe
09:33:51 kid2| local=10.1.10.9:3080 remote=10.107.0.71:56817 FD 1766 
flags=1: read/write failure: (32) Broken pipe
09:35:15 kid2| Could not parse headers from on disk object
09:40:03 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:42:03 kid2| clientIfRangeMatch: Weak ETags are not allowed in 
If-Range: "968-1455828160000" ? "968-1455828160000"
09:43:33 kid1| local=10.1.10.9:59631 remote=216.58.222.34:443 FD 2720 
flags=1: read/write failure: (32) Broken pipe
09:45:32 kid2| Could not parse headers from on disk object
09:46:30 kid2| local=10.1.10.9:3080 remote=10.23.0.44:49826 FD 2704 
flags=1: read/write failure: (110) Connection timed out
09:46:30 kid1| local=10.1.10.9:3080 remote=10.101.4.25:58434 FD 4631 
flags=1: read/write failure: (110) Connection timed out
09:46:55 kid1| local=10.1.10.9:3080 remote=10.11.0.186:63418 FD 159 
flags=1: read/write failure: (110) Connection timed out
09:47:33 kid1| clientIfRangeMatch: Weak ETags are not allowed in 
If-Range: "a80649-16652-52ce81a00d4dd" ? "a80649-16652-52ce81a00d4
dd"
09:50:03 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:50:28 kid1| Could not parse headers from on disk object
09:50:28 kid1| varyEvaluateMatch: Oops. Not a Vary object on second 
attempt, 'http://pix04.revsci.net/D08734/a1/0/3/0.js?DM_LOC=%3D
http%3A%2F%2Fna.com%3FdlxInitiated%3Dtrue%26nada%3D%26naid%3D2015121611542932923036123812%26namp%3D' 
'accept-encoding="gzip,%20deflate,%20sdch
"'
09:50:28 kid1| clientProcessHit: Vary object loop!
09:50:46 kid1| helperHandleRead: unexpected reply on channel 0 from 
redirector #Hlpr301 'OK'
09:50:46 kid1| helperHandleRead: unexpected reply on channel 0 from 
redirector #Hlpr301 'OK'
09:52:22 kid1| local=10.1.10.9:3080 remote=10.88.11.1:49932 FD 75 
flags=1: read/write failure: (110) Connection timed out
09:53:14 kid2| local=10.1.10.9:3080 remote=10.1.4.168:9637 FD 3038 
flags=1: read/write failure: (110) Connection timed out
09:54:07 kid1| Could not parse headers from on disk object
09:54:10 kid1| Could not parse headers from on disk object
09:54:12 kid2| urlParse: URL too large (8300 bytes)
09:54:17 kid2| Could not parse headers from on disk object
09:55:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
09:59:25 kid1| local=10.1.10.9:3080 remote=10.61.0.128:3955 FD 1589 
flags=1: read/write failure: (32) Broken pipe
10:01:37 kid2| local=10.1.10.9:3080 remote=10.11.0.125:60064 FD 1663 
flags=1: read/write failure: (110) Connection timed out
10:03:06 kid2| WARNING: Closing client connection due to lifetime timeout
10:03:06 kid2| 
http://r1---sn-p5qlsu7s.gvt1.com/edgedl/release2/u0bdrsci12g6umtr3oo309iyvgxltyrngktaw787s86k670suu6f9v28nnnnj
wjgjpd8vbs8iw7mpw7vh9r3ad5al7r607gwgwz/48.0.2564.116_48.0.2564.109_chrome_updater.exe?expire=1456764484&ip=177.74.57.130&ipbits=0&pl=24&shardb
ypass=yes&sparams=expire,ip,ipbits,mm,mn,ms,mv,nh,pl,shardbypass&signature=7BD8296B68294B5A2BF5A09E26F963E5E721CA22.7E237A019B036E4D5A5C9CDA30
CCF0CBA2932C45&key=cms1&req_id=81d1131ec532cde1&redirect_counter=4&cms_redirect=yes&mm=26&mn=sn-p5qlsu7s&ms=tsu&mt=1456750037&mv=m
10:03:19 kid2| Could not parse headers from on disk object
10:03:22 kid2| Could not parse headers from on disk object
10:03:30 kid2| Could not parse headers from on disk object
10:03:38 kid1| Could not parse headers from on disk object
10:03:46 kid2| Could not parse headers from on disk object
10:05:03 kid2| snmpHandleUdp: FD 55 recvfrom: (11) Resource temporarily 
unavailable
10:05:28 kid2| local=10.1.10.9:3080 remote=10.122.0.114:50243 FD 5349 
flags=1: read/write failure: (32) Broken pipe
10:10:02 kid1| snmpHandleUdp: FD 29 recvfrom: (11) Resource temporarily 
unavailable
10:10:13 kid1| urlParse: URL too large (8335 bytes)
10:11:01 kid1| local=10.1.10.9:3080 remote=10.1.4.17:50039 FD 1621 
flags=1: read/write failure: (110) Connection timed out
10:11:07 kid1| local=10.1.10.9:3080 remote=10.1.3.247:56870 FD 322 
flags=1: read/write failure: (32) Broken pipe

Best Regards,

-- 
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160301/78598b8b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: egfhiihh.png
Type: image/png
Size: 27674 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160301/78598b8b/attachment-0001.png>