[squid-users] Sudden but sustained high bandwidth usage
Yuri Voinov
yvoinov at gmail.com
Mon Mar 7 20:16:26 UTC 2016
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
08.03.16 0:38, Heiler Bemerguy пишет:
> skyrocketing = using our maximum link download bandwidth.
> This machine is only proxying. Not being a firewall, not a router, nor
a gateway. It has access to the internet through our gateway/firewall
(pfsense).
> Lots of LAN clients are connected to the proxy, this is their only way
to the internet. 1 interface, debian linux. EXT4 FS. CPU/MEM usage is
always stable.
> Clients use it explicitly or via wpad. Never transparently. Now I'm
using 3 workers, because 1 is not enough and we have spare cores.
> *It's a VM machine with netapp storage. lots of raid disks. **
> **SQUID was running perfectly without cache_dirs. *
VM+netapp has own tweaks, quirks and features. And "squid running
perfectly runs without cache_dirs" directly pointed to problem(s). (This
is excluding not enough storage for store _all_ Windows updates approx.
from 2013 ;))
>
> I think squid is downloading and redownloading the same files over and
over again because: 1- these are segmented downloads and
range_offset_limit is set to NONE for these files. 2- it can't store the
downloaded files on the cache but I don't know why!
>
> 1457358960.737 8399 10.23.0.63 *TCP_SWAPFAIL_MISS*/206 1058138 GET
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2013/10/ie11-windows6.1-x64-en-us_ddec9ddc256ffa7d97831af148f6cc45130c6857.exe
- HIER_DIRECT/201.30.251.43 application/octet-stream
> 1457358987.869 22464 10.88.10.5 *TCP_SWAPFAIL_MISS*/206 1416417 GET
http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/ie11-windows6.1-kb3141092-x64_0f7a98b9dc9f5c7ac73f6f543bf004a15e4d7be8.psf
- HIER_DIRECT/201.30.251.26 application/octet-stream
>
> Squid Object Cache: Version 3.5.15-20160301-r13999
> Build Info:
> Service Name: squid
> Start Time: Fri, 04 Mar 2016 21:15:47 GMT
> Current Time: Mon, 07 Mar 2016 18:14:37 GMT
> Connection information for squid:
> Number of clients accessing cache: 4543
> Number of HTTP requests received: 7305505
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 1765.1
> Average ICP messages per minute since start: 0.0
> Select loop called: 365044476 times, 11.146 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 22.0%, 60min: 21.2%
> Hits as % of bytes sent: 5min: 3.9%, 60min: 7.3%
> Memory hits as % of hit requests: 5min: 2.5%, 60min: 2.9%
> Disk hits as % of hit requests: 5min: 28.1%, 60min: 26.8%
> Storage Swap size: 179951936 KB
> Storage Swap capacity: 45.1% used, 54.9% free
> Storage Mem size: 4194176 KB
> Storage Mem capacity: 100.0% used, 0.0% free
> Mean Object Size: 46.07 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.06514 0.07007
> Cache Misses: 0.09730 0.10075
> Cache Hits: 0.00055 0.00055
> Near Hits: 0.06521 0.06757
> Not-Modified Replies: 0.00055 0.00055
> DNS Lookups: 0.00019 0.00019
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 248329.183 seconds
> CPU Time: 21525.705 seconds
> CPU Usage: 8.67%
> CPU Usage, 5 minute avg: 52.35%
> CPU Usage, 60 minute avg: 47.90%
> Maximum Resident Size: 85809792 KB
> Page faults with physical i/o: 104
> Memory accounted for:
> Total accounted: 164356 KB
> memPoolAlloc calls: 1801863396
> memPoolFree calls: 1811836496
> File descriptor usage for squid:
> Maximum number of file descriptors: 81920
> Largest file desc currently in use: 6157
> Number of file desc currently in use: 8216
> Files queued for open: 0
> Available number of file descriptors: 73704
> Reserved number of file descriptors: 500
> Store Disk files open: 11
> Internal Data Structures:
> 25054 StoreEntries
> 391 StoreEntries with MemObjects
> 96086 Hot Object Cache Items
> 3905959 on-disk objects
>
> Best Regards,
>
> --
> Heiler Bemerguy - (91) 98151-4894
> Assessor Técnico - CINBESA (91) 3184-1751
>
> Em 07/03/2016 14:33, Eliezer Croitoru escreveu:
>> On 07/03/2016 16:29, Heiler Bemerguy wrote:
>>> We're still getting all these SWAPFAIL and our link is
>>> skyrocketing...... please help! I think it didn't happen on older
>>> versions (.14 and below)
>>
>> Hey,
>>
>> What do you mean by skyrocketing?? like in the graph??
>> Also it is not clear to me something about the machine, is this
machine a FW\ROUTER\GW?
>> If so is it for a lan?
>> How many interface this machine has?
>> Is it a pfsense? if so what version?
>> What is the FS used for the cache directories?
>> Did you also measured CPU when you see the spikes? if so what is it?
>> How clients access the proxy service? transparently or using a
browser setings or WPAD with dhcp settings?
>> Also I have seen you are using 2 workers, is it because one worker
doesn't seem to do the job?
>> Did you tried to change the values of:
>> cache_swap_low 98
>> cache_swap_high 99
>>
>> from this high to lower numbers such as:
>> cache_swap_low 90
>> cache_swap_high 95
>>
>> or even lower?
>> cache_swap_low 80
>> cache_swap_high 85
>>
>> I am unsure about this since in the docs at:
>> http://www.squid-cache.org/Doc/config/cache_swap_low/
>> http://www.squid-cache.org/Doc/config/cache_swap_high/
>>
>> the ROCK storage is not mentioned.
>>
>> Also on what hardware are you running? what disks?
>>
>> All the above are important and in your case it is possible that
there is something wrong in how the network is planned or the software
doing something wrong.
>>
>> In scenarios like this I offer to verify two things:
>> - test what happens when you disable disk cache.(from CPU, bandwidth,
DISK aspect)
>> - dump the cache manager info page to see basic statistics about the
proxy traffic using:
http:/cache_ip_or_visiblie_host_name:3128/squid-internal-mgr/info
>>
>> For scenarios like this I started working on a logging\monitoring
service\script that will run in the background of the machine and will
dump content of some statistics to enable couple squid developers eyes
to see these and then have a better understanding the nature of the issue.
>> For now the script\service is not ready and will not be able to help
us so we need these dumps and information..
>>
>> Eliezer
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJW3eGaAAoJENNXIZxhPexGBcMIAIRTiHjnUXqixXiZnlYgTVZG
/fMYoCik8wo5iMCcZj35jVmfbGFKUYFgfc9S1NgyrrzO+8cfTxQUhF3QDESjqmu6
+uFLh1o91bsMtFpXutruRF3YwOz/tvep8r3G0/gmnuzvZo47ZrFztH6eti0tpkDa
l9mK3dnrrkpwNb69yBx81MKvt9hshK4iNIRdKC9LtlY7xdthnQqdLySinDtPJlV/
bWp3IcTycxqTR5KFVgsTxQnzJdSQaz0He3Bxm0yms2xxwMTnr8Hggk8jB3vY+aFh
NKy8WB01N/+oIb2eBtbhW35o9Hu3MYVfCO8oxKSRGGky++DitY0Cr0X02SPSo5g=
=yyyX
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160308/7b9fe12f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x613DEC46.asc
Type: application/pgp-keys
Size: 2437 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160308/7b9fe12f/attachment-0001.key>
More information about the squid-users
mailing list