[squid-users] Sudden but sustained high bandwidth usage
Heiler Bemerguy
heiler.bemerguy at cinbesa.com.br
Mon Mar 7 18:38:49 UTC 2016
skyrocketing = using our maximum link download bandwidth.
This machine is only proxying. Not being a firewall, not a router, nor a
gateway. It has access to the internet through our gateway/firewall
(pfsense).
Lots of LAN clients are connected to the proxy, this is their only way
to the internet. 1 interface, debian linux. EXT4 FS. CPU/MEM usage is
always stable.
Clients use it explicitly or via wpad. Never transparently. Now I'm
using 3 workers, because 1 is not enough and we have spare cores.
It's a VM machine with netapp storage. lots of raid disks.
SQUID was running perfectly without cache_dirs.
I think squid is downloading and redownloading the same files over and
over again because: 1- these are segmented downloads and
range_offset_limit is set to NONE for these files. 2- it can't store the
downloaded files on the cache but I don't know why!
1457358960.737 8399 10.23.0.63 *TCP_SWAPFAIL_MISS*/206 1058138 GET
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2013/10/ie11-windows6.1-x64-en-us_ddec9ddc256ffa7d97831af148f6cc45130c6857.exe
- HIER_DIRECT/201.30.251.43 application/octet-stream
1457358987.869 22464 10.88.10.5 *TCP_SWAPFAIL_MISS*/206 1416417 GET
http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/ie11-windows6.1-kb3141092-x64_0f7a98b9dc9f5c7ac73f6f543bf004a15e4d7be8.psf
- HIER_DIRECT/201.30.251.26 application/octet-stream
Squid Object Cache: Version 3.5.15-20160301-r13999
Build Info:
Service Name: squid
Start Time: Fri, 04 Mar 2016 21:15:47 GMT
Current Time: Mon, 07 Mar 2016 18:14:37 GMT
Connection information for squid:
Number of clients accessing cache: 4543
Number of HTTP requests received: 7305505
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 1765.1
Average ICP messages per minute since start: 0.0
Select loop called: 365044476 times, 11.146 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 22.0%, 60min: 21.2%
Hits as % of bytes sent: 5min: 3.9%, 60min: 7.3%
Memory hits as % of hit requests: 5min: 2.5%, 60min: 2.9%
Disk hits as % of hit requests: 5min: 28.1%, 60min: 26.8%
Storage Swap size: 179951936 KB
Storage Swap capacity: 45.1% used, 54.9% free
Storage Mem size: 4194176 KB
Storage Mem capacity: 100.0% used, 0.0% free
Mean Object Size: 46.07 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.06514 0.07007
Cache Misses: 0.09730 0.10075
Cache Hits: 0.00055 0.00055
Near Hits: 0.06521 0.06757
Not-Modified Replies: 0.00055 0.00055
DNS Lookups: 0.00019 0.00019
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 248329.183 seconds
CPU Time: 21525.705 seconds
CPU Usage: 8.67%
CPU Usage, 5 minute avg: 52.35%
CPU Usage, 60 minute avg: 47.90%
Maximum Resident Size: 85809792 KB
Page faults with physical i/o: 104
Memory accounted for:
Total accounted: 164356 KB
memPoolAlloc calls: 1801863396
memPoolFree calls: 1811836496
File descriptor usage for squid:
Maximum number of file descriptors: 81920
Largest file desc currently in use: 6157
Number of file desc currently in use: 8216
Files queued for open: 0
Available number of file descriptors: 73704
Reserved number of file descriptors: 500
Store Disk files open: 11
Internal Data Structures:
25054 StoreEntries
391 StoreEntries with MemObjects
96086 Hot Object Cache Items
3905959 on-disk objects
Best Regards,
--
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751
Em 07/03/2016 14:33, Eliezer Croitoru escreveu:
> On 07/03/2016 16:29, Heiler Bemerguy wrote:
>> We're still getting all these SWAPFAIL and our link is
>> skyrocketing...... please help! I think it didn't happen on older
>> versions (.14 and below)
>
> Hey,
>
> What do you mean by skyrocketing?? like in the graph??
> Also it is not clear to me something about the machine, is this
> machine a FW\ROUTER\GW?
> If so is it for a lan?
> How many interface this machine has?
> Is it a pfsense? if so what version?
> What is the FS used for the cache directories?
> Did you also measured CPU when you see the spikes? if so what is it?
> How clients access the proxy service? transparently or using a browser
> setings or WPAD with dhcp settings?
> Also I have seen you are using 2 workers, is it because one worker
> doesn't seem to do the job?
> Did you tried to change the values of:
> cache_swap_low 98
> cache_swap_high 99
>
> from this high to lower numbers such as:
> cache_swap_low 90
> cache_swap_high 95
>
> or even lower?
> cache_swap_low 80
> cache_swap_high 85
>
> I am unsure about this since in the docs at:
> http://www.squid-cache.org/Doc/config/cache_swap_low/
> http://www.squid-cache.org/Doc/config/cache_swap_high/
>
> the ROCK storage is not mentioned.
>
> Also on what hardware are you running? what disks?
>
> All the above are important and in your case it is possible that there
> is something wrong in how the network is planned or the software doing
> something wrong.
>
> In scenarios like this I offer to verify two things:
> - test what happens when you disable disk cache.(from CPU, bandwidth,
> DISK aspect)
> - dump the cache manager info page to see basic statistics about the
> proxy traffic using:
> http:/cache_ip_or_visiblie_host_name:3128/squid-internal-mgr/info
>
> For scenarios like this I started working on a logging\monitoring
> service\script that will run in the background of the machine and will
> dump content of some statistics to enable couple squid developers eyes
> to see these and then have a better understanding the nature of the
> issue.
> For now the script\service is not ready and will not be able to help
> us so we need these dumps and information..
>
> Eliezer
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160307/61ace5ff/attachment-0001.html>
More information about the squid-users
mailing list