[squid-users] Sudden but sustained high bandwidth usage
L.P.H. van Belle
belle at bazuin.nl
Wed Mar 9 07:59:18 UTC 2016
With the settings i already told you. Today is ms update day and hee.. its caching my windows updates .. so go try them out.
1457510088.427 1 192.168.249.141 TCP_HIT/200 575 HEAD http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/windows6.1-kb3139852-x64_90af4c434e6fc8b0ed83cbd545b7488b0b84f2d7.psf - HIER_NONE/- application/octet-stream
1457510090.608 1 192.168.249.190 TCP_HIT/200 575 HEAD http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/windows6.1-kb3139852-x64_90af4c434e6fc8b0ed83cbd545b7488b0b84f2d7.psf - HIER_NONE/- application/octet-stream
1457510090.609 1 192.168.249.190 TCP_HIT/200 576 HEAD http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/windows6.1-kb3139940-x64_da3dd52959aed99d211cd36b503e387213bf07c4.psf - HIER_NONE/- application/octet-stream
1457510090.611 1 192.168.249.190 TCP_HIT/200 576 HEAD http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/windows6.1-kb3139940-x64_da3dd52959aed99d211cd36b503e387213bf07c4.psf - HIER_NONE/- application/octet-stream
1457510091.060 29 192.168.249.190 TCP_MISS/206 40207 GET http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/windows6.1-kb3139852-x64_90af4c434e6fc8b0ed83cbd545b7488b0b84f2d7.psf - HIER_DIRECT/8.253.82.142 multipart/byteranges
1457510091.899 1 192.168.249.190 TCP_HIT/200 591 HEAD http://au.download.windowsupdate.com/c/msdownload/update/software/secu/2016/02/windows6.1-kb3139398-x64_22584717f067444ad85f4e7a0333bc05273d4d73.psf - HIER_NONE/- application/octet-stream
1457510091.901 0 192.168.249.190 TCP_HIT/200 591 HEAD http://au.download.windowsupdate.com/c/msdownload/update/software/secu/2016/02/windows6.1-kb3139398-x64_22584717f067444ad85f4e7a0333bc05273d4d73.psf - HIER_NONE/- application/octet-stream
1457510093.151 1 192.168.249.190 TCP_HIT/200 577 HEAD http://au.download.windowsupdate.com/d/msdownload/update/software/updt/2016/02/windows6.1-kb3138612-x64_72ea89719674b61f69a5bbc93d9377e2ce56e7bd.psf - HIER_NONE/- application/octet-stream
1457510093.153 0 192.168.249.190 TCP_HIT/200 577 HEAD http://au.download.windowsupdate.com/d/msdownload/update/software/updt/2016/02/windows6.1-kb3138612-x64_72ea89719674b61f69a5bbc93d9377e2ce56e7bd.psf - HIER_NONE/- application/octet-stream
Van: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] Namens Heiler Bemerguy
Verzonden: dinsdag 8 maart 2016 23:39
Aan: squid-users at lists.squid-cache.org
Onderwerp: Re: [squid-users] Sudden but sustained high bandwidth usage
While debugging, found this:
2016/03/08 18:22:49.212 kid2| 88,3| client_side_reply.cc(463) cacheHit: clientCacheHit: http://au.v4.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-x64-v5.34_e0074d1fa34d00f8b35e6d5c7be867292222c263.exe, 0 bytes
2016/03/08 18:22:49.212 kid2| 88,3| client_side_reply.cc(470) cacheHit: clientCacheHit: swapin failure for http://au.v4.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-x64-v5.34_e0074d1fa34d00f8b35e6d5c7be867292222c263.exe
This is what generating TCP_SWAPFAIL_MISS for a lot of files, and it redownloads them over and over
Em 07/03/2016 19:08, Heiler Bemerguy escreveu:
I know what is happening. I just don't know how to fix it without breaking windows updates caching.
The "extra" traffic is coming from windows updates mirrors.
acl wupdatecachable url_regex -i (microsoft|windowsupdate)\.com.*\.(cab|exe|ms[i|u|f]|dat|zip|psf|appx|appxbundle|esd)
range_offset_limit none wupdatecachable
Commenting this second line is enough to put an end to the insane downloads. But then all the RANGE requests of windows updates won't be cached.
I don't know how to explain these FD numbers. I'm using EXT4 and I don't know what are vmware cache disks.
Oh, and it seems not only windows updates' range requests are giving trouble:
1457381845.725 241 10.101.1.23 TCP_SWAPFAIL_MISS/206 33123 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.036 293 10.101.1.23 TCP_SWAPFAIL_MISS/206 151352 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.120 66 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.310 66 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.397 60 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.514 63 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.633 65 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.747 63 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.839 63 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381846.949 81 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381847.041 61 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381847.192 118 10.101.1.23 TCP_SWAPFAIL_MISS/206 65897 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381847.447 207 10.101.1.23 TCP_SWAPFAIL_MISS/206 121720 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381862.824 639 10.101.1.23 TCP_SWAPFAIL_MISS/206 33122 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381865.720 2879 10.101.1.23 TCP_SWAPFAIL_MISS/206 221234 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381866.078 343 10.101.1.23 TCP_SWAPFAIL_MISS/206 33130 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381867.107 995 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381867.523 400 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381867.830 291 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381868.082 238 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381868.318 219 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381868.503 171 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381868.695 174 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381868.880 170 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381869.179 286 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381869.424 231 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381869.761 322 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381870.008 232 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381870.236 214 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381870.429 178 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381870.609 174 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381870.805 180 10.101.1.23 TCP_SWAPFAIL_MISS/206 65897 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
1457381871.008 189 10.101.1.23 TCP_SWAPFAIL_MISS/206 65898 GET http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf - HIER_DIRECT/201.49.144.135 application/pdf
Best Regards,
--
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751
Em 07/03/2016 18:00, Eliezer Croitoru escreveu:
On 07/03/2016 22:08, Yuri Voinov wrote:
90 Gb first, 300 Gb second.
Thanks but...
Wouldn't it be much simpler and cheaper to just use WSUS instead all of the hassle??(if it's a closed business environment)
And when does the TCP_SWAPFAIL_MISS happens? always?
And a little tweak for the squid.conf
read_ahead_gap 4096 KB
The above doesn't match your environment bandwidth.
You are just spending too much bandwidth on requests that might not be fulfilled.
Try changing the settings to:
read_ahead_gap 128 KB
and see if it helps with something.
Also since your issue is bandwidth and users are not allowed to the Internet without the proxy I would try to dump the pfsense pf states to see what happens in the network layer, what src IP is consuming all this bandwidth.(or get a wider picture)
Also squid would not be the answer for a fully reasonable network usage, it only comes to help with couple specific things and not to mirror the whole Internet or even just MS as it is.(not saying that your case is a trial such as this)
MS has more then one API that can be mistaken as a download and it can consume more then actually required caching.
I do not know exactly what this means from the info page:
Maximum number of file descriptors: 81920
Largest file desc currently in use: 6157
Number of file desc currently in use: 8216
If the number of FD currently in use is 8216 then the largest file desc currently in use doesn't match.
This specific question might be a bug or expected result but I do not know and Amos or Alex or others might know the answer for this specific info page issue.
Another question I have which might be related(I have experienced such issues with GlusterFS in the past), how is the VM cache disk\s are connected? Is it connected directly to the VM or in the hypervisor level?
How do you mount them(fstab)? are these on the root disk or do you have couple disks mounted?
Did you had the chance to try to use other FS then EXT4? reiserFS? XFS? other?
The above questions are related to the TCP_SWAPFAIL_MISS.
Since there is an issue and you are probably only on the "buffering" testing stage of the cache_dir I would try to somehow reproduce the issue but it's not clear to me what is the exact way to replicate the issue.
Thanks,
Eliezer
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
--
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160309/74eb09c7/attachment-0001.html>
More information about the squid-users
mailing list