[squid-users] Sudden but sustained high bandwidth usage
Heiler Bemerguy
heiler.bemerguy at cinbesa.com.br
Tue Mar 8 22:38:56 UTC 2016
While debugging, found this:
2016/03/08 18:22:49.212 kid2| 88,3| client_side_reply.cc(463) cacheHit:
clientCacheHit:
http://au.v4.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-x64-v5.34_e0074d1fa34d00f8b35e6d5c7be867292222c263.exe,
0 bytes
2016/03/08 18:22:49.212 kid2| 88,3| client_side_reply.cc(470) cacheHit:
clientCacheHit: swapin failure for
http://au.v4.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-x64-v5.34_e0074d1fa34d00f8b35e6d5c7be867292222c263.exe
This is what generating TCP_SWAPFAIL_MISS for a lot of files, and it
redownloads them over and over
Em 07/03/2016 19:08, Heiler Bemerguy escreveu:
>
>
> I know what is happening. I just don't know how to fix it without
> breaking windows updates caching.
> The "extra" traffic is coming from windows updates mirrors.
>
> /acl wupdatecachable url_regex -i
> (microsoft|windowsupdate)\.com.*\.(cab|exe|ms[i|u|f]|dat|zip|psf|appx|appxbundle|esd)//
> //range_offset_limit none wupdatecachable/
>
> Commenting this second line is enough to put an end to the insane
> downloads. But then all the RANGE requests of windows updates won't be
> cached.
>
> I don't know how to explain these FD numbers. I'm using EXT4 and I
> don't know what are vmware cache disks.
>
> Oh, and it seems not only windows updates' range requests are giving
> trouble:
>
> 1457381845.725 241 10.101.1.23 TCP_SWAPFAIL_MISS/206 33123 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.036 293 10.101.1.23 TCP_SWAPFAIL_MISS/206 151352 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.120 66 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.310 66 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.397 60 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.514 63 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.633 65 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.747 63 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.839 63 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381846.949 81 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381847.041 61 10.101.1.23 TCP_SWAPFAIL_MISS/206 33129 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381847.192 118 10.101.1.23 TCP_SWAPFAIL_MISS/206 65897 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381847.447 207 10.101.1.23 TCP_SWAPFAIL_MISS/206 121720 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2015-10/1043520/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381862.824 639 10.101.1.23 TCP_SWAPFAIL_MISS/206 33122 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381865.720 2879 10.101.1.23 TCP_SWAPFAIL_MISS/206 221234 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381866.078 343 10.101.1.23 TCP_SWAPFAIL_MISS/206 33130 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381867.107 995 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381867.523 400 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381867.830 291 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381868.082 238 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381868.318 219 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381868.503 171 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381868.695 174 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381868.880 170 10.101.1.23 TCP_SWAPFAIL_MISS/206 33128 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381869.179 286 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381869.424 231 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381869.761 322 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381870.008 232 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381870.236 214 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381870.429 178 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381870.609 174 10.101.1.23 TCP_SWAPFAIL_MISS/206 65896 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381870.805 180 10.101.1.23 TCP_SWAPFAIL_MISS/206 65897 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
> 1457381871.008 189 10.101.1.23 TCP_SWAPFAIL_MISS/206 65898 GET
> http://www.stf.jus.br/arquivo/biblioteca/NovasAquisicoes/2016-02/1022313/sumario.pdf
> - HIER_DIRECT/201.49.144.135 application/pdf
>
> Best Regards,
>
> --
> Heiler Bemerguy - (91) 98151-4894
> Assessor Técnico - CINBESA (91) 3184-1751
>
> Em 07/03/2016 18:00, Eliezer Croitoru escreveu:
>> On 07/03/2016 22:08, Yuri Voinov wrote:
>>> 90 Gb first, 300 Gb second.
>> Thanks but...
>> Wouldn't it be much simpler and cheaper to just use WSUS instead all
>> of the hassle??(if it's a closed business environment)
>> And when does the TCP_SWAPFAIL_MISS happens? always?
>> And a little tweak for the squid.conf
>> read_ahead_gap 4096 KB
>>
>> The above doesn't match your environment bandwidth.
>> You are just spending too much bandwidth on requests that might not
>> be fulfilled.
>> Try changing the settings to:
>> read_ahead_gap 128 KB
>>
>> and see if it helps with something.
>> Also since your issue is bandwidth and users are not allowed to the
>> Internet without the proxy I would try to dump the pfsense pf states
>> to see what happens in the network layer, what src IP is consuming
>> all this bandwidth.(or get a wider picture)
>> Also squid would not be the answer for a fully reasonable network
>> usage, it only comes to help with couple specific things and not to
>> mirror the whole Internet or even just MS as it is.(not saying that
>> your case is a trial such as this)
>> MS has more then one API that can be mistaken as a download and it
>> can consume more then actually required caching.
>>
>> I do not know exactly what this means from the info page:
>> Maximum number of file descriptors: 81920
>> Largest file desc currently in use: 6157
>> Number of file desc currently in use: 8216
>>
>> If the number of FD currently in use is 8216 then the largest file
>> desc currently in use doesn't match.
>> This specific question might be a bug or expected result but I do not
>> know and Amos or Alex or others might know the answer for this
>> specific info page issue.
>>
>> Another question I have which might be related(I have experienced
>> such issues with GlusterFS in the past), how is the VM cache disk\s
>> are connected? Is it connected directly to the VM or in the
>> hypervisor level?
>> How do you mount them(fstab)? are these on the root disk or do you
>> have couple disks mounted?
>> Did you had the chance to try to use other FS then EXT4? reiserFS?
>> XFS? other?
>>
>> The above questions are related to the TCP_SWAPFAIL_MISS.
>> Since there is an issue and you are probably only on the "buffering"
>> testing stage of the cache_dir I would try to somehow reproduce the
>> issue but it's not clear to me what is the exact way to replicate the
>> issue.
>>
>> Thanks,
>> Eliezer
>
>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
--
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160308/496c84b6/attachment-0001.html>
More information about the squid-users
mailing list