[squid-users] Sudden but sustained high bandwidth usage

Mon Mar 14 14:32:35 UTC 2016

Hi Eliezer and Joe!!!

Thank you very much for your support.

I have done a test here too. I've replaced 3.5.15 with 3.5.14 and the 
high bandwidth (associated with SWAPFAIL) is GONE.

I've checked twice the sources diffs between 14 and 15 and can't tell 
what break this.. but I'm running 3.5.14 for 3 days without any 
download-loop sucking all our bandwidth.

I'm still having a SWAPFAIL here and there, and a lot of MISSES for 
files that should have been cached... but no high bandwidth !!

access.log:
1457958469.926    152 10.101.1.163 TCP_MISS/206 6674 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958474.598    356 10.101.1.163 TCP_SWAPFAIL_MISS/206 9142 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958478.292    676 10.101.1.163 TCP_MISS/206 10509 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958481.165    220 10.101.1.163 TCP_MISS/206 11681 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958483.012    958 10.101.1.163 TCP_MISS/206 14003 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958484.334   1155 10.101.1.163 TCP_MISS/206 19973 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958485.967   1630 10.101.1.163 TCP_MISS/206 33511 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958488.804   2835 10.101.1.163 TCP_MISS/206 37266 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958492.630   3824 10.101.1.163 TCP_MISS/206 43918 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958498.908   6274 10.101.1.163 TCP_MISS/206 93504 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream
1457958505.432   6523 10.101.1.163 TCP_MISS/206 72478 GET 
http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2016/03/windows-kb890830-v5.34_1a3d7e16cc62bb4fc4322eabde9cd8c9605f33ce.exe 
- HIER_DIRECT/13.107.4.50 application/octet-stream

conf:
cache_mem 4 GB
maximum_object_size_in_memory 512 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
maximum_object_size 10 GB

cache_dir rock /cache2/rock1 90000 min-size=0 max-size=32768
cache_dir rock /cache/rock1 150000 min-size=32769 max-size=5368725504
cache_dir rock /cache/rock2 150000 min-size=5368725505 max-size=10737418240

acl wupdatecachable url_regex -i 
(microsoft|windowsupdate)\.com.*\.(cab|exe|ms[i|u|f]|dat|zip|psf|appx|appxbundle|esd)
range_offset_limit -1 wupdatecachable
quick_abort_pct 90
refresh_pattern -i 
(microsoft|windowsupdate)\.com.*\.(cab|exe|ms[i|u|f]|dat|zip|psf|appx|appxbundle|esd) 
483840 80% 483840 override-expire ignore-private ignore-no-store

Best Regards,

-- 
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751

Em 13/03/2016 03:32, Eliezer Croitoru escreveu:
> OK it's pretty simple to reproduce on any machine what so ever on 
> 3.5.15-2.
>
> open two terminals on two machines more or less.
> Then run on one the next command
> watch -n 0.2 "http_proxy=http://IP_OP_PROXY:3128/ curl --silent 
> --range 20-40 http://ngtech.co.il/squid/videos/sosp2011_27.mp4 | wc -c"
>
> And on the other one:
> "http_proxy=http://IP_OP_PROXY:3128/ wget 
> http://ngtech.co.il/squid/videos/sosp2011_27.mp4
>
> You will see that when the download is finished and at the same time 
> the download of the partial content is still running, the 
> SWAPFAIL_MISS happens.
>
> If for some reason it won't work for someone(I doubt that there is 
> faster then full env RAM only (hypervisor+vms+disks) so lower the 0.2 
> to 0.1 .
>
> Please file a bug report with the above details,
>
> Eliezer
>
> On 11/03/2016 15:55, Heiler Bemerguy wrote:
>>
>> Hi Eliezer,
>>
>> We usually don't restart it ever. Only recently I've been restarting it
>> because of these issues. The shutdown_lifetime is set to 5 seconds only.
>>
>> We are still getting SWAPFAIL_MISS without any apparent reason, and if
>> it is for a RANGE request, it would multiply it to many parallel
>> connections.... I'd like to track it down but I'm afraid I don't have
>> the knowledge to do it...
>>
>> root at proxy:/var/log/squid# tail -f access.log |grep SWAPF
>> 1457703376.593    383 10.88.100.100 TCP_SWAPFAIL_MISS/206 533 GET
>> http://vdownloader.com/wp-content/uploads/multiwebsite-570x321.png -
>> HIER_DIRECT/104.25.245.28 image/png
>> 1457703376.942    726 10.88.100.100 TCP_SWAPFAIL_MISS/206 530 GET
>> http://vdownloader.com/wp-content/uploads/social-media-share-570x321.jpg
>> - HIER_DIRECT/104.25.244.28 image/jpeg
>> 1457703376.964    746 10.88.100.100 TCP_SWAPFAIL_MISS/206 529 GET
>> http://vdownloader.com/wp-content/uploads/Cnet-logo.png -
>> HIER_DIRECT/104.25.244.28 image/png
>> 1457703540.685 106669 10.101.1.50 TCP_SWAPFAIL_MISS/200 3342325 GET
>> http://www.rarlab.com/rar/wrar531br.exe - HIER_DIRECT/5.135.104.98
>> application/octet-stream
>> 1457703631.055   4088 10.101.1.130 TCP_SWAPFAIL_MISS/206 33062 GET
>> http://www.cetapnet.com.br/arquivos_cetap/arquivos/pma_001_2015_anexo_03.pdf 
>>
>> - HIER_DIRECT/200.219.214.204 application/pdf
>> 1457703637.471   6407 10.101.1.130 TCP_SWAPFAIL_MISS/206 352287 GET
>> http://www.cetapnet.com.br/arquivos_cetap/arquivos/pma_001_2015_anexo_03.pdf 
>>
>> - HIER_DIRECT/200.219.214.204 application/pdf
>> 1457703755.673    262 10.72.0.24 TCP_SWAPFAIL_MISS/206 21736 GET
>> http://ciac.ufpa.br/phocadownload/EDITAL_018-2016-RET-HOM_REC_2-CH-SISU-2016.pdf 
>>
>> - HIER_DIRECT/200.239.64.160 application/pdf
>> 1457703755.712     35 10.72.0.24 TCP_SWAPFAIL_MISS/206 65938 GET
>> http://ciac.ufpa.br/phocadownload/EDITAL_018-2016-RET-HOM_REC_2-CH-SISU-2016.pdf 
>>
>> - HIER_DIRECT/200.239.64.160 application/pdf
>>
>> Best Regards,
>>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users