[squid-users] AUFS vs. DISKS
FredB
fredbmail at free.fr
Mon Jul 20 08:46:00 UTC 2015
>
> Fred and Fred;
>
> Could you guys who have been seeing these warnings logged please
> present a grep of those cache.log lines so I can get a better handle
> on
> how many doublings your queues are actually requiring ?
>
> I count 5 and 6 warnings respectively in FredB's two earlier log
> traces.
> So 8*(2^6)=512 entries with an extra x2 normal I/O thread count
> making
> it about 1K queue entries required minimum.
>
> Cheers
> Amos
>
>
Hi,
Tested one hour.
cache_dir aufs /cache1 250000 512 512
cache_dir aufs /cache2 250000 512 512
/dev/sdb1 413G 223G 170G 57% /cache1
/dev/sdc1 413G 223G 170G 57% /cache2
Squid Cache: Version 3.4.13
configure options: '--build=x86_64-linux-gnu' '--prefix=/' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/lib/squid' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--mandir=/usr/share/man' '--enable-inline' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-auth-negotiate--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-basic-auth-helpers=LDAP,digest' '--enable-digest-auth-helpers=ldap,password' '--enable-arp-acl' '--enable-esi' '--disable-translation' '--with-logdir=/var/log/squid' '--with-filedescriptors=65536' '--with-large-files' '--disable-snmp' '--with-default-user=squid' '--disable-ipv6' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -g -Wall -O2' 'LDFLAGS=' 'CPPFLAGS=' 'CXXFLAGS=-g -O2 -g -Wall -O2' '--enable-ltdl-convenience'
grep "2015:09:40:" /var/log/squid/access.log | wc -l
16408 -> 273 r/s
grep "2015:10:00:" /var/log/squid/access.log | wc -l
18259 -> 304 r/s
grep "2015:10:20:" /var/log/squid/access.log | wc -l
17690 -> 294 r/s
grep "2015:10:30:" /var/log/squid/access.log | wc -l
18294 -> 304 r/s
Unfortunately actually the load is reduced by 2 or 3, but I think it still interesting
2015/07/20 09:34:50 kid1| Store rebuilding is 79.56% complete
2015/07/20 09:35:04 kid1| Done reading /cache2 swaplog (9705370 entries)
2015/07/20 09:35:04 kid1| Done reading /cache1 swaplog (9782135 entries)
2015/07/20 09:35:04 kid1| Finished rebuilding storage from disk.
2015/07/20 09:35:04 kid1| 19487505 Entries scanned
2015/07/20 09:35:04 kid1| 0 Invalid entries.
2015/07/20 09:35:04 kid1| 0 With invalid flags.
2015/07/20 09:35:04 kid1| 19487505 Objects loaded.
2015/07/20 09:35:04 kid1| 0 Objects expired.
2015/07/20 09:35:04 kid1| 0 Objects cancelled.
2015/07/20 09:35:04 kid1| 0 Duplicate URLs purged.
2015/07/20 09:35:04 kid1| 0 Swapfile clashes avoided.
2015/07/20 09:35:04 kid1| Took 58.79 seconds (331500.06 objects/sec).
2015/07/20 09:35:04 kid1| Beginning Validation Procedure
2015/07/20 09:35:05 kid1| 262144 Entries Validated so far.
2015/07/20 09:35:05 kid1| 524288 Entries Validated so far.
2015/07/20 09:35:05 kid1| 786432 Entries Validated so far.
2015/07/20 09:35:05 kid1| 1048576 Entries Validated so far.
2015/07/20 09:35:05 kid1| 1310720 Entries Validated so far.
2015/07/20 09:35:05 kid1| 1572864 Entries Validated so far.
2015/07/20 09:35:05 kid1| 1835008 Entries Validated so far.
2015/07/20 09:35:05 kid1| 2097152 Entries Validated so far.
2015/07/20 09:35:05 kid1| 2359296 Entries Validated so far.
2015/07/20 09:35:05 kid1| 2621440 Entries Validated so far.
2015/07/20 09:35:05 kid1| 2883584 Entries Validated so far.
2015/07/20 09:35:05 kid1| 3145728 Entries Validated so far.
2015/07/20 09:35:05 kid1| 3407872 Entries Validated so far.
2015/07/20 09:35:05 kid1| 3670016 Entries Validated so far.
2015/07/20 09:35:05 kid1| 3932160 Entries Validated so far.
2015/07/20 09:35:05 kid1| 4194304 Entries Validated so far.
2015/07/20 09:35:05 kid1| 4456448 Entries Validated so far.
2015/07/20 09:35:05 kid1| 4718592 Entries Validated so far.
2015/07/20 09:35:05 kid1| 4980736 Entries Validated so far.
2015/07/20 09:35:05 kid1| 5242880 Entries Validated so far.
2015/07/20 09:35:06 kid1| 5505024 Entries Validated so far.
2015/07/20 09:35:06 kid1| 5767168 Entries Validated so far.
2015/07/20 09:35:06 kid1| 6029312 Entries Validated so far.
2015/07/20 09:35:06 kid1| 6291456 Entries Validated so far.
2015/07/20 09:35:06 kid1| 6553600 Entries Validated so far.
2015/07/20 09:35:06 kid1| 6815744 Entries Validated so far.
2015/07/20 09:35:06 kid1| 7077888 Entries Validated so far.
2015/07/20 09:35:06 kid1| 7340032 Entries Validated so far.
2015/07/20 09:35:06 kid1| 7602176 Entries Validated so far.
2015/07/20 09:35:06 kid1| 7864320 Entries Validated so far.
2015/07/20 09:35:06 kid1| 8126464 Entries Validated so far.
2015/07/20 09:35:06 kid1| 8388608 Entries Validated so far.
2015/07/20 09:35:06 kid1| 8650752 Entries Validated so far.
2015/07/20 09:35:06 kid1| 8912896 Entries Validated so far.
2015/07/20 09:35:06 kid1| 9175040 Entries Validated so far.
2015/07/20 09:35:06 kid1| 9437184 Entries Validated so far.
2015/07/20 09:35:06 kid1| 9699328 Entries Validated so far.
2015/07/20 09:35:06 kid1| 9961472 Entries Validated so far.
2015/07/20 09:35:06 kid1| 10223616 Entries Validated so far.
2015/07/20 09:35:06 kid1| 10485760 Entries Validated so far.
2015/07/20 09:35:06 kid1| 10747904 Entries Validated so far.
2015/07/20 09:35:07 kid1| 11010048 Entries Validated so far.
2015/07/20 09:35:07 kid1| 11272192 Entries Validated so far.
2015/07/20 09:35:07 kid1| 11534336 Entries Validated so far.
2015/07/20 09:35:07 kid1| 11796480 Entries Validated so far.
2015/07/20 09:35:07 kid1| 12058624 Entries Validated so far.
2015/07/20 09:35:07 kid1| 12320768 Entries Validated so far.
2015/07/20 09:35:07 kid1| 12582912 Entries Validated so far.
2015/07/20 09:35:07 kid1| 12845056 Entries Validated so far.
2015/07/20 09:35:07 kid1| 13107200 Entries Validated so far.
2015/07/20 09:35:07 kid1| 13369344 Entries Validated so far.
2015/07/20 09:35:07 kid1| 13631488 Entries Validated so far.
2015/07/20 09:35:07 kid1| 13893632 Entries Validated so far.
2015/07/20 09:35:07 kid1| 14155776 Entries Validated so far.
2015/07/20 09:35:07 kid1| 14417920 Entries Validated so far.
2015/07/20 09:35:07 kid1| 14680064 Entries Validated so far.
2015/07/20 09:35:07 kid1| 14942208 Entries Validated so far.
2015/07/20 09:35:07 kid1| 15204352 Entries Validated so far.
2015/07/20 09:35:07 kid1| 15466496 Entries Validated so far.
2015/07/20 09:35:07 kid1| 15728640 Entries Validated so far.
2015/07/20 09:35:07 kid1| 15990784 Entries Validated so far.
2015/07/20 09:35:08 kid1| 16252928 Entries Validated so far.
2015/07/20 09:35:08 kid1| 16515072 Entries Validated so far.
2015/07/20 09:35:08 kid1| 16777216 Entries Validated so far.
2015/07/20 09:35:08 kid1| 17039360 Entries Validated so far.
2015/07/20 09:35:08 kid1| 17301504 Entries Validated so far.
2015/07/20 09:35:08 kid1| 17563648 Entries Validated so far.
2015/07/20 09:35:08 kid1| 17825792 Entries Validated so far.
2015/07/20 09:35:08 kid1| 18087936 Entries Validated so far.
2015/07/20 09:35:08 kid1| 18350080 Entries Validated so far.
2015/07/20 09:35:08 kid1| 18612224 Entries Validated so far.
2015/07/20 09:35:08 kid1| 18874368 Entries Validated so far.
2015/07/20 09:35:08 kid1| 19136512 Entries Validated so far.
2015/07/20 09:35:08 kid1| 19398656 Entries Validated so far.
2015/07/20 09:35:08 kid1| Completed Validation Procedure
2015/07/20 09:35:08 kid1| Validated 19487497 Entries
2015/07/20 09:35:08 kid1| store_swap_size = 460804616.00 KB
2015/07/20 09:35:08 kid1| storeLateRelease: released 0 objects
2015/07/20 09:35:14 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:35:16 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:35:37 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:36:17 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:37:02 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:37:23 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 09:38:46 kid1| clientIfRangeMatch: Weak ETags are not allowed in If-Range: "34860-1341477620000" ? "34860-1341477620000"
2015/07/20 09:39:47 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:44:00 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:48:04 kid1| Could not parse headers from on disk object
2015/07/20 09:48:04 kid1| varyEvaluateMatch: Oops. Not a Vary object on second attempt, 'http://www.nafnaf.com/min/?f=/js/modernizr/modernizr.js,/js/jquery/lib/jquery/jquery.min.js,/js/jquery/lib/jquery/jquery-ui.
custom.min.js,/js/jquery/lib/jquery.easing.js,/js/jquery/lib/jquery.base64.js,/js/jquery/lib/jquery.cookie.js,/js/swiper/idangerous.swiper.min.js,/js/jquery/lib/jquery.popupwindows.js,/js/jquery/lib/fancybox/jquer
y.fancybox.pack.js,/js/jquery/lib/validate/jquery.validate.js,/js/jquery/lib/validate/additional-methods.js,/js/jquery/lib/jquery.ba-hashchange.min.js,/js/bootstrap/js/bootstrap.min.js,/js/bootstrap/js/bootstrap-d
ropdownhover.js,/js/jquery/page/jquery.ajax.js,/js/iscroll/iscroll.js,/js/jquery/page/notification/jquery.cookie.js,/js/jquery/page/pageslide/jquery.pageslide.js,/js/jquery/page/innermenu/jquery.innermenu.js,/js/j
query/customer/jquery.customer.js,/js/jquery/catalog/product/jquery.quickview.js,/js/jquery/lib/jquery.unveil.min.js,/js/jquery/catalog/product/jquery.addto.js,/js/jquery/catalog/product/jquery.quantity.js,/js/jqu
ery/checkout/cart/jquery.quantity.js,/js/jquery/checkout/cart/jquery.summary.js,/js/jquery/checkout/cart/jquery.confirmation.js,/js/jquery/wishlist/jquery.quantity.js,/js/jquery/wishlist/jquery.addto.js,/js/jquery
/wishlist/jquery.confirmation.js,/js/jquery/newsletter/jquery.newsletter.js,/js/mgt_developertoolbar/mgt_developertoolbar.js,/js/jquery/analytics/jquery.analytics.js,/js/jquery/analytics/gua/jquery.gua-main.js,/js
/jquery/analytics/gua/jquery.gua-addto.js,/js/jquery/analytics/gua/jquery.gua-customer.js,/js/jquery/analytics/gua/jquery.gua-newsletter.js,/js/jquery/analytics/jquery.mastertrack.js,/js/jquery/redirect/localizati
on/countries_locales_en.js,/js/jquery/redirect/jquery.redirect.js,/js/jquery/catalog/jquery.catalogurl.js,/js/jquery/catalog/jquery.catalogmetas.js,/js/jquery/catalog/product/view/media/jquery.media.js,/js/jquery/
catalog/product/jquery.product.js,/js/jquery/checkout/cart/jquery.addto.js,/js/jquery/reports/jquery.reports.js,/js/jquery/productalert/jquery.productalert.js,/js/jquery/analytics/jquery.tagcommander.js,/js/jquery
/analytics/jquery.eulerianretargeting.js,/js/jquery/analytics/gua/jquery.gua-product-options.js,/js/jquery/social/jquery.sendfriend.js,/js/jquery/webtostore/jquery.socloz.js,/js/jquery/lib/selectBoxIt/jquery.selec
tBoxIt.min.js,/js/jquery/social/jquery.social.js,/js/jquery/lib/validate/localization/messages_fr.js,/js/jquery/redirect/localization/countries_locales_fr.js,/skin/frontend/eplatform/default/js/efunctions.js,/skin
/frontend/eplatform/default/js/eforms.js,/skin/frontend/eplatform/nafnaf/js/functions.js&1434729537' 'accept-encoding="identity,gzip,deflate"'
2015/07/20 09:48:04 kid1| clientProcessHit: Vary object loop!
2015/07/20 09:50:42 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 09:51:06 kid1| Could not parse headers from on disk object
2015/07/20 09:51:06 kid1| varyEvaluateMatch: Oops. Not a Vary object on second attempt, 'http://www.nafnaf.com/min/?f=/js/modernizr/modernizr.js,/js/jquery/lib/jquery/jquery.min.js,/js/jquery/lib/jquery/jquery-ui.
custom.min.js,/js/jquery/lib/jquery.easing.js,/js/jquery/lib/jquery.base64.js,/js/jquery/lib/jquery.cookie.js,/js/swiper/idangerous.swiper.min.js,/js/jquery/lib/jquery.popupwindows.js,/js/jquery/lib/fancybox/jquer
y.fancybox.pack.js,/js/jquery/lib/validate/jquery.validate.js,/js/jquery/lib/validate/additional-methods.js,/js/jquery/lib/jquery.ba-hashchange.min.js,/js/bootstrap/js/bootstrap.min.js,/js/bootstrap/js/bootstrap-d
ropdownhover.js,/js/jquery/page/jquery.ajax.js,/js/iscroll/iscroll.js,/js/jquery/page/notification/jquery.cookie.js,/js/jquery/page/pageslide/jquery.pageslide.js,/js/jquery/page/innermenu/jquery.innermenu.js,/js/j
query/customer/jquery.customer.js,/js/jquery/catalog/product/jquery.quickview.js,/js/jquery/lib/jquery.unveil.min.js,/js/jquery/catalog/product/jquery.addto.js,/js/jquery/catalog/product/jquery.quantity.js,/js/jqu
ery/checkout/cart/jquery.quantity.js,/js/jquery/checkout/cart/jquery.summary.js,/js/jquery/checkout/cart/jquery.confirmation.js,/js/jquery/wishlist/jquery.quantity.js,/js/jquery/wishlist/jquery.addto.js,/js/jquery
/wishlist/jquery.confirmation.js,/js/jquery/newsletter/jquery.newsletter.js,/js/mgt_developertoolbar/mgt_developertoolbar.js,/js/jquery/analytics/jquery.analytics.js,/js/jquery/analytics/gua/jquery.gua-main.js,/js
/jquery/analytics/gua/jquery.gua-addto.js,/js/jquery/analytics/gua/jquery.gua-customer.js,/js/jquery/analytics/gua/jquery.gua-newsletter.js,/js/jquery/analytics/jquery.mastertrack.js,/js/jquery/redirect/localizati
on/countries_locales_en.js,/js/jquery/redirect/jquery.redirect.js,/js/jquery/checkout/cart/jquery.cart.js,/js/jquery/checkout/threestep/jquery.threestep.js,/js/jquery/analytics/gua/checkout/threestep/jquery.gua-fu
nnel.js,/js/jquery/analytics/jquery.tagcommander.js,/js/jquery/analytics/jquery.eulerianretargeting.js,/js/jquery/lib/validate/localization/messages_fr.js,/js/jquery/redirect/localization/countries_locales_fr.js,/
skin/frontend/eplatform/default/js/efunctions.js,/skin/frontend/eplatform/default/js/eforms.js,/skin/frontend/eplatform/nafnaf/js/functions.js,/skin/frontend/eplatform/default/js/checkout/threestep.js,/skin/fronte
nd/eplatform/default/js/checkout.js&1430749814' 'accept-encoding="identity,gzip,deflate"'
2015/07/20 09:51:06 kid1| clientProcessHit: Vary object loop!
2015/07/20 09:53:11 kid1| clientIfRangeMatch: Weak ETags are not allowed in If-Range: "34028-1397030626000" ? "34028-1397030626000"
2015/07/20 09:55:55 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 09:58:18 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:02:19 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:06:54 kid1| squidaio_queue_request: WARNING - Queue congestion
2015/07/20 10:11:21 kid1| clientIfRangeMatch: Weak ETags are not allowed in If-Range: "32422-1436345442000" ? "32422-1436345442000"
2015/07/20 10:11:36 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:11:47 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:12:20 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:12:25 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:12:32 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:12:36 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:12:39 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:12:44 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:15:00 kid1| Could not parse headers from on disk object
2015/07/20 10:15:00 kid1| varyEvaluateMatch: Oops. Not a Vary object on second attempt, 'http://www.aigle.com/min/?f=/js/modernizr/modernizr.js,/js/jquery/lib/jquery/jquery.min.js,/js/jquery/lib/jquery/jquery-ui.c
ustom.min.js,/js/jquery/lib/jquery.easing.js,/js/jquery/lib/jquery.base64.js,/js/jquery/lib/jquery.cookie.js,/js/swiper/idangerous.swiper.min.js,/js/jquery/lib/selectBoxIt/jquery.selectBoxIt.min.js,/js/jquery/lib/
jquery.popupwindows.js,/js/jquery/lib/fancybox/jquery.fancybox.pack.js,/js/jquery/lib/validate/jquery.validate.js,/js/jquery/lib/validate/additional-methods.js,/js/jquery/lib/jquery.ba-hashchange.min.js,/js/bootst
rap/js/bootstrap.min.js,/js/bootstrap/js/bootstrap-dropdownhover.js,/js/jquery/page/jquery.ajax.js,/js/iscroll/iscroll.js,/js/jquery/page/notification/jquery.cookie.js,/js/jquery/page/pageslide/jquery.pageslide.js
,/js/jquery/page/innermenu/jquery.innermenu.js,/js/jquery/customer/jquery.customer.js,/js/jquery/catalog/product/jquery.quickview.js,/js/jquery/lib/jquery.unveil.min.js,/js/jquery/catalog/product/jquery.addto.js,/
js/jquery/catalog/product/jquery.quantity.js,/js/jquery/checkout/cart/jquery.quantity.js,/js/jquery/checkout/cart/jquery.summary.js,/js/jquery/checkout/cart/jquery.confirmation.js,/js/jquery/wishlist/jquery.quanti
ty.js,/js/jquery/wishlist/jquery.addto.js,/js/jquery/wishlist/jquery.confirmation.js,/js/jquery/newsletter/jquery.newsletter.js,/js/mgt_developertoolbar/mgt_developertoolbar.js,/js/jquery/analytics/jquery.analytic
s.js,/js/jquery/analytics/jquery.mastertrack.js,/js/jquery/analytics/gtm/jquery.gtm-main.js,/js/jquery/analytics/gtm/jquery.gtm-newsletter.js,/js/jquery/analytics/gtm/jquery.gtm-cart.js,/js/jquery/analytics/gtm/jq
uery.gtm-wishlist.js,/js/jquery/analytics/gtm/jquery.gtm-quickview.js,/js/jquery/redirect/localization/countries_locales_en.js,/js/jquery/redirect/jquery.redirect.js,/js/jquery/productalert/jquery.productalert.js,
/js/jquery/catalog/jquery.catalogurl.js,/js/jquery/catalog/jquery.catalogmetas.js,/js/jquery/catalog/category/jquery.category.js,/js/jquery/catalog/category/jquery.featured.js,/js/jquery/analytics/jquery.tagcomman
der.js,/js/jquery/analytics/gtm/jquery.gtm-category.js,/js/jquery/lib/validate/localization/messages_fr.js,/js/jquery/redirect/localization/countries_locales_fr.js,/skin/frontend/eplatform/default/js/efunctions.js
,/skin/frontend/eplatform/default/js/eforms.js,/skin/frontend/eplatform/aigle/js/functions.js&1429540234' 'accept-encoding="identity,gzip,deflate"'
2015/07/20 10:15:00 kid1| clientProcessHit: Vary object loop!
2015/07/20 10:20:49 kid1| clientIfRangeMatch: Weak ETags are not allowed in If-Range: "b2222bfe4fbed01:0" ? "537965ecbcc2d01:0"
2015/07/20 10:22:50 kid1| urlParse: Illegal hostname '.xiti.com'
2015/07/20 10:30:03 kid1| urlParse: Illegal hostname '.xiti.com'
More information about the squid-users
mailing list