[squid-users] squid won't cache some files with errors " storeCreate: no swapdirs for..."

Tue Nov 24 15:55:01 UTC 2020

Hello,

I'm having some difficulties understanding some errors I'm getting with squid while trying to act as a cache of a yum repository.

Squid works "mostly" fine and cache "most" of the files going through... but some won't get into the cache and this is quite problematic for us.

It doesn't seem to depend on the size of the file or its name... I'm moving a problematic file and it will get cached just fine.

And no matter how many times I "wget" the problematic file through the proxy, the file won't get into the cache.

Config wise, I've tried multiple things.

SMP / non SMP.

Rock / aufs / in memory only...

The machine its running on is quite beefy (200GB of ram, 48 cores) and does nothing else than running squid.

And so far my config looks like this:

debug_options ALL,1 11,3 31,3 65,3 16,5 20,5 cache_mem                     1 GB maximum_object_size_in_memory 10 MB maximum_object_size           512 MB workers 3 access_log                    daemon:/var/log/squid/access.log squid refresh_pattern . 3600 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private ignore-must-revalidate ignore-reload acl authorized_domains dstdomain .hpc.edf.fr http_access allow authorized_domains http_access allow localhost http_port 10.10.10.10:3142 http_port 127.0.0.1:3142 cache_dir rock /data 3000 max-size=500MB

And without further ado, the errors I'm getting in the logs with some of the files...

----------

HTTP/1.1 200 OK^M

Date: Wed, 18 Nov 2020 14:25:56 GMT^M

Server: Apache/2.4.10 (Debian)^M

Last-Modified: Thu, 08 Oct 2020 14:54:47 GMT^M

ETag: "e6c-5b12a0365d88b"^M

Accept-Ranges: bytes^M

Content-Length: 3692^M

Vary: Accept-Encoding^M

Content-Type: application/xml^M

X-Cache: MISS from vipproxy1.mydomain^M

X-Cache-Lookup: MISS from vipproxy1.mydomain:3142^M

Via: 1.1 vipproxy1.mydomain (squid/5.0.1-VCS)^M

Connection: keep-alive^M

^M

----------

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(814) write: storeWrite: writing 1145 bytes for '3BD6C6FADCADCACA042F71576702C21D'

2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(374) mayStartSwapOut: already allowed

2020/11/18 15:25:56.616 kid3| 11,3| http.cc(1087) persistentConnStatus: conn21 local=123.456.789.15:49420 remote=123.456.789.107:80 HIER_DIRECT FD 41 flags=1 eof=0

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(814) write: storeWrite: writing 2547 bytes for '3BD6C6FADCADCACA042F71576702C21D'

2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(374) mayStartSwapOut: already allowed

2020/11/18 15:25:56.616 kid3| 11,3| http.cc(1087) persistentConnStatus: conn21 local=123.456.789.15:49420 remote=123.456.789.107:80 HIER_DIRECT FD 41 flags=1 eof=0

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(1065) complete: storeComplete: '3BD6C6FADCADCACA042F71576702C21D'

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(1258) validLength: storeEntryValidLength: Checking '3BD6C6FADCADCACA042F71576702C21D'

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(1260) validLength: storeEntryValidLength:     object_len = 4007

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(1261) validLength: storeEntryValidLength:         hdr_sz = 315

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(1262) validLength: storeEntryValidLength: content_length = 3692

2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(374) mayStartSwapOut: already allowed

2020/11/18 15:25:56.616 kid3| 20,5| store_swapout.cc(49) storeSwapOutStart: storeSwapOutStart: Begin SwapOut 'http://repos.mydomain/repos/rpm-hpc/el8/repodata/repomd.xml' to dirno -1, fileno FFFFFFFF

2020/11/18 15:25:56.616 kid3| 20,3| store_swapmeta.cc(52) storeSwapMetaBuild: storeSwapMetaBuild URL: http://repos.mydomain/repos/rpm-hpc/el8/repodata/repomd.xml

2020/11/18 15:25:56.616 kid3| 20,2| store_io.cc(38) storeCreate: storeCreate: no swapdirs for e:tw1343=sp2V/0x559eb98619f0*4

2020/11/18 15:25:56.616 kid3| 20,5| Transients.cc(295) evictCached: e:tw1343=sp2V/0x559eb98619f0*4

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: Client unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*4

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: FwdState unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(442) lock: store_client::copy locked key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: store_client::copy unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3

2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(347) mayStartSwapOut:  already rejected

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(442) lock: storeUnregister locked key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: storeUnregister unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: clientReplyContext::removeStoreReference unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*2

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: ClientHttpRequest::loggingEntry unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*1

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(482) doAbandon: e:tw1343=sp2V/0x559eb98619f0*0 via ClientHttpRequest::loggingEntry

2020/11/18 15:25:56.616 kid3| 20,5| Controller.cc(654) handleIdleEntry: destroying unlocked entry: 0x559eb98619f0 e:tw1343=sp2V/0x559eb98619f0*0

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(398) destroyStoreEntry: destroyStoreEntry: destroying 0x559eb98619f8

2020/11/18 15:25:56.616 kid3| 20,3| store.cc(380) destroyMemObject: 0x559eb9861a70 in e:tw1343=sp2V/0x559eb98619f0*0

2020/11/18 15:25:56.616 kid3| 20,5| Transients.cc(320) disconnect: e:tw1343=sp2V/0x559eb98619f0*0

2020/11/18 15:25:56.616 kid3| 20,3| MemObject.cc(108) ~MemObject: MemObject destructed, this=0x559eb9861a70

2020/11/18 15:25:56.616 kid3| 20,5| store.cc(353) ~StoreEntry: StoreEntry destructed, this=0x559eb98619f0

I'm running a rather recent version of squid that I recompiled from git with the "enable-http-violations" use flag.

Squid Cache: Version 5.0.1-VCS

Service Name: squid

This binary uses OpenSSL 1.1.1c FIPS  28 May 2019. For legal restrictions on distribution see https://www.openssl.org/source/license.html

configure options:  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--libexecdir=/usr/lib64/squid' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,PAM,POP3,RADIUS,SASL,SMB,SMB_LM' '--enable-auth-ntlm=SMB_LM,fake' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos' '--enable-storeid-rewrite-helpers=file' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-diskio' '--enable-wccpv2' '--enable-esi' '--enable-ecap' '--with-aio' '--with-default-user=squid' '--with-dl' '--with-openssl' '--with-pthreads' '--disable-arch-native' '--disable-security-cert-validators' '--with-swapdir=/var/spool/squid' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig' '--enable-http-violations'

We have resorted to use apt-cacher-ng instead of squid for the moment as we are heavily impacted "performance wise" by this lack of caching on some of the files.

If anyone has hints or suggestions of what might be the problem ?

Thx,

Guillaume.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20201124/92c3fba7/attachment-0002.htm>
-------------- next part --------------

Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20201124/92c3fba7/attachment-0003.htm>