[squid-users] WARNING: Ignoring cache entry due to a SIZE MISMATCH

[Amos Jeffries]

On 18/02/2017 3:18 a.m., Heiler Bemerguy wrote:
> 
> Em 17/02/2017 09:44, Amos Jeffries escreveu:
>> On 16/02/2017 10:40 a.m., Heiler Bemerguy wrote:
>>> Is it normal, on every restart?
>>>
>> Well, this is a check that is only performed on restart. So in a way it
>> is "normal" that it occurs on restart. It should not happen at all
>> though. AFAIK, it is a sign of cache corruption...
> 
> Right. But cache corruption, by a hardware/disc error, can not be. These
> disks were formatted and tested, 3 times already, while trying to solve
> this errors and some other stuff.
> 

Hardware errors are the most well-known, but not the only source of
corruption.

The second most common source of cache corruption is Squid being forced
to shutdown before it has fully saved the cache data. These messages are
the rock equivalent of the UFS "No such file or directory" warnings on
startup/restart.

The chances of these issues showing up increases with the size of the
cache, size of traffic going through the proxy at the shutdown time, and
shortness of the shutdown wait period.
Your 480 GB of rock caches is something I would expect to have at least
some issues with.

> 
>>> maximum_object_size 2 GB
>>> cache_dir rock /cache  120000 min-size=0 max-size=12288
>>> slot-size=12288 max-swap-rate=250 swap-timeout=350
>>> cache_dir rock /cache2 120000 min-size=10240 max-size=65536
>>> max-swap-rate=250 swap-timeout=360
>>>
>> These caches cannot store objects larger than 64KB. So where do you
>> expect the 64KB thru 2GB objects to go?
> 
> There are another cache_dirs just below these, like:
> cache_dir rock /cache3 120000 min-size=65537 max-size=262144
> max-swap-rate=250 swap-timeout=380
> cache_dir rock /cache4 120000 min-size=262145 max-swap-rate=250
> swap-timeout=500
> 

Ah. Okay.

> 
>>
>>> 2017/02/15 18:38:50 kid7| WARNING: Ignoring cache entry due to a SIZE
>>> MISMATCH 5852!=5853
>>> 2017/02/15 18:38:51 kid7| WARNING: Ignoring cache entry due to a SIZE
>>> MISMATCH 6223!=6222
>>> 2017/02/15 18:38:51 kid7| WARNING: Ignoring cache entry due to a SIZE
>>> MISMATCH 4198!=4202
>> ...
>>
>> That last line means the rock database contains an object of size 4198
>> bytes, with 4202 bytes of data in it.
>>
>>   4202 > 4198. So 4 bytes of what? the next database slot? padding data?
>>
>>
>> You have any custom patches applied to this Squid? what version is it?
>>
> 
> No patches. Squid Cache: Version 4.0.18 configure options:

Okay problems in alpha and beta code should be brought up in squid-dev
so the guys working on the changes can actually find out about problems.

"If you have any problems with a development release please write to our
... squid-dev at squid-cache.org lists. DO NOT write to squid-users with
code-related problems."


> '--enable-htcp' '--disable-maintainer-mode'
> '--disable-dependency-tracking' '--disable-wccp' '--disable-snmp'
> '--enable-inline' '--enable-async-io=32' '--enable-storeio=aufs,rock'
> '--enable-underscores' '--enable-removal-policies=lru,heap'
> '--enable-http-violations' '--disable-ident-lookups'
> '--with-large-files' '--enable-ssl' '--enable-ltdl-convenience'
> '--prefix=/usr' '--localstatedir=/var' '--libexecdir=/lib/squid'
> '--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
> '--with-default-user=proxy' '--with-logdir=/var/log'
> '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=16384'
> '--with-aufs-threads=32' '--disable-translation'
> 
> My only */Suspicion/* is the different slot-size from kid7 (/cache). But
> these messages also appears on kid8 (/cache2), although  with much, MUCH
> less frequency..... (while rebuilding/restarting, always)
> 

I dont think that matters in this case. The slot size is a multiple (3x)
of the common disk I/O page size (4KB).

The sizes being mentioned by Squid are the amont of data stored in the
slot. The slot metadata says there are N bytes inside, the object
metadata says it is M bytes big ... " SIZE MISMATCH   M != N".

If "kill -9" or equivalent was used on kid7 during previous shutdown.
Then is will not be able to save all the slot updates it is probably
still waiting to write to disk. Resulting in what you see.

Amos