[squid-users] rock storage integrity

[Amos Jeffries]

On 5/12/2015 4:37 a.m., Hussam Al-Tayeb wrote:
> Hi. I am using squid with rock storage right now to cache computer
> updates for my Linux computers. It works well.
> Since this is a database, it is possible for part of the database to
> get corrupted through a crash or incorrect poweroff?
> I know from sql database that incorrect shutdowns can cause binary
> corruption.
> 
> I had an incorrect poweroff yesterday but cache.log did not list
> anything weird.

Such types of corruption only happen if files were actively in the
process of writing data when the power went out (read does not count).

The way Squid uses a separate memory cache as the front-line I/O area
and Diskers cache as a secondary the risks of that happening are a bit
lower than with DB services that process directly to disk.


Also, unless you tuned them to a different size the rock slots/cells are
sized to match the OS natural page sizes. So there is almost no delay in
the wrote(2) within which the corruption can happen.

Your underlying FS can also play a part in preventing the corruption.

> 
> 2015/12/03 01:00:11| Store rebuilding is 0.31% complete
> 2015/12/03 01:01:00| Finished rebuilding storage from disk.
> 2015/12/03 01:01:00|    319999 Entries scanned
> 2015/12/03 01:01:00|         0 Invalid entries.
> 2015/12/03 01:01:00|         0 With invalid flags.
> 2015/12/03 01:01:00|     55523 Objects loaded.
> 2015/12/03 01:01:00|         0 Objects expired.
> 2015/12/03 01:01:00|         0 Objects cancelled.
> 2015/12/03 01:01:00|         0 Duplicate URLs purged.
> 2015/12/03 01:01:00|         0 Swapfile clashes avoided.
> 2015/12/03 01:01:00|   Took 49.94 seconds (1111.79 objects/sec).
> 2015/12/03 01:01:00| Beginning Validation Procedure
> 2015/12/03 01:01:00|   Completed Validation Procedure
> 2015/12/03 01:01:00|   Validated 0 Entries
> 2015/12/03 01:01:00|   store_swap_size = 3187216.00 KB
> 
> Nevertheless, what would be the best way to check if there was some
> damage to the database (unusable slots/cells/whatever)?

Squid does that automatically on (every) startup. The log lines you
quote above are the scan happening and its results.

If Squid had found any corruption there would have been a non-0 value on
the lines:

> 2015/12/03 01:01:00|         0 Invalid entries.
> 2015/12/03 01:01:00|         0 With invalid flags.

and possibly some WARNING notices above the "Finished rebuilding" line
if important problems were found.

Amos