[squid-dev] Rock store stopped accessing discs

[Alex Rousskov]

On 03/07/2017 08:40 AM, Heiler Bemerguy wrote:
> I'm using squid 4.0.18
> 
> And noticed something: (iostat -x 5)
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda               0,00     0,00    0,00    0,25     0,00 28,00 224,00     0,00    8,00    0,00    8,00   8,00   0,20
> sdc               0,00     0,00    0,00    0,00     0,00 0,00    0,00     0,00    0,00    0,00    0,00   0,00   0,00
> sdb               0,00     0,00    0,00    0,00     0,00 0,00    0,00     0,00    0,00    0,00    0,00   0,00   0,00
> sdd               0,00     0,00    0,00    0,00     0,00 0,00    0,00     0,00    0,00    0,00    0,00   0,00   0,00
> 
> No hds are being accessed, only the main (SDA) one (which logs are
> saved). Btw squid is sending 80mbit/s to the network, as iftop told me.
> 
> cache.log:
> 
> 2017/03/07 05:23:59 kid5| ERROR: worker I/O push queue for /cache4/rock overflow: ipcIo5.206991w9
> 2017/03/07 05:24:10 kid5| WARNING: communication with /cache4/rock may be too slow or disrupted for about 7.00s; rescued 304 out of 304 I/Os
> 2017/03/07 08:00:30 kid5| WARNING: abandoning 1 /cache2/rock I/Os after at least 7.00s timeout
> 2017/03/07 10:50:45 kid5| WARNING: abandoning 1 /cache2/rock I/Os after at least 7.00s timeout

I presume your iostat output covers 5 seconds. The cache.log output
spans 5 hours. Were there no cache disk traffic during those 5 hours? Do
those iostat 5 seconds match the timestamp of any single cache.log WARNING?


> squid.conf:
> 
> cache_dir rock /cache2 110000 min-size=0 max-size=65536 max-swap-rate=200 swap-timeout=360
> cache_dir rock /cache3 110000 min-size=65537 max-size=262144 max-swap-rate=200 swap-timeout=380
> cache_dir rock /cache4 110000 min-size=262145 max-swap-rate=200 swap-timeout=500
> 
> Should I raise any values? tweak something?

Yes, but it is not yet clear what. If you suspect that your disks cannot
handle the load, decrease max-swap-rate. However, there is currently no
firm evidence that your disks cannot handle the load. It could be
something else like insufficient IPC RAM or Squid bugs.

Any Squid kid crashes? How many Squid workers do you use?

Can you collect enough iostat 5-second outputs to correlate with
long-term cache.log messages? I would also collect other system activity
during those hours. The "atop" tool may be useful for collecting
everything in one place.

Alex.