[squid-users] rock storage and max-swap-rate

[Alex Rousskov]

On 01/18/2018 03:16 PM, Ivan Larionov wrote:

> cache_dir max-swap-rate documentation says that swap in requests
> contribute to measured swap rate. However in our squid 4 load test we
> see that read_ops + write_ops significantly exceeds the max-swap-rate we
> set and squid doesn't limit it.

In this context, a single Squid "op" is a read or write request from
worker to the disker process. These requests are up to one I/O page in
size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().

* A single read request usually ends up being a single pread(2) system
call that reads at most one I/O page worth of data from disk. See
diskerRead().

* A single write request usually ends up being a single pwrite(2) system
call that writes at most one I/O page worth of data to disk. However, if
that single pwrite() does not write everything a worker has requested to
write, then Squid will make more pwrite() calls, up to 10 calls total.
See diskerWrite().

Within a single cache miss transaction, the rock code should accumulate
small swapout requests from Store into page-size write requests to
disker, but I do not remember how complete those optimizations are: It
is possible that smaller-than-page writes get through to diskers,
increasing the number of write requests. Same for reading cache hits.


What is the "op" in read_ops and write_ops you have measured?


Since Squid does not (and does not want to) have access to low-level
disk stats and since Squid cannot assume exlusive disk ownership, the
rate-limiting feature for rock cache_dirs does not know how many
low-level disk operations the disk is doing and how those operations
correspond to what Squid is asking the disk to do.


HTH,

Alex.


> I tried to set it to 200 to confirm that it actually works and saw that
> it does. Squid started warning about exceeding max-swap-rate. But looks
> like real limit is higher than the value we set in configuration.
> 
> Hardware:
> 
> AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> burstable.
> 
> Config:
> 
> cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> max-swap-rate=1200 slot-size=16384
> 
> IOPS squid pushes under our load test:
> 
> read ~800 ops/sec
> write ~1100 ops/sec
> 
> In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> ops/sec and after spending too much "burst balance" we started getting
> throttled from AWS side.
> 
> Could you please comment on this behavior? What the limit should we set
> to stay under 1500 ops/sec for swap in + swap out operations?
> 
> Thanks.
> 
> Squid version:
> 
> Squid Cache: Version 4.0.22
> Service Name: squid
> configure options:  '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> '--localstatedir=/var' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--enable-epoll'
> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
> 
> -- 
> With best regards, Ivan Larionov.
> 
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>