[squid-users] rock storage and max-swap-rate

[Alex Rousskov]

On 01/18/2018 04:04 PM, Ivan Larionov wrote:

> So if I understand you correctly max-swap-rate doesn't limit disk IOPS

Correct. Squid does not know what the disk- or OS-level stats are.


> but limits squid swap ops instead 

If you define a "swap op" as reading or writing a single I/O page for
the purpose of swapping in or swapping out a cached object, then yes.
Processing an HTTP transaction may involve reading and writing many I/O
pages. See my original response for a more precise definition.


> and every squid operation could in
> theory use more than 1 disk IO operation. 

Yes, depending on the file system, disk, etc.

Ideally, rock cache_dirs should use raw disk partitions, but we do not
have a sponsor for that work.


> This means we can't really say
> "limit swap ops to 1500 because our disk can handle 1500 iops" but
> should figure out the number after testing different values.

Correct.


> Ok, I suppose I'll just do what Rock documentation says – will test
> different values and figure out what works for us.

Sounds like a good plan! You may even find a good/stable correlation
between Squid swap I/O ops and low-level disk I/O ops. Please consider
reporting your findings for others to reuse.


Cheers,

Alex.


> On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov wrote:
> 
>     On 01/18/2018 03:16 PM, Ivan Larionov wrote:
> 
>     > cache_dir max-swap-rate documentation says that swap in requests
>     > contribute to measured swap rate. However in our squid 4 load test we
>     > see that read_ops + write_ops significantly exceeds the max-swap-rate we
>     > set and squid doesn't limit it.
> 
>     In this context, a single Squid "op" is a read or write request from
>     worker to the disker process. These requests are up to one I/O page in
>     size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().
> 
>     * A single read request usually ends up being a single pread(2) system
>     call that reads at most one I/O page worth of data from disk. See
>     diskerRead().
> 
>     * A single write request usually ends up being a single pwrite(2) system
>     call that writes at most one I/O page worth of data to disk. However, if
>     that single pwrite() does not write everything a worker has requested to
>     write, then Squid will make more pwrite() calls, up to 10 calls total.
>     See diskerWrite().
> 
>     Within a single cache miss transaction, the rock code should accumulate
>     small swapout requests from Store into page-size write requests to
>     disker, but I do not remember how complete those optimizations are: It
>     is possible that smaller-than-page writes get through to diskers,
>     increasing the number of write requests. Same for reading cache hits.
> 
> 
>     What is the "op" in read_ops and write_ops you have measured?
> 
> 
>     Since Squid does not (and does not want to) have access to low-level
>     disk stats and since Squid cannot assume exlusive disk ownership, the
>     rate-limiting feature for rock cache_dirs does not know how many
>     low-level disk operations the disk is doing and how those operations
>     correspond to what Squid is asking the disk to do.
> 
> 
>     HTH,
> 
>     Alex.
> 
> 
>     > I tried to set it to 200 to confirm that it actually works and saw
>     that
>     > it does. Squid started warning about exceeding max-swap-rate. But
>     looks
>     > like real limit is higher than the value we set in configuration.
>     >
>     > Hardware:
>     >
>     > AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
>     > burstable.
>     >
>     > Config:
>     >
>     > cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
>     > max-swap-rate=1200 slot-size=16384
>     >
>     > IOPS squid pushes under our load test:
>     >
>     > read ~800 ops/sec
>     > write ~1100 ops/sec
>     >
>     > In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
>     > ops/sec and after spending too much "burst balance" we started getting
>     > throttled from AWS side.
>     >
>     > Could you please comment on this behavior? What the limit should
>     we set
>     > to stay under 1500 ops/sec for swap in + swap out operations?
>     >
>     > Thanks.
>     >
>     > Squid version:
>     >
>     > Squid Cache: Version 4.0.22
>     > Service Name: squid
>     > configure options:  '--program-prefix=' '--prefix=/usr'
>     > '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
>     > '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
>     > '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
>     > '--datadir=/usr/share' '--sharedstatedir=/usr/com'
>     > '--localstatedir=/var' '--mandir=/usr/share/man'
>     > '--infodir=/usr/share/info' '--enable-epoll'
>     > '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
>     > '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
>     > '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
>     >
>     > --
>     > With best regards, Ivan Larionov.
>     >
>     >
>     > _______________________________________________
>     > squid-users mailing list
>     > squid-users at lists.squid-cache.org
>     <mailto:squid-users at lists.squid-cache.org>
>     > http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
>     >
> 
> 
> 
> 
> -- 
> With best regards, Ivan Larionov.