[squid-users] rock storage and max-swap-rate

[Ivan Larionov]

Thank you for the fast reply!

read_ops and write_ops is AWS EBS metric and in general it correlates with
OS-level reads/s writes/s stats which iostat shows.

So if I understand you correctly max-swap-rate doesn't limit disk IOPS but
limits squid swap ops instead and every squid operation could in theory use
more than 1 disk IO operation. This means we can't really say "limit swap
ops to 1500 because our disk can handle 1500 iops" but should figure out
the number after testing different values.

Ok, I suppose I'll just do what Rock documentation says – will test
different values and figure out what works for us.

Thanks again.

On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 01/18/2018 03:16 PM, Ivan Larionov wrote:
>
> > cache_dir max-swap-rate documentation says that swap in requests
> > contribute to measured swap rate. However in our squid 4 load test we
> > see that read_ops + write_ops significantly exceeds the max-swap-rate we
> > set and squid doesn't limit it.
>
> In this context, a single Squid "op" is a read or write request from
> worker to the disker process. These requests are up to one I/O page in
> size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().
>
> * A single read request usually ends up being a single pread(2) system
> call that reads at most one I/O page worth of data from disk. See
> diskerRead().
>
> * A single write request usually ends up being a single pwrite(2) system
> call that writes at most one I/O page worth of data to disk. However, if
> that single pwrite() does not write everything a worker has requested to
> write, then Squid will make more pwrite() calls, up to 10 calls total.
> See diskerWrite().
>
> Within a single cache miss transaction, the rock code should accumulate
> small swapout requests from Store into page-size write requests to
> disker, but I do not remember how complete those optimizations are: It
> is possible that smaller-than-page writes get through to diskers,
> increasing the number of write requests. Same for reading cache hits.
>
>
> What is the "op" in read_ops and write_ops you have measured?
>
>
> Since Squid does not (and does not want to) have access to low-level
> disk stats and since Squid cannot assume exlusive disk ownership, the
> rate-limiting feature for rock cache_dirs does not know how many
> low-level disk operations the disk is doing and how those operations
> correspond to what Squid is asking the disk to do.
>
>
> HTH,
>
> Alex.
>
>
> > I tried to set it to 200 to confirm that it actually works and saw that
> > it does. Squid started warning about exceeding max-swap-rate. But looks
> > like real limit is higher than the value we set in configuration.
> >
> > Hardware:
> >
> > AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> > burstable.
> >
> > Config:
> >
> > cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> > max-swap-rate=1200 slot-size=16384
> >
> > IOPS squid pushes under our load test:
> >
> > read ~800 ops/sec
> > write ~1100 ops/sec
> >
> > In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> > ops/sec and after spending too much "burst balance" we started getting
> > throttled from AWS side.
> >
> > Could you please comment on this behavior? What the limit should we set
> > to stay under 1500 ops/sec for swap in + swap out operations?
> >
> > Thanks.
> >
> > Squid version:
> >
> > Squid Cache: Version 4.0.22
> > Service Name: squid
> > configure options:  '--program-prefix=' '--prefix=/usr'
> > '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> > '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> > '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> > '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> > '--localstatedir=/var' '--mandir=/usr/share/man'
> > '--infodir=/usr/share/info' '--enable-epoll'
> > '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> > '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> > '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
> >
> > --
> > With best regards, Ivan Larionov.
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
> >
>
>


-- 
With best regards, Ivan Larionov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180118/39e034e3/attachment-0001.html>