[squid-users] rock storage and max-swap-rate
rousskov at measurement-factory.com
Thu Jan 18 23:22:51 UTC 2018
On 01/18/2018 04:04 PM, Ivan Larionov wrote:
> So if I understand you correctly max-swap-rate doesn't limit disk IOPS
Correct. Squid does not know what the disk- or OS-level stats are.
> but limits squid swap ops instead
If you define a "swap op" as reading or writing a single I/O page for
the purpose of swapping in or swapping out a cached object, then yes.
Processing an HTTP transaction may involve reading and writing many I/O
pages. See my original response for a more precise definition.
> and every squid operation could in
> theory use more than 1 disk IO operation.
Yes, depending on the file system, disk, etc.
Ideally, rock cache_dirs should use raw disk partitions, but we do not
have a sponsor for that work.
> This means we can't really say
> "limit swap ops to 1500 because our disk can handle 1500 iops" but
> should figure out the number after testing different values.
> Ok, I suppose I'll just do what Rock documentation says – will test
> different values and figure out what works for us.
Sounds like a good plan! You may even find a good/stable correlation
between Squid swap I/O ops and low-level disk I/O ops. Please consider
reporting your findings for others to reuse.
> On Thu, Jan 18, 2018 at 2:54 PM, Alex Rousskov wrote:
> On 01/18/2018 03:16 PM, Ivan Larionov wrote:
> > cache_dir max-swap-rate documentation says that swap in requests
> > contribute to measured swap rate. However in our squid 4 load test we
> > see that read_ops + write_ops significantly exceeds the max-swap-rate we
> > set and squid doesn't limit it.
> In this context, a single Squid "op" is a read or write request from
> worker to the disker process. These requests are up to one I/O page in
> size. A single I/O page is 32*1024 bytes. See Ipc::Mem::PageSize().
> * A single read request usually ends up being a single pread(2) system
> call that reads at most one I/O page worth of data from disk. See
> * A single write request usually ends up being a single pwrite(2) system
> call that writes at most one I/O page worth of data to disk. However, if
> that single pwrite() does not write everything a worker has requested to
> write, then Squid will make more pwrite() calls, up to 10 calls total.
> See diskerWrite().
> Within a single cache miss transaction, the rock code should accumulate
> small swapout requests from Store into page-size write requests to
> disker, but I do not remember how complete those optimizations are: It
> is possible that smaller-than-page writes get through to diskers,
> increasing the number of write requests. Same for reading cache hits.
> What is the "op" in read_ops and write_ops you have measured?
> Since Squid does not (and does not want to) have access to low-level
> disk stats and since Squid cannot assume exlusive disk ownership, the
> rate-limiting feature for rock cache_dirs does not know how many
> low-level disk operations the disk is doing and how those operations
> correspond to what Squid is asking the disk to do.
> > I tried to set it to 200 to confirm that it actually works and saw
> > it does. Squid started warning about exceeding max-swap-rate. But
> > like real limit is higher than the value we set in configuration.
> > Hardware:
> > AWS GP2 EBS (SSD) 600GB, 1500 iops baseline performance, 3000 iops
> > burstable.
> > Config:
> > cache_dir rock /mnt/services/squid/cache 435200 swap-timeout=500
> > max-swap-rate=1200 slot-size=16384
> > IOPS squid pushes under our load test:
> > read ~800 ops/sec
> > write ~1100 ops/sec
> > In summary it gives us ~1900 ops/sec which exceeds AWS limit of 1500
> > ops/sec and after spending too much "burst balance" we started getting
> > throttled from AWS side.
> > Could you please comment on this behavior? What the limit should
> we set
> > to stay under 1500 ops/sec for swap in + swap out operations?
> > Thanks.
> > Squid version:
> > Squid Cache: Version 4.0.22
> > Service Name: squid
> > configure options: '--program-prefix=' '--prefix=/usr'
> > '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> > '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> > '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> > '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> > '--localstatedir=/var' '--mandir=/usr/share/man'
> > '--infodir=/usr/share/info' '--enable-epoll'
> > '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
> > '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> > '--with-large-files' '--with-maxfd=16384' '--enable-htcp'
> > --
> > With best regards, Ivan Larionov.
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > http://lists.squid-cache.org/listinfo/squid-users
> With best regards, Ivan Larionov.
More information about the squid-users