[squid-users] Rock datastore, CFLAGS and a crash that (may be) known

Wed Feb 17 14:00:24 UTC 2016

--march=core2 it was, all others worked fine.  Not that they provided any improvement to the actual performance, but they didn't break anything either.  Now, I'll get to work tuning my cache dirs..  Thanks!

> -----Original Message-----
> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
> Behalf Of Jester Purtteman
> Sent: Wednesday, February 17, 2016 5:37 AM
> To: 'Marcus Kool' <marcus.kool at urlfilterdb.com>; 'Eliezer Croitoru'
> <eliezer at ngtech.co.il>; squid-users at lists.squid-cache.org; 'Amos Jeffries'
> <squid3 at treenet.co.nz>
> Subject: Re: [squid-users] Rock datastore, CFLAGS and a crash that (may be)
> known
> 
> Dear Eliezer, Amos and Marcus,
> 
> Thank you, and sorry for the late reply, day jobs are a menace to productivity
> :)
> 
> So, in order of responses:  Eliezer:
> > Before digging into the details of the issue, can you supply the OS details?
> > What OS are you using? What distribution?
> > 32 or 64 bit?
> > can you also add the output of "squid -v" for both 3.5.14 and 3.5.13 ?
> 
> I am running Ubuntu 14.04.2 updated to the latest apt-get binaries, 64-bit
> version, 4 processors, 24-gb of "ram" allocated under the VM.  This is all on a
> vmware ESXi 6.0 host, so I recognize that compiler flags are probably a bit like
> throwing water balloons at Jaws from a performance stand point.  The
> counter point is, with performance as bad as a VM, you need all the help you
> can get.  As much as anything, it was a curiousity.
> 
> Squid -v for a working configuration is as follows:
> 
> Squid Cache: Version 3.5.14
> Service Name: squid
> configure options:  '--with-pthreads' '--prefix=/usr' '--localstatedir=/var' '--
> libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--
> sysconfdir=/etc/squid' '--with-default-user=proxy' '--with-logdir=/var/log' '--
> with-pidfile=/var/run/squid.pid' '--enable-linux-netfilter' '--enable-cache-
> digests' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-async-io=30' '--
> enable-http-violations' '--enable-zph-qos' '--with-netfilter-conntrack' '--with-
> filedescriptors=65536' '--with-large-files' --enable-ltdl-convenience
> 
> I cut and pasted the configuration string I'd used with 3.5.13, added "--with-
> pthreads", and had no problems.  Here is the working 3.5.13 -v output:
> 
> Squid Cache: Version 3.5.13
> Service Name: squid
> configure options:  '--prefix=/usr' '--localstatedir=/var' '--
> libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--
> sysconfdir=/etc/squid' '--with-default-user=proxy' '--with-logdir=/var/log' '--
> with-pidfile=/var/run/squid.pid' '--enable-linux-netfilter' '--enable-cache-
> digests' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-async-io=30' '--
> enable-http-violations' '--enable-zph-qos' '--with-netfilter-conntrack' '--with-
> filedescriptors=65536' '--with-large-files' --enable-ltdl-convenience
> 
> Amos:
> 
> > > cache_dir rock /var/spool/squid/rock/1 64000 swap-timeout=600
> > > max-swap-rate=600 min-size=0 max-size=128KB
> > >
> > > cache_dir rock /var/spool/squid/rock/2 102400 swap-timeout=600
> > > max-swap-rate=600 min-size=128KB max-size=256KB
> > >
> > > cache_dir aufs /var/spool/squid/aufs/1 200000 16 128 min-size=256KB
> > > max-size=4096KB
> > >
> > > cache_dir aufs /var/spool/squid/aufs/2 1500000 16 128
> > > min-size=4096KB max-size=8196000KB
> >
> > NP: don't forget to isolate the AUFS cache_dir within each worker.
> > Either with the squid.conf if-else-endif syntax, or ${process_id} macros.
> 
> Right now I'm only running one worker, although I was hoping to gain a little
> speed by having several dickers.  Is that going to require the additional squid-
> conf syntax?  I had understood that to only be relevant to multiple workers,
> and as I understand it, workers don't gracefully share resources yet.  One of
> my future efforts will be to attempt to divvy up workers by traffic (if I can
> figure out a way to manage it with ACLs or something) so that windows
> updates and a few other (stasticically big file) sites are given to one worker,
> and the rest go to the other worker.  I'm hoping to be able to cache big files a
> little more efficiently, currently they can lead to little seizures while squid
> opens a 2-gb file into memory, seizures that update software doesn't notice
> but end users do.  I'm thinking isolation may be the answer there.  Just
> thinking out loud, I'll grep the docs tonight and think about that more,
> pointers welcome.
> 
> > Three potential problems I can see here:
> >
> >  1) are those flags the current values or your old findings? things
> > might have changed in some nasty subtle way.
> >
> >  2) that Squid is now tuned to that exact VM emulation running on that
> > exact underlying host-OS hardware. Any variation of the two and you
> > risk SEGFAULT. see (1)
> >
> I haven't repeated it in a while, but I went through a pretty lengthy process
> to figure out what those were, and the only update has been new kernels
> since then.  That said, it’s a fair point, how tightly wound do I want squid to
> my exact hardware config.
> 
> >  3) I dont think this expansion method is safe. Better to go something
> > like ./configure BOO="..." CFLAGS="${BOO}" CXXFLAGS="${BOO}"
> >
> Hey, that’s a good tip, I'll do that.
> 
> >  4) current Squid versions add -march=native automatically.
> > Unless you also add the --disable-arch-native option, it might be
> > clashing with your choice of CPU flags (at least the -march=core2).
> > Also
> > (1) and (2) are relevant when building with -march=native.
> >
> ...
> > I've learnt just enough in this area myself to know that the best
> > thing to do is find someone in the embedded hardware area (or an
> > expert in that specific
> > hardware) and ask their advice on flags.
> 
> Yes, or dedicate some time to experimentation and watching compile traffic
> go, and testing testing testing.  You can save yourself a 5-minute
> conversation with months of diligent effort :).
> 
> > > (3)    Does the strategy above make sense?  My thinking is to segregate
> the
> > > small cache items into a rock datastore, and the big items into an
> > > aufs datastore.
> >
> > Yes that is the recommended practice.
> In the interest of time, I think I may work more on tuning the datastores and
> come back to compiler flags later.
> 
> > > I would
> > > like to get multiple disker processes running, I think it would
> > > probably help in my environment, but it's not supremely critical.
> > > Anyway, there is a note at the end of the bug saying that this
> > > wasn't seen for a while, and I thought I'd say "I've seen it!
> > > Maybe!"  let me know if I am creating this bug through a creative
> > > mistake, or if you have
> > other ideas here.  Thanks!
> >
> > Your original build options should have provided you with an SMP
> > version of Squid for both versions of 3.5. All you need for the basic
> > SMP operation is UDS sockets and /dev/shm shared memory.
> >
> > The rest is squid.conf details.
> 
> It is my experience that squid.conf knows all.  I'll endeavor to read what is
> actually written in the docs, one more time :).  My main point earlier had
> been "oh, cflags wasn't working, interesting, I wonder what happens when I
> use my overly calibrated, haphazardly prepared string on it?  Oh look,
> fireworks!"  but I guess that was a predictable result looking back.
> 
> 
> Last, but certainly not least, Marcus:
> 
> > > ./configure CFLAGS="-march=core2 -mcx16 -msahf -mno-movbe -mno-
> aes
> > -mno-pclmul -mno-popcnt -mno-sse4 -msse4.1" CXXFLAGS="${CFLAGS}" --
> > with-pthreads --prefix=/usr   --localstatedir=/var
> > > --libexecdir=/usr/lib/squid    --srcdir=.   --datadir=/usr/share/squid --
> > sysconfdir=/etc/squid   --with-default-user=proxy --with-logdir=/var/log   --
> > with-pidfile=/var/run/squid.pid
> > > --enable-linux-netfilter  --enable-cache-digests --enable-
> > storeio=ufs,aufs,diskd,rock  --enable-async-io=30
> > --enable-http-violations -- enable-zph-qos --with-netfilter-conntrack
> > > --with-filedescriptors=65536 --with-large-files
> >
> > I do not recommend setting options this way, especially -mcx16 -msahf
> > and others may produce code that does not run correctly on all systems.
> >
> > I recommend using
> >     CFLAGS='-g -O2 -Wall -march=native' CXXFLAGS='-g -O2 -Wall -
> > march=native'
> > to get the best performance since it turns on all possible processor
> > features like mcx16 if and only if the used system supports it.
> 
> It sounds like -march-native is probably unnecessary too, but I will give '-g -
> O2 -Wall' a try and let you know if I come up with a less explosive result.
> 
> Sorry for the massive three way email, hope it was still readable.  I didn't
> want to leave anyone out of the reply, and I do appreciate everyone's input.
> Before getting back to me (unless I'm proposing doing something insanely
> stupid) I'll try Marcus's flags, and as time permits I'll add in the other ones
> until something gets sparky.  It sounds like this is probably not terribly useful,
> but I hate leaving something unanswered that isn't going to take me long to
> sort out.
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users