[squid-users] Rock datastore, CFLAGS and a crash that (may be) known

[Jester Purtteman]

Dear Eliezer, Amos and Marcus,

Thank you, and sorry for the late reply, day jobs are a menace to productivity :)

So, in order of responses:  Eliezer:
> Before digging into the details of the issue, can you supply the OS details?
> What OS are you using? What distribution?
> 32 or 64 bit?
> can you also add the output of "squid -v" for both 3.5.14 and 3.5.13 ?

I am running Ubuntu 14.04.2 updated to the latest apt-get binaries, 64-bit version, 4 processors, 24-gb of "ram" allocated under the VM.  This is all on a vmware ESXi 6.0 host, so I recognize that compiler flags are probably a bit like throwing water balloons at Jaws from a performance stand point.  The counter point is, with performance as bad as a VM, you need all the help you can get.  As much as anything, it was a curiousity. 

Squid -v for a working configuration is as follows:

Squid Cache: Version 3.5.14
Service Name: squid
configure options:  '--with-pthreads' '--prefix=/usr' '--localstatedir=/var' '--libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-default-user=proxy' '--with-logdir=/var/log' '--with-pidfile=/var/run/squid.pid' '--enable-linux-netfilter' '--enable-cache-digests' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-async-io=30' '--enable-http-violations' '--enable-zph-qos' '--with-netfilter-conntrack' '--with-filedescriptors=65536' '--with-large-files' --enable-ltdl-convenience

I cut and pasted the configuration string I'd used with 3.5.13, added "--with-pthreads", and had no problems.  Here is the working 3.5.13 -v output:

Squid Cache: Version 3.5.13
Service Name: squid
configure options:  '--prefix=/usr' '--localstatedir=/var' '--libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-default-user=proxy' '--with-logdir=/var/log' '--with-pidfile=/var/run/squid.pid' '--enable-linux-netfilter' '--enable-cache-digests' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-async-io=30' '--enable-http-violations' '--enable-zph-qos' '--with-netfilter-conntrack' '--with-filedescriptors=65536' '--with-large-files' --enable-ltdl-convenience

Amos:

> > cache_dir rock /var/spool/squid/rock/1 64000 swap-timeout=600
> > max-swap-rate=600 min-size=0 max-size=128KB
> >
> > cache_dir rock /var/spool/squid/rock/2 102400 swap-timeout=600
> > max-swap-rate=600 min-size=128KB max-size=256KB
> >
> > cache_dir aufs /var/spool/squid/aufs/1 200000 16 128 min-size=256KB
> > max-size=4096KB
> >
> > cache_dir aufs /var/spool/squid/aufs/2 1500000 16 128 min-size=4096KB
> > max-size=8196000KB
> 
> NP: don't forget to isolate the AUFS cache_dir within each worker.
> Either with the squid.conf if-else-endif syntax, or ${process_id} macros.

Right now I'm only running one worker, although I was hoping to gain a little speed by having several dickers.  Is that going to require the additional squid-conf syntax?  I had understood that to only be relevant to multiple workers, and as I understand it, workers don't gracefully share resources yet.  One of my future efforts will be to attempt to divvy up workers by traffic (if I can figure out a way to manage it with ACLs or something) so that windows updates and a few other (stasticically big file) sites are given to one worker, and the rest go to the other worker.  I'm hoping to be able to cache big files a little more efficiently, currently they can lead to little seizures while squid opens a 2-gb file into memory, seizures that update software doesn't notice but end users do.  I'm thinking isolation may be the answer there.  Just thinking out loud, I'll grep the docs tonight and think about that more, pointers welcome.

> Three potential problems I can see here:
> 
>  1) are those flags the current values or your old findings? things might have
> changed in some nasty subtle way.
> 
>  2) that Squid is now tuned to that exact VM emulation running on that exact
> underlying host-OS hardware. Any variation of the two and you risk
> SEGFAULT. see (1)
> 
I haven't repeated it in a while, but I went through a pretty lengthy process to figure out what those were, and the only update has been new kernels since then.  That said, it’s a fair point, how tightly wound do I want squid to my exact hardware config.

>  3) I dont think this expansion method is safe. Better to go something like
> ./configure BOO="..." CFLAGS="${BOO}" CXXFLAGS="${BOO}"
> 
Hey, that’s a good tip, I'll do that.

>  4) current Squid versions add -march=native automatically.
> Unless you also add the --disable-arch-native option, it might be clashing with
> your choice of CPU flags (at least the -march=core2). Also
> (1) and (2) are relevant when building with -march=native.
>
...
> I've learnt just enough in this area myself to know that the best thing to do is
> find someone in the embedded hardware area (or an expert in that specific
> hardware) and ask their advice on flags.

Yes, or dedicate some time to experimentation and watching compile traffic go, and testing testing testing.  You can save yourself a 5-minute conversation with months of diligent effort :).

> > (3)    Does the strategy above make sense?  My thinking is to segregate the
> > small cache items into a rock datastore, and the big items into an
> > aufs datastore.
> 
> Yes that is the recommended practice.
In the interest of time, I think I may work more on tuning the datastores and come back to compiler flags later.

> > I would
> > like to get multiple disker processes running, I think it would
> > probably help in my environment, but it's not supremely critical.
> > Anyway, there is a note at the end of the bug saying that this wasn't
> > seen for a while, and I thought I'd say "I've seen it! Maybe!"  let me
> > know if I am creating this bug through a creative mistake, or if you have
> other ideas here.  Thanks!
> 
> Your original build options should have provided you with an SMP version of
> Squid for both versions of 3.5. All you need for the basic SMP operation is
> UDS sockets and /dev/shm shared memory.
> 
> The rest is squid.conf details.

It is my experience that squid.conf knows all.  I'll endeavor to read what is actually written in the docs, one more time :).  My main point earlier had been "oh, cflags wasn't working, interesting, I wonder what happens when I use my overly calibrated, haphazardly prepared string on it?  Oh look, fireworks!"  but I guess that was a predictable result looking back.


Last, but certainly not least, Marcus:

> > ./configure CFLAGS="-march=core2 -mcx16 -msahf -mno-movbe -mno-aes
> -mno-pclmul -mno-popcnt -mno-sse4 -msse4.1" CXXFLAGS="${CFLAGS}" --
> with-pthreads --prefix=/usr   --localstatedir=/var
> > --libexecdir=/usr/lib/squid    --srcdir=.   --datadir=/usr/share/squid --
> sysconfdir=/etc/squid   --with-default-user=proxy --with-logdir=/var/log   --
> with-pidfile=/var/run/squid.pid
> > --enable-linux-netfilter  --enable-cache-digests --enable-
> storeio=ufs,aufs,diskd,rock  --enable-async-io=30 --enable-http-violations --
> enable-zph-qos --with-netfilter-conntrack
> > --with-filedescriptors=65536 --with-large-files
> 
> I do not recommend setting options this way, especially -mcx16 -msahf and
> others may produce code that does not run correctly on all systems.
> 
> I recommend using
>     CFLAGS='-g -O2 -Wall -march=native' CXXFLAGS='-g -O2 -Wall -
> march=native'
> to get the best performance since it turns on all possible processor features
> like mcx16 if and only if the used system supports it.

It sounds like -march-native is probably unnecessary too, but I will give '-g -O2 -Wall' a try and let you know if I come up with a less explosive result.  

Sorry for the massive three way email, hope it was still readable.  I didn't want to leave anyone out of the reply, and I do appreciate everyone's input.  Before getting back to me (unless I'm proposing doing something insanely stupid) I'll try Marcus's flags, and as time permits I'll add in the other ones until something gets sparky.  It sounds like this is probably not terribly useful, but I hate leaving something unanswered that isn't going to take me long to sort out.