[squid-users] Rock datastore, CFLAGS and a crash that (may be) known

[Amos Jeffries]

On 17/02/2016 3:32 a.m., Jester Purtteman wrote:
> Greetings Squid users,
> 
>  
> 
> With 3.5.14 out and activating CFLAGS, I am getting into trouble.  Funny
> too, I spent a lot of time wondering why it wasn't adding CFLAGS in earlier
> builds.  In any event, I have a 3.5.13 instance configured as follows:
> 
>  
> 
> ./configure --prefix=/usr   --localstatedir=/var
> --libexecdir=/usr/lib/squid    --srcdir=.   --datadir=/usr/share/squid
> --sysconfdir=/etc/squid   --with-default-user=proxy   --with-logdir=/var/log
> --with-pidfile=/var/run/squid.pid --enable-linux-netfilter
> --enable-cache-digests --enable-storeio=ufs,aufs,diskd,rock
> --enable-async-io=30 --enable-http-violations --enable-zph-qos
> --with-netfilter-conntrack --with-filedescriptors=65536 --with-large-files
> 
>  
> 
> It has a quartet of cache-dirs (I'm still testing and monkeying) as follows:
> 
> cache_dir rock /var/spool/squid/rock/1 64000 swap-timeout=600
> max-swap-rate=600 min-size=0 max-size=128KB
> 
> cache_dir rock /var/spool/squid/rock/2 102400 swap-timeout=600
> max-swap-rate=600 min-size=128KB max-size=256KB
> 
> cache_dir aufs /var/spool/squid/aufs/1 200000 16 128 min-size=256KB
> max-size=4096KB
> 
> cache_dir aufs /var/spool/squid/aufs/2 1500000 16 128 min-size=4096KB
> max-size=8196000KB

NP: don't forget to isolate the AUFS cache_dir within each worker.
Either with the squid.conf if-else-endif syntax, or ${process_id} macros.


> 
> Permissions are all proxy.proxy for the cache dirs and everything is happily
> running.  When I read that the CFLAGS bug was solved, I thought "hey, didn't
> I do some terrible thing to determine what cflags are correct on a vmware
> virtual instance?" and dug up the cflags that I came up with.  I then
> compiled 3.5.14 as follows:
> 
>  
> 
> ./configure CFLAGS="-march=core2 -mcx16 -msahf -mno-movbe -mno-aes
> -mno-pclmul -mno-popcnt -mno-sse4 -msse4.1" CXXFLAGS="${CFLAGS}"

Three potential problems I can see here:

 1) are those flags the current values or your old findings? things
might have changed in some nasty subtle way.

 2) that Squid is now tuned to that exact VM emulation running on that
exact underlying host-OS hardware. Any variation of the two and you risk
SEGFAULT. see (1)

 3) I dont think this expansion method is safe. Better to go something
like ./configure BOO="..." CFLAGS="${BOO}" CXXFLAGS="${BOO}"

 4) current Squid versions add -march=native automatically.
Unless you also add the --disable-arch-native option, it might be
clashing with your choice of CPU flags (at least the -march=core2). Also
(1) and (2) are relevant when building with -march=native.


> --with-pthreads --prefix=/usr   --localstatedir=/var
> --libexecdir=/usr/lib/squid    --srcdir=.   --datadir=/usr/share/squid
> --sysconfdir=/etc/squid   --with-default-user=proxy   --with-logdir=/var/log
> --with-pidfile=/var/run/squid.pid --enable-linux-netfilter
> --enable-cache-digests --enable-storeio=ufs,aufs,diskd,rock
> --enable-async-io=30 --enable-http-violations --enable-zph-qos
> --with-netfilter-conntrack --with-filedescriptors=65536 --with-large-files
> 
>  
> 
> This leads to the following in the cache log, and a crash.
> 
>  
> 
> <<<SNIP
> 
> FATAL: Ipc::Mem::Segment::open failed to
> shm_open(/squid-var.spool.squid.rock.1_spaces.shm): (2) No such file or
> directory
> 

AFAIK the "No such file" is related to whether the /dev/shm system
device is operating properly (off by default on some Linux), and whether
the name Squid is requesting is too long (MacOS is nasty like that).
It should not be related to the CPU flags being set. And the code Squid
uses for that is C++ anyway, so the releases flag change should not be
related.


> 
> This looks similar to a bug
> http://bugs.squid-cache.org/show_bug.cgi?id=3880#c1 that was already
> reported, but I don't know enough to say with certainty. 

Its not. That group of issues all specifically log as "timeout" error of
one sort or another.

> It does look like
> these compile options are allowing squid to launch with multiple processes
> and do other things that I think I might want, but I can't tell for sure.
> So, it does lead me to a few questions:
> 
>  
> 
> (1)    Do these flags make sense?  I only half know what half of them do,
> but they appear to basically just be supported flags on a ESXi virtual
> machine given my hardware.  I have googled, just not a lot of light shed on
> this instance, thoughts and insights are appreciated.
> 

I've learnt just enough in this area myself to know that the best thing
to do is find someone in the embedded hardware area (or an expert in
that specific hardware) and ask their advice on flags.

In principle it looks like you are headed in the right direction.
Setting the host machine CPU type, adding and removing the bits the VM
layer does (or not) supports providing access to.

But I'm not educated enough on it myself to give a good answer there and
I dont think anyone else could without detaisl of the specific hardware
and VM systems you are using. Which goes back to finding yourself a
friendly expert :-)



> (2)    Are my rock stores lagging out, and how would you recommend tuning
> them if so?

No. The workers are not even able to open UDS channels to talk to each
other.

> 
> (3)    Does the strategy above make sense?  My thinking is to segregate the
> small cache items into a rock datastore, and the big items into an aufs
> datastore.

Yes that is the recommended practice.

> 
> (4)    Do you have any pointers on calculating the size of rocks and aufs
> stores based on disk performance etc?  I'm guessing that there is sort of a
> logical size to make a specific rock and aufs based on how big of items you
> store in it and so on.  Is there some way I can apply some math and find
> bottlenecks?

IFyou have a trace of your hetworks HTTP traffic over any good length of
time. Graph it and see where the peaks are. You should see one around 0,
one in the low KB range (16-32'ish) and one low MB range (Youtubes most
popular video codec size).

Tune the rock to store a reasonable portion of the lower hump(s). And
AUFS for the larger ones. You may even want to have a dedicated rock for
the 0-1KB ones which make it to disk, although those are usually memory
objects.


> Finally, 3.5.14 does run fine when compiled with the first set (even with
> --with-pthreads added) so I think this is probably cflags related. 

cflags is only used on C files, which these days is limited to some one
or two third-party portability wrappers we still use. So not likely.
If that sentence above was right and not a typo of .13 , then you just
proved it was fine.

I suspect the problem is your custom flags colliding with -march=native.


> I would
> like to get multiple disker processes running, I think it would probably
> help in my environment, but it's not supremely critical.  Anyway, there is a
> note at the end of the bug saying that this wasn't seen for a while, and I
> thought I'd say "I've seen it! Maybe!"  let me know if I am creating this
> bug through a creative mistake, or if you have other ideas here.  Thanks!

Your original build options should have provided you with an SMP version
of Squid for both versions of 3.5. All you need for the basic SMP
operation is UDS sockets and /dev/shm shared memory.

The rest is squid.conf details.

Amos