[squid-users] Optimizing squid

[Alex Rousskov]

On 02/25/2016 09:58 AM, Heiler Bemerguy wrote:

> So, to start from the start, after seeing squid was totally stable and
> fast, running with NO cache_dirs, I tried to add only 2 rockstore
> cache_dirs to test.

> cache_dir rock /cache2/rock1 20000 min-size=0 max-size=4096 slot-size=2048
> cache_dir rock /cache2/rock2 30000 min-size=4097 max-size=16384 slot-size=4096


> (ps.: I know it would be nice to use one store PER partition/disk/lun
> whatever.. but I'm trying to lessen disk wasting by using small
> slot-sizes for small files.. am I wrong?)

There is nothing wrong with the desire to reduce waste. However, you are
combining multiple optimization goals into one mess that would be
difficult to untangle. I recommend that you focus on _one_ primary goal
and, once that goal is accomplished, move on to the next one. I see a
growing list of [potential] goals in your emails:

  1. Solve occasional 100% CPU utilization problem.
  2. Optimize Squid performance on beefy hardware.
  3. Reduce disk space waste.
  4. Solve "malformed cache entry" problems.

In general, if something does not work, report a bug to bugzilla or your
support team and, if possible, back off or simplify to avoid that bug
while you focus on your primary goal. Solving 100% CPU usage problems
while battling cache rebuild problems, optimizing cache space
allocation, learning to interpret Squid logs, and juggling contradictory
squid-users advice is a recipe for lots of headaches and little progress!


> /2016/02/25 13:42:09 kid3| Loading cache_dir #1 from /cache2/rock2/rock//
> /2016/02/25 13:42:09 kid2| Loading cache_dir #0 from /cache2/rock1/rock//
> /2016/02/25 13:42:09 kid3| Store rebuilding is 0.01% complete//
> /2016/02/25 13:42:09 kid2| Store rebuilding is 0.01% complete/
> 
> Rebuilding what? just creating the huge files I think...

The "huge files" were already created during squid -z.

As Amos have said, rock store is [re]building an in-memory cache index.
Other stores do that too, but rock store takes longer under most common
circumstances. Please note that rock does not know that you have created
a new rock db a few minutes ago.

Patches optimizing index build and/or log messages are welcomed.


> Then:
> /2016/02/25 13:42:19 kid1| WARNING: swapfile header inconsistent with
> available data

I do not know what causes these in your environment. Do you see them
when _not_ using any optional options on the cache_dir lines and not
sending any HTTP requests to Squid?

> 2016/02/25 13:42:40 kid1| ctx: enter level  0:
> 'http://static.bn-static.com/pg/0plcB0QjJpBbwN7rMMDjKKO5Z63Nhu3zfPw==.gif'

It looks like you Squid is receiving some traffic while rebuilding its
index. Bugs notwithstanding, that should work and is supported, but it
may be helpful to know whether you get any errors when there is no
traffic while cache index is being built.


> 2016/02/25 13:42:43 kid2|   10239992 Entries scanned
> 2016/02/25 13:42:43 kid2|        14 Invalid entries.///

> What entry?

Squid calls HTTP responses stored in its cache "cache entries" or "store
entries". IIRC, in rock case, the entries in this specific log message
are actually rock db slots, but the Squid "core" code doing this
reporting does not know that.


> why malformed? 

Squid either does not know or is configured not to say. You can enable
more verbose logging to see more details. Overall, this problem is best
handled as a bug report than a squid-users discussion IMO, but see above
for some first triage steps.

If you report a bug, please specify what Squid version you are using and
on what OS.


> Wasn't it just a empty store?! it just created it.......

This could be a Squid bug, but AFAICT, you also sent some traffic
through Squid so the store may not be empty because of that.


HTH,

Alex.