[squid-users] Optimizing squid

[Amos Jeffries]

On 26/02/2016 5:58 a.m., Heiler Bemerguy wrote:
> 
> Hi Alex, Eliezer, Yuri, Amos..
> 
> So, to start from the start, after seeing squid was totally stable and
> fast, running with NO cache_dirs, I tried to add only 2 rockstore
> cache_dirs to test.
> 
> conf:
> /cache_dir rock /cache2/rock1 20000 min-size=0 max-size=4096
> slot-size=2048//
> //cache_dir rock /cache2/rock2 30000 min-size=4097 max-size=16384
> slot-size=4096/
> (ps.: I know it would be nice to use one store PER partition/disk/lun
> whatever.. but I'm trying to lessen disk wasting by using small
> slot-sizes for small files.. am I wrong?)
> 
> Then squid -z:
> /2016/02/25 13:42:00 kid2| Creating Rock db: /cache2/rock1/rock//
> //2016/02/25 13:42:00 kid3| Creating Rock db: /cache2/rock2/rock/
> 
> Then running squid for the first time with these newly created rock
> stores....
> 
> /2016/02/25 13:42:09 kid3| Loading cache_dir #1 from /cache2/rock2/rock//
> //2016/02/25 13:42:09 kid2| Loading cache_dir #0 from /cache2/rock1/rock//
> //2016/02/25 13:42:09 kid3| Store rebuilding is 0.01% complete//
> //2016/02/25 13:42:09 kid2| Store rebuilding is 0.01% complete/
> 
> Rebuilding what? just creating the huge files I think...

The cache index for those rock DB.

Unlike UFS which stores a swap.state file, rock rebuilds its index on
each startup.

> 
> Then:
> /2016/02/25 13:42:19 kid1| WARNING: swapfile header inconsistent with
> available data
> 2016/02/25 13:42:21 kid2| WARNING: cache_dir[0]: Ignoring malformed
> cache entry meta data at 6943832064
<snip repeats>
> 2016/02/25 13:42:40 kid1| ctx: enter level  0:
> 'http://static.bn-static.com/pg/0plcB0QjJpBbwN7rMMDjKKO5Z63Nhu3zfPw==.gif'
> 2016/02/25 13:42:40 kid1| WARNING: swapfile header inconsistent with
> available data
> 2016/02/25 13:42:40 kid2| WARNING: cache_dir[0]: Ignoring malformed
> cache entry meta data at 19581075456
> 2016/02/25 13:42:41 kid2| WARNING: cache_dir[0]: Ignoring malformed
> cache entry meta data at 19757760512
> 2016/02/25 13:42:43 kid2| Finished rebuilding storage from disk.
> 2016/02/25 13:42:43 kid2|   10239992 Entries scanned
> 2016/02/25 13:42:43 kid2|        14 Invalid entries.///
> 
> What entry? why malformed? Wasn't it just a empty store?! it just
> created it.......
> 


Did you wait for the -z background processes to finish creating the 50GB
of disk allocation before starting the main Squid process ?

Are your workers trying to serve up traffic to or from the cache before
the rebuild has completed?


As you can see from the log timestamps on startup it will take ~30-60
sec for the rock caches of that size to be loaded in your system.


Amos