[squid-users] Squid 3.5.21 - High CPU (100%)

Mon Sep 26 14:22:06 UTC 2016

Is there anything that you guys can suggest I do around the cache?
Should I try a different store type? A different filesystem type perhaps?

If your store has a configuration knob that effectively limits disk
writing rate, then use it to limit that rate to avoid overflowing the queue.

You can also consider using rock store that has an explicit write rate
limiting option (but comes with other problems that may or may not
affect your setup).

Adding more physical disk spindles helps, of course.

Thanks Alex. My next approach is to add more spindles. In fact I am currently testing with 4 cache dirs. All aufs currently. After adding these cache dirs, Squid started crashing every 20mins or so..

--cut-- (Cache.log)
FATAL: Received Segment Violation...dying.
2016/09/26 15:54:54 kid1| Closing HTTP port 0.0.0.0:8080
2016/09/26 15:54:54 kid1| storeDirWriteCleanLogs: Starting...
2016/09/26 15:54:54 kid1|     65536 entries written so far.
2016/09/26 15:54:54 kid1|   Finished.  Wrote 98159 entries.
2016/09/26 15:54:54 kid1|   Took 0.05 seconds (2044425.47 entries/sec).
CPU Usage: 1230.157 seconds = 166.490 user + 1063.666 sys
Maximum Resident Size: 3535840 KB
Page faults with physical i/o: 0
2016/09/26 15:54:57 kid1| Set Current Directory to /var/cache/squid/
2016/09/26 15:54:57 kid1| Starting Squid Cache version 3.5.21 for x86_64-pc-linux-gnu...
2016/09/26 15:54:57 kid1| Service Name: squid
2016/09/26 15:54:57 kid1| Process ID 16054
2016/09/26 15:54:57 kid1| Process Roles: worker
2016/09/26 15:54:57 kid1| With 65536 file descriptors available
2016/09/26 15:54:57 kid1| Initializing IP Cache...
2016/09/26 15:54:57 kid1| DNS Socket created at 0.0.0.0, FD 7
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding nameserver xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding nameserver xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding nameserver xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Logfile: opening log daemon:/var/log/squid//access.log
2016/09/26 15:54:57 kid1| Logfile Daemon: opening log /var/log/squid//access.log
2016/09/26 15:54:57 kid1| Store logging disabled
2016/09/26 15:54:57 kid1| Swap maxSize 41943040 + 4194304 KB, estimated 3549026 objects
2016/09/26 15:54:57 kid1| Target number of buckets: 177451
2016/09/26 15:54:57 kid1| Using 262144 Store buckets
2016/09/26 15:54:57 kid1| Max Mem  size: 4194304 KB
2016/09/26 15:54:57 kid1| Max Swap size: 41943040 KB
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c1/ (clean log)
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c2/ (clean log)
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c3/ (clean log)
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c4/ (clean log)
2016/09/26 15:54:57 kid1| Using Least Load store dir selection
2016/09/26 15:54:57 kid1| Set Current Directory to /var/cache/squid/
2016/09/26 15:54:57 kid1| Finished loading MIME types and icons.
2016/09/26 15:54:57 kid1| HTCP Disabled.
2016/09/26 15:54:57 kid1| Sending SNMP messages from 0.0.0.0:3401
2016/09/26 15:54:57 kid1| Configuring Parent xxxxxx8/80/0
2016/09/26 15:54:57 kid1| Configuring Parent xxxxxx/80/0
2016/09/26 15:54:57 kid1| Configuring Sibling xxxxxx/8080/3131
2016/09/26 15:54:57 kid1| Squid plugin modules loaded: 0
2016/09/26 15:54:57 kid1| Adaptation support is off.
2016/09/26 15:54:57 kid1| Accepting HTTP Socket connections at local=0.0.0.0:8080 remote=[::] FD 18 flags=9
2016/09/26 15:54:57 kid1| Accepting SNMP messages on 0.0.0.0:3401
2016/09/26 15:54:57 kid1| Store rebuilding is 33.00% complete
2016/09/26 15:54:57 kid1| Done reading /var/cache/squid-c3/ swaplog (8660 entries)
2016/09/26 15:54:57 kid1| Done reading /var/cache/squid-c1/ swaplog (12119 entries)
2016/09/26 15:54:58 kid1| Done reading /var/cache/squid-c2/ swaplog (30684 entries)
2016/09/26 15:54:58 kid1| Done reading /var/cache/squid-c4/ swaplog (46696 entries)
2016/09/26 15:54:58 kid1| Finished rebuilding storage from disk.
2016/09/26 15:54:58 kid1|     98159 Entries scanned
2016/09/26 15:54:58 kid1|         0 Invalid entries.
2016/09/26 15:54:58 kid1|         0 With invalid flags.
2016/09/26 15:54:58 kid1|     98159 Objects loaded.
2016/09/26 15:54:58 kid1|         0 Objects expired.
2016/09/26 15:54:58 kid1|         0 Objects cancelled.
2016/09/26 15:54:58 kid1|         0 Duplicate URLs purged.
2016/09/26 15:54:58 kid1|         0 Swapfile clashes avoided.
2016/09/26 15:54:58 kid1|   Took 0.61 seconds (162197.49 objects/sec).
2016/09/26 15:54:58 kid1| Beginning Validation Procedure
2016/09/26 15:54:58 kid1|   Completed Validation Procedure
2016/09/26 15:54:58 kid1|   Validated 98159 Entries
2016/09/26 15:54:58 kid1|   store_swap_size = 11495920.00 KB
2016/09/26 15:54:58 kid1| storeLateRelease: released 0 objects

--cut--

 I tried rock but was stuck with this error for a while and decided to go back to aufs until I can do some more reading up.

FATAL: Rock cache_dir at /var/cache/squid-c3//rock failed to open db file: (2) No such file or directory

I am currently testing again after recompiling squid..  The rest of my proxy cluster has the "cache deny all" configuration directive.. So nothing is cached.

Disclaimer:
http://www.shopriteholdings.co.za/Pages/ShopriteE-mailDisclaimer.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160926/2bdb086b/attachment-0001.html>