[squid-users] TCP out of memory

Thu Dec 21 08:25:07 UTC 2017

BTW, I set icap_service_failure_limit -1 because If I don't the HTTP clients get the ERR_ICAP_FAILURE page for *a very long time* if I restart the c-icap service.
I have no idea how much they would have to wait because I can't afford users seeing this page for more than a minute.
The following test was done in 2 minutes with the default icap_service_failure_limit:

# /etc/init.d/squid reload

# /etc/init.d/c-icap restart

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 21 Dec 2017 08:07:07 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 21 Dec 2017 08:07:07 GMT
Last-Modified: Thu, 21 Dec 2017 08:07:07 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20171101-re69e56c
Build Info:
Service Name: squid
Start Time:     Mon, 18 Dec 2017 08:29:33 GMT
Current Time:   Thu, 21 Dec 2017 08:07:07 GMT
Connection information for squid:
Number of clients accessing cache:      569
Number of HTTP requests received:       3381083
Number of ICP messages received:        0
Number of ICP messages sent:    0
Number of queued ICP replies:   0
Number of HTCP messages received:       0
Number of HTCP messages sent:   0
Request failure ratio:   0.00
Average HTTP requests per minute since start:   786.7
Average ICP messages per minute since start:    0.0
Select loop called: 93431818 times, 2.760 ms avg
Cache information for squid:
Hits as % of all requests:      5min: 0.9%, 60min: 2.0%
Hits as % of bytes sent:        5min: 5.2%, 60min: 9.5%
Memory hits as % of hit requests:       5min: 88.4%, 60min: 60.2%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size:      29156 KB
Storage Swap capacity:  89.0% used, 11.0% free
Storage Mem size:       30648 KB
Storage Mem capacity:   93.5% used,  6.5% free
Mean Object Size:       18.03 KB
Requests given to unlinkd:      49975
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   0.05633  0.04047
Cache Misses:          0.10281  0.11465
Cache Hits:            0.00000  0.00000
Near Hits:             0.01745  0.01955
Not-Modified Replies:  0.00000  0.00000
DNS Lookups:           0.04048  0.03868
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time:        257853.454 seconds
CPU Time:       6005.270 seconds
CPU Usage:      2.33%
CPU Usage, 5 minute avg:        6.76%
CPU Usage, 60 minute avg:       2.99%
Maximum Resident Size: 5549424 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       1004217 KB
memPoolAlloc calls: 980681181
memPoolFree calls:  999432183
File descriptor usage for squid:
Maximum number of file descriptors:   65536
Largest file desc currently in use:   4399
Number of file desc currently in use: 4052
Files queued for open:                   0
Available number of file descriptors: 61484
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
2074 StoreEntries
1911 StoreEntries with MemObjects
1687 Hot Object Cache Items
1617 on-disk objects

Two minutes later and clients are still seeing ERR_ICAP_FAILURE, so I'm setting back to icap_service_failure_limit -1.

# /etc/init.d/squid reload

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Thu, 21 Dec 2017 08:09:26 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 21 Dec 2017 08:09:26 GMT
Last-Modified: Thu, 21 Dec 2017 08:09:26 GMT
X-Cache: MISS from inf-fw2
X-Cache-Lookup: MISS from inf-fw2:3128
Connection: close

Squid Object Cache: Version 3.5.27-20171101-re69e56c
Build Info:
Service Name: squid
Start Time:     Mon, 18 Dec 2017 08:29:33 GMT
Current Time:   Thu, 21 Dec 2017 08:09:26 GMT
Connection information for squid:
Number of clients accessing cache:      569
Number of HTTP requests received:       3382478
Number of ICP messages received:        0
Number of ICP messages sent:    0
Number of queued ICP replies:   0
Number of HTCP messages received:       0
Number of HTCP messages sent:   0
Request failure ratio:   0.00
Average HTTP requests per minute since start:   786.6
Average ICP messages per minute since start:    0.0
Select loop called: 93457469 times, 2.761 ms avg
Cache information for squid:
Hits as % of all requests:      5min: 1.2%, 60min: 2.0%
Hits as % of bytes sent:        5min: 10.9%, 60min: 9.7%
Memory hits as % of hit requests:       5min: 88.3%, 60min: 60.1%
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.1%
Storage Swap size:      29156 KB
Storage Swap capacity:  89.0% used, 11.0% free
Storage Mem size:       30648 KB
Storage Mem capacity:   93.5% used,  6.5% free
Mean Object Size:       18.03 KB
Requests given to unlinkd:      49975
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   0.05046  0.04047
Cache Misses:          0.11465  0.11465
Cache Hits:            0.00000  0.00000
Near Hits:             0.00000  0.01955
Not-Modified Replies:  0.00000  0.00000
DNS Lookups:           0.04237  0.03868
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time:        257992.456 seconds
CPU Time:       6009.530 seconds
CPU Usage:      2.33%
CPU Usage, 5 minute avg:        4.90%
CPU Usage, 60 minute avg:       2.98%
Maximum Resident Size: 5549728 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted:       997578 KB
memPoolAlloc calls: 980907766
memPoolFree calls:  999669215
File descriptor usage for squid:
Maximum number of file descriptors:   65536
Largest file desc currently in use:   4399
Number of file desc currently in use: 3676
Files queued for open:                   0
Available number of file descriptors: 61860
Reserved number of file descriptors:   100
Store Disk files open:                   0
Internal Data Structures:
1895 StoreEntries
1732 StoreEntries with MemObjects
1687 Hot Object Cache Items
1617 on-disk objects

Clients are now browsing, and squid/c-icap are apparently communicating.

Vieri