[squid-users] Crash: every 1-2 hour: kernel: Out of memory: Kill process (squid)

[noc at forceline.net]

Anybody can help?
Maby I need to change mailing list?

--
Sergey

> -----Original Message-----
> From: noc at forceline.net [mailto:noc at forceline.net]
> Sent: Wednesday, December 14, 2016 2:40 PM
> To: 'squid-users at lists.squid-cache.org'
> Subject: Crash: every 1-2 hour: kernel: Out of memory: Kill process
> (squid)
> 
> 
> Hello. I wrote earlier in wrong location:
> http://bugs.squid-cache.org/show_bug.cgi?id=4647
> 
> > Squid eats all RAM, then eats all swap in a hour and killed by
> kernel.
> >I was try to turn off cache, change squid version, change some
> configuration parameters by this guide http://wiki.squid-
> cache.org/SquidFaq/SquidMemory except malloc, but nothing helps.
> 
> I made some config changes in accordance with the advice of Amos
> Jeffries (via on). But it does not help.
> This trouble somehow linked with https.
> If wccp redirects only 80 port - works fine.
>   wccp2_service_info 70 protocol = tcp flags = dst_ip_hash priority =
> 231 ports = 80
> If wccp redirects 443 too - then squid overflows and killed by kernel
>   wccp2_service_info 70 protocol = tcp flags = dst_ip_hash priority =
> 231 ports = 80,443
> 
> ---Before it died (HTTPS on):
> Mem:  16291720k total, 16125288k used,   166432k free,      540k
> buffers
> Swap:  8216568k total,  8112628k used,   103940k free,    27112k cached
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 30858 squid     20   0 22.7g  14g 3612 S  8.0 94.6  14:50.82 squid
> 
> 
> # free -m
>              total       used       free     shared    buffers
> cached
> Mem:         15909      15750        158          0          0
> 26
> -/+ buffers/cache:      15723        186
> Swap:         8023       7936         87
> 
> 
> Start Time:	Sat, 10 Dec 2016 07:52:50 GMT
> Current Time:	Sat, 10 Dec 2016 09:39:45 GMT
> 
> Connection information for squid:
> 	Number of clients accessing cache:	1305
> 	Number of HTTP requests received:	193434
> 	Number of ICP messages received:	0
> 	Number of ICP messages sent:	0
> 	Number of queued ICP replies:	0
> 	Number of HTCP messages received:	0
> 	Number of HTCP messages sent:	0
> 	Request failure ratio:	 0.00
> 	Average HTTP requests per minute since start:	1809.2
> 	Average ICP messages per minute since start:	0.0
> 	Select loop called: 4529796 times, 1.416 ms avg
> Cache information for squid:
> 	Hits as % of all requests:	5min: 0.0%, 60min: 0.0%
> 	Hits as % of bytes sent:	5min: 0.1%, 60min: -0.0%
> 	Memory hits as % of hit requests:	5min: 0.0%, 60min: 0.0%
> 	Disk hits as % of hit requests:	5min: 0.0%, 60min: 0.0%
> 	Storage Swap size:	82044 KB
> 	Storage Swap capacity:	80.1% used, 19.9% free
> 	Storage Mem size:	107876 KB
> 	Storage Mem capacity:	20.6% used, 79.4% free
> 	Mean Object Size:	29.54 KB
> 	Requests given to unlinkd:	9258
> Median Service Times (seconds)  5 min    60 min:
> 	HTTP Requests (All):   0.10857  0.04519
> 	Cache Misses:          0.01648  0.00678
> 	Cache Hits:            0.00000  0.00000
> 	Near Hits:             0.00000  0.00000
> 	Not-Modified Replies:  0.00000  0.00000
> 	DNS Lookups:           0.00860  0.00779
> 	ICP Queries:           0.00000  0.00000
> Resource usage for squid:
> 	UP Time:	6415.101 seconds
> 	CPU Time:	902.767 seconds
> 	CPU Usage:	14.07%
> 	CPU Usage, 5 minute avg:	15.97%
> 	CPU Usage, 60 minute avg:	13.96%
> 	Maximum Resident Size: 62241760 KB
> 	Page faults with physical i/o: 32647
> Memory accounted for:
> 	Total accounted:       1073388 KB
> 	memPoolAlloc calls:     12969
> 	memPoolFree calls:   35802441
> File descriptor usage for squid:
> 	Maximum number of file descriptors:   100000
> 	Largest file desc currently in use:   28744
> 	Number of file desc currently in use: 28738
> 	Files queued for open:                   0
> 	Available number of file descriptors: 71262
> 	Reserved number of file descriptors:   100
> 	Store Disk files open:                   0
> Internal Data Structures:
> 	 57337 StoreEntries
> 	 54560 StoreEntries with MemObjects
> 	    52 Hot Object Cache Items
> 	  2777 on-disk objects
> 
> ---after:
> /var/log/messages
> kernel: 11733 total pagecache pages
> kernel: 8957 pages in swap cache
> kernel: Swap cache stats: add 21118384, delete 21109427, find
> 12110273/12422740
> kernel: Free swap  = 0kB
> kernel: Total swap = 8216568kB
> kernel: 4194303 pages RAM
> kernel: 121373 pages reserved
> kernel: 11781 pages shared
> kernel: 4023631 pages non-shared
> ...omitted...
> kernel: Out of memory: Kill process 30858 (squid) score 954 or
> sacrifice child
> kernel: Killed process 30868, UID 23, (log_file_daemon) total-
> vm:26640kB, anon-rss:48kB, file-rss:512kB
> (squid-1): I don't handle this error well!
> Dec 10 12:44:27 localhost squid[30855]: Squid Parent: (squid-1) process
> 30858 exited due to signal 9 with status 0
> 
> 
> In attach all /var/log/messages output.
> Main task for the server is to block bad sites and bypass others on
> same IPs.
> Any ideas?
> 
> --
> Sergey