[squid-users] bad kernel logs on high traffic server
Ahmed Allzaeem
ahmed.zaeem at netstream.ps
Fri Nov 7 20:07:45 UTC 2014
Hi guys , I have server with about 1000 users and its okay with 2 workers
Now I added more 1000 and its now 2000 users with 4 workers
But sometime some process of squid get killed suddenly and here is log below
,
Here is sysctl.conf
[root at Largerock-squid ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
################net.ipv4.ip_forward = 0
net.ipv4.ip_forward = 1
# Controls source route verification
#################net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.rp_filter = 0
###################################
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
###############################
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
################################
###############################
fs.file-max = 1131072
net.local.dgram.recvspace = 1262144
net.local.dgram.maxdgram = 116384
net.netfilter.nf_conntrack_max = 1130000
########################################
net.nf_conntrack_max = 1131072
#########################################
####kernel: possible SYN flooding on port 80. Sending cookies.#####
net.ipv4.tcp_syncookies=0
im wondering wt should I do , or modify my kernel linux
the error code is below :
Nov 7 14:52:16 Largerock-squid snmpd[1576]: Connection from UDP:
[xxx]:57067xxx]
Nov 7 14:52:17 Largerock-squid kernel: snmpd invoked oom-killer:
gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Nov 7 14:52:17 Largerock-squid kernel: snmpd cpuset=/ mems_allowed=0
Nov 7 14:52:17 Largerock-squid kernel: Pid: 1576, comm: snmpd Not tainted
2.6.32-431.el6.x86_64 #1
Nov 7 14:52:17 Largerock-squid kernel: Call Trace:
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff810d05b1>] ?
cpuset_print_task_mems_allowed+0x91/0xb0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81122960>] ?
dump_header+0x90/0x1b0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8122798c>] ?
security_real_capable_noaudit+0x3c/0x70
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81122de2>] ?
oom_kill_process+0x82/0x2a0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81122d21>] ?
select_bad_process+0xe1/0x120
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81123220>] ?
out_of_memory+0x220/0x3c0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8112fb3c>] ?
__alloc_pages_nodemask+0x8ac/0x8d0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81167a9a>] ?
alloc_pages_current+0xaa/0x110
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8111fd57>] ?
__page_cache_alloc+0x87/0x90
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8111f73e>] ?
find_get_page+0x1e/0xa0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81120cf7>] ?
filemap_fault+0x1a7/0x500
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8114a084>] ?
__do_fault+0x54/0x530
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8114a657>] ?
handle_pte_fault+0xf7/0xb00
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81447c54>] ?
move_addr_to_kernel+0x64/0x70
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff810129de>] ?
copy_user_generic+0xe/0x20
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8114b28a>] ?
handle_mm_fault+0x22a/0x300
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8104a8d8>] ?
__do_page_fault+0x138/0x480
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff81448119>] ?
sys_sendto+0x139/0x190
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff810149b9>] ?
read_tsc+0x9/0x20
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff810a70a1>] ?
ktime_get_ts+0xb1/0xf0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8119fb18>] ?
poll_select_copy_remaining+0xf8/0x150
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8152d45e>] ?
do_page_fault+0x3e/0xa0
Nov 7 14:52:17 Largerock-squid kernel: [<ffffffff8152a815>] ?
page_fault+0x25/0x30
Nov 7 14:52:17 Largerock-squid kernel: Mem-Info:
Nov 7 14:52:17 Largerock-squid kernel: Node 0 DMA per-cpu:
Nov 7 14:52:17 Largerock-squid kernel: CPU 0: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 1: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 2: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 3: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 4: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 5: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 6: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: CPU 7: hi: 0, btch: 1 usd:
0
Nov 7 14:52:17 Largerock-squid kernel: Node 0 DMA32 per-cpu:
Nov 7 14:52:17 Largerock-squid kernel: CPU 0: hi: 186, btch: 31 usd:
77
Nov 7 14:52:17 Largerock-squid kernel: CPU 1: hi: 186, btch: 31 usd:
34
Nov 7 14:52:17 Largerock-squid kernel: CPU 2: hi: 186, btch: 31 usd:
31
Nov 7 14:52:17 Largerock-squid kernel: CPU 3: hi: 186, btch: 31 usd:
33
Nov 7 14:52:17 Largerock-squid kernel: CPU 4: hi: 186, btch: 31 usd:
24
Nov 7 14:52:17 Largerock-squid kernel: CPU 5: hi: 186, btch: 31 usd:
24
Nov 7 14:52:17 Largerock-squid kernel: CPU 6: hi: 186, btch: 31 usd:
22
Nov 7 14:52:17 Largerock-squid kernel: CPU 7: hi: 186, btch: 31 usd:
17
Nov 7 14:52:17 Largerock-squid kernel: Node 0 Normal per-cpu:
Nov 7 14:52:17 Largerock-squid kernel: CPU 0: hi: 186, btch: 31 usd:
148
Nov 7 14:52:17 Largerock-squid kernel: CPU 1: hi: 186, btch: 31 usd:
59
Nov 7 14:52:17 Largerock-squid kernel: CPU 2: hi: 186, btch: 31 usd:
64
Nov 7 14:52:17 Largerock-squid kernel: CPU 3: hi: 186, btch: 31 usd:
34
Nov 7 14:52:17 Largerock-squid kernel: CPU 4: hi: 186, btch: 31 usd:
21
Nov 7 14:52:17 Largerock-squid kernel: CPU 5: hi: 186, btch: 31 usd:
22
Nov 7 14:52:17 Largerock-squid kernel: CPU 6: hi: 186, btch: 31 usd:
50
Nov 7 14:52:17 Largerock-squid kernel: CPU 7: hi: 186, btch: 31 usd:
55
Nov 7 14:52:17 Largerock-squid kernel: active_anon:3847274
inactive_anon:2365500 isolated_anon:0
Nov 7 14:52:17 Largerock-squid kernel: active_file:77 inactive_file:0
isolated_file:32
Nov 7 14:52:17 Largerock-squid kernel: unevictable:0 dirty:1 writeback:0
unstable:0
Nov 7 14:52:17 Largerock-squid kernel: free:41667 slab_reclaimable:28547
slab_unreclaimable:280933
Nov 7 14:52:17 Largerock-squid kernel: mapped:2365635 shmem:2365544
pagetables:48036 bounce:0
Nov 7 14:52:17 Largerock-squid kernel: Node 0 DMA free:15548kB min:36kB
low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15156kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
yes
Nov 7 14:52:17 Largerock-squid kernel: lowmem_reserve[]: 0 3000 26482 26482
Nov 7 14:52:17 Largerock-squid kernel: Node 0 DMA32 free:101504kB
min:7652kB low:9564kB high:11476kB active_anon:1681480kB
inactive_anon:47812kB active_file:8kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB
dirty:0kB writeback:0kB mapped:47836kB shmem:47812kB
slab_reclaimable:18296kB slab_unreclaimable:591000kB kernel_stack:56kB
pagetables:3520kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64
all_unreclaimable? yes
Nov 7 14:52:17 Largerock-squid kernel: lowmem_reserve[]: 0 0 23482 23482
Nov 7 14:52:17 Largerock-squid kernel: Node 0 Normal free:49616kB
min:59892kB low:74864kB high:89836kB active_anon:13707616kB
inactive_anon:9414188kB active_file:300kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):128kB present:24046080kB mlocked:0kB
dirty:4kB writeback:0kB mapped:9414704kB shmem:9414364kB
slab_reclaimable:95892kB slab_unreclaimable:533024kB kernel_stack:1648kB
pagetables:188624kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:458 all_unreclaimable? yes
Nov 7 14:52:17 Largerock-squid kernel: lowmem_reserve[]: 0 0 0 0
Nov 7 14:52:17 Largerock-squid kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB
1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15548kB
Nov 7 14:52:17 Largerock-squid kernel: Node 0 DMA32: 23974*4kB 1*8kB 0*16kB
0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 102048kB
Nov 7 14:52:17 Largerock-squid kernel: Node 0 Normal: 11163*4kB 0*8kB
0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB =
48748kB
Nov 7 14:52:17 Largerock-squid kernel: 2365580 total pagecache pages
Nov 7 14:52:17 Largerock-squid kernel: 0 pages in swap cache
Nov 7 14:52:17 Largerock-squid kernel: Swap cache stats: add 0, delete 0,
find 0/0
Nov 7 14:52:17 Largerock-squid kernel: Free swap = 0kB
Nov 7 14:52:17 Largerock-squid kernel: Total swap = 0kB
Nov 7 14:52:17 Largerock-squid kernel: 6881264 pages RAM
Nov 7 14:52:17 Largerock-squid kernel: 147804 pages reserved
Nov 7 14:52:17 Largerock-squid kernel: 4911130 pages shared
Nov 7 14:52:17 Largerock-squid kernel: 4272662 pages non-shared
Nov 7 14:52:17 Largerock-squid kernel: [ pid ] uid tgid total_vm
rss cpu oom_adj oom_score_adj name
Nov 7 14:52:17 Largerock-squid kernel: [ 500] 0 500 2737
179 5 -17 -1000 udevd
Nov 7 14:52:17 Largerock-squid kernel: [ 921] 0 921 2662
107 6 -17 -1000 udevd
Nov 7 14:52:17 Largerock-squid kernel: [ 922] 0 922 2662
107 7 -17 -1000 udevd
Nov 7 14:52:17 Largerock-squid kernel: [ 1230] 0 1230 23294
68 4 -17 -1000 auditd
Nov 7 14:52:17 Largerock-squid kernel: [ 1254] 0 1254 1540
24 4 0 0 portreserve
Nov 7 14:52:17 Largerock-squid kernel: [ 1261] 0 1261 63856
235 1 0 0 rsyslogd
Nov 7 14:52:17 Largerock-squid kernel: [ 1290] 0 1290 2738
62 6 0 0 irqbalance
Nov 7 14:52:17 Largerock-squid kernel: [ 1304] 32 1304 4744
62 6 0 0 rpcbind
Nov 7 14:52:17 Largerock-squid kernel: [ 1322] 29 1322 5837
114 1 0 0 rpc.statd
Nov 7 14:52:17 Largerock-squid kernel: [ 1432] 81 1432 5351
71 2 0 0 dbus-daemon
Nov 7 14:52:17 Largerock-squid kernel: [ 1448] 0 1448 47336
223 4 0 0 cupsd
Nov 7 14:52:17 Largerock-squid kernel: [ 1473] 0 1473 1020
27 5 0 0 acpid
Nov 7 14:52:17 Largerock-squid kernel: [ 1482] 68 1482 9476
234 1 0 0 hald
Nov 7 14:52:17 Largerock-squid kernel: [ 1483] 0 1483 5082
54 4 0 0 hald-runner
Nov 7 14:52:17 Largerock-squid kernel: [ 1512] 0 1512 5612
41 5 0 0 hald-addon-inpu
Nov 7 14:52:17 Largerock-squid kernel: [ 1530] 68 1530 4484
36 4 0 0 hald-addon-acpi
Nov 7 14:52:17 Largerock-squid kernel: [ 1551] 0 1551 96434
124 1 0 0 automount
Nov 7 14:52:17 Largerock-squid kernel: [ 1567] 0 1567 1570
32 4 0 0 mcelog
Nov 7 14:52:17 Largerock-squid kernel: [ 1576] 0 1576 49460
955 5 0 0 snmpd
Nov 7 14:52:17 Largerock-squid kernel: [ 1588] 0 1588 16655
177 4 -17 -1000 sshd
Nov 7 14:52:17 Largerock-squid kernel: [ 1664] 0 1664 20326
226 6 0 0 master
Nov 7 14:52:17 Largerock-squid kernel: [ 1677] 89 1677 20389
221 0 0 0 qmgr
Nov 7 14:52:17 Largerock-squid kernel: [ 1688] 0 1688 27580
35 1 0 0 abrtd
Nov 7 14:52:17 Largerock-squid kernel: [ 1703] 0 1703 29324
152 2 0 0 crond
Nov 7 14:52:17 Largerock-squid kernel: [ 1717] 500 1717 5166698
14730 4 0 0 squid
Nov 7 14:52:17 Largerock-squid kernel: [ 1719] 500 1719 5168582
21614 0 0 0 squid
Nov 7 14:52:17 Largerock-squid kernel: [ 1721] 500 1721 6119473
2180266 6 0 0 squid
Nov 7 14:52:17 Largerock-squid kernel: [ 1722] 500 1722 6136601
2165217 4 0 0 squid
Nov 7 14:52:17 Largerock-squid kernel: [ 1723] 500 1723 6252323
2575690 4 0 0 squid
Nov 7 14:52:17 Largerock-squid kernel: [ 1724] 500 1724 5950985
1737617 0 0 0 squid
Nov 7 14:52:17 Largerock-squid kernel: [ 1742] 0 1742 5385
46 1 0 0 atd
Nov 7 14:52:17 Largerock-squid kernel: [ 1754] 0 1754 15587
89 4 0 0 certmonger
Nov 7 14:52:17 Largerock-squid kernel: [ 1770] 0 1770 1016
18 4 0 0 mingetty
Nov 7 14:52:17 Largerock-squid kernel: [ 1772] 0 1772 1016
18 3 0 0 mingetty
Nov 7 14:52:17 Largerock-squid kernel: [ 1774] 0 1774 1016
19 6 0 0 mingetty
Nov 7 14:52:17 Largerock-squid kernel: [ 1776] 0 1776 1016
19 2 0 0 mingetty
Nov 7 14:52:17 Largerock-squid kernel: [ 1778] 0 1778 1016
17 6 0 0 mingetty
Nov 7 14:52:17 Largerock-squid kernel: [ 1780] 0 1780 1016
19 5 0 0 mingetty
Nov 7 14:52:17 Largerock-squid kernel: [ 7149] 89 7149 20346
220 4 0 0 pickup
Nov 7 14:52:17 Largerock-squid kernel: [ 7295] 0 7295 25095
264 6 0 0 sshd
Nov 7 14:52:17 Largerock-squid kernel: [ 7297] 0 7297 27085
90 7 0 0 bash
Nov 7 14:52:17 Largerock-squid kernel: [ 7316] 0 7316 28210
211 0 0 0 htop
Nov 7 14:52:17 Largerock-squid kernel: [ 7339] 0 7339 27085
91 5 0 0 bash
Nov 7 14:52:17 Largerock-squid kernel: [ 7360] 0 7360 25224
31 6 0 0 tailf
Nov 7 14:52:17 Largerock-squid kernel: [ 7361] 0 7361 27085
93 3 0 0 bash
Nov 7 14:52:17 Largerock-squid kernel: [ 7380] 0 7380 27057
220 4 0 0 nano
Nov 7 14:52:17 Largerock-squid kernel: [ 7383] 0 7383 27085
89 5 0 0 bash
Nov 7 14:52:17 Largerock-squid kernel: [ 7402] 0 7402 27590
226 0 0 0 iptraf
Nov 7 14:52:17 Largerock-squid kernel: [ 7412] 0 7412 27085
91 4 0 0 bash
Nov 7 14:52:17 Largerock-squid kernel: [ 7431] 0 7431 25224
30 5 0 0 tailf
Nov 7 14:52:17 Largerock-squid kernel: Out of memory: Kill process 1723
(squid) score 384 or sacrifice child
Nov 7 14:52:17 Largerock-squid kernel: Killed process 1723, UID 500,
(squid) total-vm:25009292kB, anon-rss:4356424kB, file-rss:5946336kB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20141107/f13fe65d/attachment-0001.html>
More information about the squid-users
mailing list