[squid-users] Memory leak (was: Squid 3.x never_direct and DNS requests problem.)
FUSTE Emmanuel
emmanuel.fuste at thalesgroup.com
Fri Aug 18 11:52:55 UTC 2017
Le 24/01/2017 à 10:55, FUSTE Emmanuel a écrit :
> Le 23/01/2017 à 23:41, Amos Jeffries a écrit :
>> On 24/01/2017 3:58 a.m., FUSTE Emmanuel wrote:
>>> All was carefully checked and nothing in my configuration (acl etc ...)
>>> explain why Squid insist to do DNS requests for requests forwarded to
>>> the peer(s).
>>>
>> <snip>
>>> #bug #4575
>>> url_rewrite_extras XXX
>>> store_id_extras XXX
>> I dont think that workaround is working.
>>
>>> ------------------------------------
>>>
>>> Since the switch from 3.5.12 to 3.5.19/23, I am able to use a simpler
>>> work around (I switched directly from 3.5.12 to 3.5.19 so I don't know
>>> when the behavior changed):
>>> Instead of installing a fake local DNS server and using
>>> dns_nameservers 127.0.0.1
>>> I could use
>>> dns_nameservers none
>>> Squid warn about non usable DNS and proceed normally. Before (tested
>>> with 3.5.12 and lower) Squid hang.
>>>
>> :-) nice.
>>
>> I'm prety sure this is still bug 4575. I've added a comment there to
>> mention how the workaround is broken, and your improved one.
>>
> Thank you !
> If there's anything I can help with to solve this bug, I'd be happy to.
>
> Emmanuel.
>
It seems that using this this workaround is a bad idea.
It expose or induce HUGE memory leak:
Current memory usage:
Pool Obj Size Chunks Allocated In
Use Idle Allocations Saved Rate
(bytes) KB/ch obj/ch (#) used free part
%Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB)
high (KB) high (hrs) %alloc (#) (KB) high (KB) (#)
%cnt %vol (#)/sec
cbdata idns_query (6) 8696
689173 5852587 5852587 0.00 87.949 689173
5852587 5852587 0.00 100.000 0 0 0 0
0.000 0.000 0.000
mem_node 4136 129945 524856 534368 3.85
7.887 129801 524275 534368 3.85 99.889 144
582 19715 4096894 0.873 9.316 0.003
fqdncache_entry 160 340128 53145 53145 0.00
0.799 340128 53145 53145 0.00 100.000 0 0
0 0 0.000 0.000 0.000
ipcache_entry 128 343083 42886 42886 0.00
0.644 343083 42886 42886 0.00 100.000 0 0
1 1 0.000 0.000 0.000
Short Strings 40 822953 32147 34373 0.28
0.483 819903 32028 34373 0.28 99.629 3050
120 605 247675706 52.776 5.447 0.166
cbdata generic_cbdata (14) 32
1026262 32071 32071 0.00 0.482 1026262 32071
32071 0.00 100.000 0 0 1 60 0.000 0.000 0.001
16KB Strings 16384 1306 20896
42624 3.15 0.314 1215 19440 42624 3.15
93.032 91 1456 13488 1976589 0.421 17.805 0.001
HttpHeaderEntry 56 372842 20390 21773 0.26
0.306 371478 20316 21773 0.26 99.634 1364 75
385 51897666 11.059 1.598 0.035
MemObject 328 35177 11268 12032 0.27
0.169 34892 11177 12032 0.27 99.190 285 92
233 1599153 0.341 0.288 0.001
HttpReply 280 35179 9620 10274 0.26
0.145 34893 9542 10274 0.26 99.187 286 79
199 4877844 1.039 0.751 0.003
Long Strings 512 18426 9213 9967 0.30
0.138 18176 9088 9967 0.30 98.643 250 125
203 5984528 1.275 1.685 0.004
Digest Scheme nonce's 72
109609 7707 14004 3.17 0.116 109609 7707
14004 3.17 100.000 0 0 7186 29051 0.006
0.001 0.001
Medium Strings 128 53288 6661 7158 0.26
0.100 53025 6629 7158 0.26 99.506 263 33 157
11608569 2.474 0.817 0.008
4KB Strings 4096 1104 4416 4916 0.28
0.066 1077 4308 4916 0.28 97.554 27 108
304 435808 0.093 0.981 0.000
StoreEntry 120 35177 4123 4402 0.27
0.062 34892 4089 4402 0.27 99.190 285 34 85
1599153 0.341 0.106 0.001
cbdata clientReplyContext (17)
4320 913 3852 5932
1.00 0.058 840 3544 5932 1.00 92.004 73 308
1047 1805429 0.385 4.288 0.001
cbdata ClientSocketContext (16)
4256 913 3795 5844
1.00 0.057 840 3492 5844 1.00 92.004 73 304
1031 1805429 0.385 4.225 0.001
1KB Strings 1024 3793 3793 4299 0.29
0.057 3766 3766 4299 0.29 99.288 27 27
155 421553 0.090 0.237 0.000
cbdata MemBuf (11) 64 35210 2201 2354 0.27
0.033 34920 2183 2354 0.27 99.176 290 19 48
9673363 2.061 0.340 0.006
HttpHdrCc 96 18070 1695 1802 0.27
0.025 17940 1682 1802 0.27 99.281 130 13 43
1112930 0.237 0.059 0.001
HttpRequest 1784 913 1591 2450 1.00
0.024 840 1464 2450 1.00 92.004 73 128
433 1805487 0.385 1.771 0.001
LRU policy node 24 35104 823 878 0.26
0.012 34823 817 878 0.26 99.200 281 7 17
45784 0.010 0.001 0.000
Comm::Connection 200 4116 804 1334 3.15
0.012 3786 740 1334 3.15 91.983 330 65
201 2095512 0.447 0.230 0.001
MD5 digest 16 35177 550 587 0.27
0.008 34892 546 587 0.27 99.190 285 5 12
1804763 0.385 0.016 0.001
cbdata Server (13) 416 1318 536 1140 3.15
0.008 1230 500 1140 3.15 93.323 88 36
346 305920 0.065 0.070 0.000
16K Buffer 16384 33 528
1488 1.23 0.008 17 272 1488 1.23 51.515 16
256 1040 395413 0.084 3.562 0.000
64K Buffer 65536 8 512
1344 3.17 0.008 1 64 1344 3.17 12.500 7 448
1344 199003 0.042 7.170 0.000
MemBlob 48 8078 379 418 0.30
0.006 7945 373 418 0.30 98.354 133 7 40
63235417 13.475 1.669 0.042
cbdata ClientHttpRequest (15) 384
913 343 528 1.00 0.005 840 315 528 1.00
92.004 73 28 93 1805429 0.385 0.381 0.001
cbdata TunnelStateData (25) 304
894 266 405 1.84 0.004 822 245 405 1.84
91.946 72 22 74 177202 0.038 0.030 0.000
cbdata clientStreamNode (18) 128
1826 229 352 1.00 0.003 1680 210 352 1.00
92.004 146 19 62 3610290 0.769 0.254 0.002
ClientInfo 448 417 183 183 0.04 0.003
417 183 183 0.04 100.000 0 0 0 0
0.000 0.000 0.000
Auth::Digest::UserRequest 152
820 122 195 1.84 0.002 761 113 195 1.84
92.805 59 9 36 1745806 0.372 0.146 0.001
8K Buffer 8192 10 80 568 0.60 0.001
1 8 568 0.60 10.000 9 72 560 2164806
0.461 9.750 0.001
Auth::Digest::User 208 350 72 79 0.18 0.001
345 71 79 0.18 98.571 5 2 13 125838
0.027 0.014 0.000
4K Buffer 4096 13 52 124 2.55 0.001
10 40 124 2.55 76.923 3 12 84 1392001
0.297 3.135 0.001
MimeEntry 128 177 23 23 4.42 0.000
177 23 23 4.42 100.000 0 0 0 0 0.000
0.000 0.000
2K Buffer 2048 10 20 68 2.25 0.000
5 10 68 2.25 50.000 5 10 62 15326460
3.266 17.257 0.010
AuthUserIP 64 306 20 21 0.20 0.000
306 20 21 0.20 100.000 0 0 2 2 0.000
0.000 0.000
NotePairs::Entry 48 399 19 35 0.31 0.000
338 16 35 0.31 84.712 61 3 9 4729264
1.008 0.125 0.003
acl_proxy_auth_match_cache 40
306 12 13 0.20 0.000 306 12 13 0.20
100.000 0 0 2 2 0.000 0.000 0.000
cbdata HttpStateData (29) 320
33 11 30 1.23 0.000 17 6 30 1.23
51.515 16 5 21 337305 0.072 0.059 0.000
AuthUserHashPointer 24 344 9 9 0.20 0.000
344 9 9 0.20 100.000 0 0 1 0 0.000
0.000 0.000
cbdata store_client (22) 160
37 6 15 1.23 0.000 19 3 15 1.23
51.351 18 3 11 1665410 0.355 0.146 0.001
cbdata FwdState (28) 176 33 6 17 1.23 0.000
17 3 17 1.23 51.515 16 3 12 337300
0.072 0.033 0.000
cbdata IdleConnList (30) 4160
1 5 5 4.41 0.000 1 5 5 4.41 100.000
0 0 5 9196 0.002 0.021 0.000
netdbEntry 168 17 3 3 0.27 0.000
17 3 3 0.27 100.000 0 0 0 0 0.000
0.000 0.000
cbdata helper_server (24) 240
10 3 3 2.63 0.000 10 3 3 2.63
100.000 0 0 0 0 0.000 0.000 0.000
Acl::AndNode 160 13 3 3 4.42 0.000
13 3 3 4.42 100.000 0 0 0 0 0.000
0.000 0.000
cbdata ACLFilledChecklist (20)
456 4 2 6 2.63
0.000 1 1 6 2.63 25.000 3 2 6
2917323 0.622 0.731 0.002
cbdata Tree (3) 216 6 2 2 4.42 0.000 6
2 2 4.42 100.000 0 0 0 0 0.000 0.000
0.000
cbdata CachePeer (4) 872 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ev_entry 48 16 1 2 0.49 0.000
11 1 2 0.49 68.750 5 1 2 43525 0.009
0.001 0.000
net_db_name 32 21 1 1 0.27 0.000
21 1 1 0.27 100.000 0 0 0 0 0.000
0.000 0.000
acl_ip_data 96 7 1 1 4.42 0.000 7
1 1 4.42 100.000 0 0 1 4 0.000 0.000
0.000
HttpHdrScTarget 88 7 1 1 0.19 0.000 7
1 1 0.19 100.000 0 0 1 73 0.000
0.000 0.000
cbdata BodyPipe (32) 152 4 1 4 0.76 0.000 1
1 4 0.76 25.000 3 1 4 33979 0.007
0.003 0.000
cbdata ConnOpener (27) 136 4
1 4 0.49 0.000 0 0 4 0.49 0.000 4 1
4 250616 0.053 0.019 0.000
cbdata ClientRequestContext (19)
104 5 1 4 0.60
0.000 0 0 4 0.60 0.000 5 1 4 1827236
0.389 0.104 0.001
ACLStrategised 160 3 1 1 4.42 0.000 3
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
cbdata CbDataList (1) 40 12
1 1 4.42 0.000 12 1 1 4.42 100.000 0
0 0 0 0.000 0.000 0.000
ACLSourceIP 144 3 1 1 4.42 0.000 3
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
cbdata Logfile (9) 352 1 1 1 3.75 0.000 1
1 1 3.75 100.000 0 0 1 0 0.000 0.000
0.000
ACLStringData 56 6 1 1 4.42 0.000 6
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
wordlist 16 21 1 4 4.42 0.000
21 1 4 4.42 100.000 0 0 4 18 0.000
0.000 0.000
ACLStrategised 160 2 1 1 4.42 0.000 2
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
Acl::NotNode 160 2 1 1 4.42 0.000 2
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLRegexData 16 18 1 1 4.42 0.000
18 1 1 4.42 100.000 0 0 0 0 0.000
0.000 0.000
RegexList 88 3 1 1 4.42 0.000 3
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLUserData 64 4 1 1 4.42 0.000 4
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
cbdata ps_state (26) 256 1 1 1 4.41 0.000 0
0 1 4.41 0.000 1 1 1 543789 0.116
0.077 0.000
cbdata ErrorState (21) 232 1
1 1 1.91 0.000 0 0 1 1.91 0.000 1 1
1 1271537 0.271 0.162 0.001
cbdata TcpAcceptor (12) 104 2
1 1 4.42 0.000 2 1 1 4.42 100.000 0
0 0 0 0.000 0.000 0.000
cbdata CbDataList (33) 96 2
1 3 1.34 0.000 0 0 3 1.34 0.000 2
1 3 738165 0.157 0.039 0.000
cbdata helper (7) 168 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLStrategised 160 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLProxyAuth 152 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLDestinationIP 144 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
cbdata StateData (23) 48 3
1 1 2.63 0.000 0 0 1 2.63 0.000 3
1 1 1155036 0.246 0.030 0.001
Helper::Request 48 3 1 1 2.63 0.000 0
0 1 2.63 0.000 3 1 1 1155036 0.246
0.030 0.001
HttpHdrSc 16 7 1 1 0.19 0.000 7
1 1 0.19 100.000 0 0 1 73 0.000
0.000 0.000
cbdata RemovalPolicy (10) 104
1 1 1 4.42 0.000 1 1 1 4.42 100.000
0 0 0 0 0.000 0.000 0.000
ACLHTTPHeaderData 48 2 1 1 4.42 0.000 2
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
FwdServer 24 3 1 1 4.41 0.000 0
0 1 4.41 0.000 3 1 1 1631367 0.348
0.022 0.001
cbdata RemovalPurgeWalker (34) 72
1 1 1 4.17 0.000 0 0 1 4.17 0.000
1 1 1 6250 0.001 0.000 0.000
cbdata CbDataList (2) 64 1
1 1 4.42 0.000 1 1 1 4.42 100.000 0
0 0 0 0.000 0.000 0.000
ACLNoteData 40 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
CacheDigest 40 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLDomainData 16 2 1 1 4.42 0.000 2
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLHierCodeData 32 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLMethodData 16 2 1 1 4.42 0.000 2
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLTimeData 32 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
ACLASN 16 2 1 1 4.42 0.000 2
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
HttpHdrRange 32 1 1 1 2.85 0.000 0
0 1 2.85 0.000 1 1 1 7768 0.002
0.000 0.000
ACLProtocolData 16 1 1 1 4.42 0.000 1
1 1 4.42 100.000 0 0 0 0 0.000 0.000
0.000
HttpHdrRangeSpec 16 1 1 1 2.85 0.000 0
0 1 2.85 0.000 1 1 1 7768 0.002
0.000 0.000
32K Buffer 32768 0 0 96
4.28 0.000 0 0 96 4.28 -1.000 0 0 96
534 0.000 0.010 0.000
dlink_node 24 0 0 1 4.41 0.000 0
0 1 4.41 -1.000 0 0 1 7 0.000 0.000
0.000
HttpHdrContRange 24 0 0 1 3.76 0.000 0
0 1 3.76 -1.000 0 0 1 657 0.000
0.000 0.000
cbdata netdbExchangeState (31)
4176 0 0 5 4.41
0.000 0 0 5 4.41 -1.000 0 0 5 0
0.000 0.000 0.000
cbdata StatObjectsState (35) 48
0 0 1 0.14 0.000 0 0 1 0.14 -1.000
0 0 1 1 0.000 0.000 0.000
cbdata StoreSearchHashIndex (36)
104 0 0 1 0.14
0.000 0 0 1 0.14 -1.000 0 0 1 1
0.000 0.000 0.000
Total 1 4164421 6654528 6654528
0.00 100.000 4156051 6650022 6650022 0.00 99.932
8370 4506 27047 461504099 98.340 95.012 0.312
Cumulative allocated volume: 181.888 GB
Current overhead: 34740 bytes (0.001%)
Idle pool limit: 5.00 MB
Total Pools created: 111
Pools ever used: 102 (shown above)
Currently in use: 85
String Pool Impact
(%strings) (%volume)
Short Strings 91 42
Medium Strings 6 9
Long Strings 2 12
1KB Strings 0 5
4KB Strings 0 6
16KB Strings 0 26
Other Strings 0 1
Large buffers: 0 (0 KB)
A special/real "dns_nameservers none" handling need to be implemented.
So I'm back with fake local DNS server.
We will see if my Squid instances will survive more than 8h with that.
Emmanuel.
More information about the squid-users
mailing list