[squid-users] leaking memory in squid 3.4.8 and 3.4.7.
Victor Sudakov
sudakov at sibptus.tomsk.ru
Wed Oct 1 07:45:18 UTC 2014
Amos Jeffries wrote:
> >>>
> >>> You have 200 MB of RAM locked up in IDENT lookups.
> >>>
> >>> Probably http://bugs.squid-cache.org/show_bug.cgi?id=3803
> >>
> >> I have temporarily disabled IDENT-related acls. Squid still grows
> >> in memory, but more slowly. So IDENT was certainly one of the
> >> major causes of the leak.
> >>
> >> Could we make another iteration of looking at the cachemgr.cgi
> >> memory counters?
> >>
> >
> > Attaching two cachemgr reports: right after squid restart and
> > several hours later (grown to 816M in SIZE).
> >
>
> These are still showing over 47K IDENT lookups.
>
> Probably you did not disable all the uses of IDENT. You need to both
> set "ident_access deny all" and remove use of ident type ACLs.
Bingo! After setting "ident_access deny all" squid does not grow
infinitely any more. However, it remains a major CPU hog.
I am attaching the output of mgr:info, mgr:mem and top as requested.
===
last pid: 81001; load averages: 1.49, 1.48, 1.32 up 12+14:20:21 12:58:39
180 processes: 2 running, 178 sleeping
Mem: 478M Active, 244M Inact, 189M Wired, 43M Cache, 107M Buf, 2640K Free
Swap: 2548M Total, 67M Used, 2481M Free, 2% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
39322 squid 1 117 0 352M 323M CPU0 0 150:05 95.36% squid
84342 squid 1 44 0 5700K 4140K sbwait 1 0:09 0.00% ntlm_auth
84529 squid 1 44 0 4676K 2272K sbwait 1 0:03 0.00% ntlm_auth
39325 squid 1 44 0 4508K 1452K piperd 1 0:03 0.00% unlinkd
84724 squid 1 44 0 3652K 1892K sbwait 1 0:02 0.00% ntlm_auth
84725 squid 1 44 0 3652K 1740K sbwait 1 0:01 0.00% ntlm_auth
84726 squid 1 44 0 3652K 1632K sbwait 1 0:01 0.00% ntlm_auth
93981 squid 1 44 0 3652K 1556K sbwait 0 0:01 0.00% ntlm_auth
12271 squid 1 44 0 3652K 1500K sbwait 0 0:01 0.00% ntlm_auth
12272 squid 1 44 0 3652K 1464K sbwait 1 0:00 0.00% ntlm_auth
12273 squid 1 44 0 3652K 1436K sbwait 0 0:00 0.00% ntlm_auth
12274 squid 1 44 0 3652K 1408K sbwait 1 0:00 0.00% ntlm_auth
13032 squid 1 44 0 3652K 1392K sbwait 1 0:00 0.00% ntlm_auth
13033 squid 1 44 0 3652K 1384K sbwait 1 0:00 0.00% ntlm_auth
14750 squid 1 44 0 3652K 1376K sbwait 1 0:00 0.00% ntlm_auth
14751 squid 1 44 0 3652K 1364K sbwait 0 0:00 0.00% ntlm_auth
14752 squid 1 44 0 3652K 1356K sbwait 1 0:00 0.00% ntlm_auth
18076 squid 1 44 0 3652K 1344K sbwait 1 0:00 0.00% ntlm_auth
HTTP/1.1 200 OK
Server: squid/3.4.8
Mime-Version: 1.0
Date: Wed, 01 Oct 2014 06:00:28 GMT
Content-Type: text/plain
Expires: Wed, 01 Oct 2014 06:00:28 GMT
Last-Modified: Wed, 01 Oct 2014 06:00:28 GMT
X-Cache: MISS from proxy.sibptus.transneft.ru
X-Cache-Lookup: MISS from proxy.sibptus.transneft.ru:3128
Via: 1.1 proxy.sibptus.transneft.ru (squid/3.4.8)
Connection: close
Squid Object Cache: Version 3.4.8
Build Info:
Start Time: Tue, 30 Sep 2014 15:56:38 GMT
Current Time: Wed, 01 Oct 2014 06:00:28 GMT
Connection information for squid:
Number of clients accessing cache: 170
Number of HTTP requests received: 375550
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 445.1
Average ICP messages per minute since start: 0.0
Select loop called: 32559927 times, 1.555 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 3.2%, 60min: 4.1%
Hits as % of bytes sent: 5min: 22.1%, 60min: 11.3%
Memory hits as % of hit requests: 5min: 7.6%, 60min: 20.8%
Disk hits as % of hit requests: 5min: 18.8%, 60min: 20.1%
Storage Swap size: 471890 KB
Storage Swap capacity: 90.0% used, 10.0% free
Storage Mem size: 130484 KB
Storage Mem capacity: 99.6% used, 0.4% free
Mean Object Size: 23.64 KB
Requests given to unlinkd: 65340
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 1.00114 0.44492
Cache Misses: 2.13280 1.00114
Cache Hits: 0.94847 0.05951
Near Hits: 1.11539 0.68577
Not-Modified Replies: 0.52331 0.08265
DNS Lookups: 0.30421 0.12472
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 50629.945 seconds
CPU Time: 9112.184 seconds
CPU Usage: 18.00%
CPU Usage, 5 minute avg: 97.13%
CPU Usage, 60 minute avg: 75.39%
Maximum Resident Size: 1369760 KB
Page faults with physical i/o: 483
Memory accounted for:
Total accounted: 164823 KB
memPoolAlloc calls: 39
memPoolFree calls: 463653713
File descriptor usage for squid:
Maximum number of file descriptors: 16384
Largest file desc currently in use: 709
Number of file desc currently in use: 433
Files queued for open: 0
Available number of file descriptors: 15951
Reserved number of file descriptors: 100
Store Disk files open: 6
Internal Data Structures:
20041 StoreEntries
6627 StoreEntries with MemObjects
6597 Hot Object Cache Items
19964 on-disk objects
HTTP/1.1 200 OK
Server: squid/3.4.8
Mime-Version: 1.0
Date: Wed, 01 Oct 2014 06:00:35 GMT
Content-Type: text/plain
Expires: Wed, 01 Oct 2014 06:00:35 GMT
Last-Modified: Wed, 01 Oct 2014 06:00:35 GMT
X-Cache: MISS from proxy.sibptus.transneft.ru
X-Cache-Lookup: MISS from proxy.sibptus.transneft.ru:3128
Via: 1.1 proxy.sibptus.transneft.ru (squid/3.4.8)
Connection: close
Current memory usage:
Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate
(bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec
mem_node 4120 32586 131108 132665 4.19 79.209 32586 131108 132665 4.19 100.000 0 0 0 0 0.000 0.000 43.406
Medium Strings 128 110027 13754 23103 0.52 8.309 110027 13754 23103 0.52 100.000 0 0 0 0 0.000 0.000 8433.202
Short Strings 36 248037 8721 12456 0.52 5.268 248037 8721 12456 0.52 100.000 0 0 0 0 0.000 0.000 9548.156
NotePairs::Entry 24 108787 2550 4292 0.52 1.540 108787 2550 4292 0.52 100.000 0 0 0 0 0.000 0.000 8299.001
HttpHeaderEntry 28 62727 1716 2381 2.42 1.036 62727 1716 2381 2.42 100.000 0 0 0 0 0.000 0.000 418.636
4K Buffer 4096 308 1232 2152 0.41 0.744 308 1232 2152 0.41 100.000 0 0 0 0 0.000 0.000 14.047
StoreEntry 60 20030 1174 1311 2.54 0.709 20030 1174 1311 2.54 100.000 0 0 0 0 0.000 0.000 12.050
HttpReply 172 6647 1117 1601 2.42 0.675 6647 1117 1601 2.42 100.000 0 0 0 0 0.000 0.000 36.288
MemObject 136 6625 880 1264 2.42 0.532 6625 880 1264 2.42 100.000 0 0 0 0 0.000 0.000 12.138
cbdata clientReplyContext (20) 4224 109 450 933 4.00 0.272 109 450 933 4.00 100.000 0 0 0 0 0.000 0.000 12.833
cbdata ClientSocketContext (19) 4188 109 446 925 4.00 0.269 109 446 925 4.00 100.000 0 0 0 0 0.000 0.000 12.833
HttpRequest 1400 247 338 468 0.44 0.204 247 338 468 0.44 100.000 0 0 0 0 0.000 0.000 12.833
MD5 digest 16 20030 313 350 2.54 0.189 20030 313 350 2.54 100.000 0 0 0 0 0.000 0.000 15.109
LRU policy node 12 26505 311 369 2.54 0.188 26505 311 369 2.54 100.000 0 0 0 0 0.000 0.000 4.109
HttpHdrCc 60 4456 262 347 2.44 0.158 4456 262 347 2.44 100.000 0 0 0 0 0.000 0.000 13.567
16K Buffer 16384 16 256 1040 1.31 0.155 16 256 1040 1.31 100.000 0 0 0 0 0.000 0.000 6.651
cbdata MemBuf (13) 36 6662 235 336 2.42 0.141 6662 235 336 2.42 100.000 0 0 0 0 0.000 0.000 60.437
cbdata IdleConnList (38) 4136 28 114 598 4.34 0.068 28 114 598 4.34 100.000 0 0 0 0 0.000 0.000 1.871
Comm::Connection 156 734 112 236 0.44 0.068 734 112 236 0.44 100.000 0 0 0 0 0.000 0.000 27.589
cbdata ConnStateData (17) 220 376 81 131 0.41 0.049 376 81 131 0.41 100.000 0 0 0 0 0.000 0.000 5.702
ipcache_entry 64 1199 75 79 0.53 0.045 1199 75 79 0.53 100.000 0 0 0 0 0.000 0.000 1.353
ClientInfo 340 170 57 57 0.02 0.034 170 57 57 0.02 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ACLFilledChecklist (23) 296 145 42 60 0.53 0.025 145 42 60 0.53 100.000 0 0 0 0 0.000 0.000 12.833
Long Strings 512 64 32 108 1.53 0.019 64 32 108 1.53 100.000 0 0 0 0 0.000 0.000 17.954
cbdata ClientHttpRequest (18) 172 109 19 38 4.00 0.011 109 19 38 4.00 100.000 0 0 0 0 0.000 0.000 12.833
8K Buffer 8192 2 16 584 4.35 0.010 2 16 584 4.35 100.000 0 0 0 0 0.000 0.000 14.123
cbdata clientStreamNode (21) 72 218 16 32 4.00 0.009 218 16 32 4.00 100.000 0 0 0 0 0.000 0.000 25.667
2K Buffer 2048 7 14 124 4.49 0.008 7 14 124 4.49 100.000 0 0 4 19 0.000 0.000 99.532
1KB Strings 1024 11 11 57 4.28 0.007 11 11 57 4.28 100.000 0 0 0 0 0.000 0.000 9.698
cbdata helper_stateful_server (26) 156 68 11 11 1.53 0.006 68 11 11 1.53 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ClientRequestContext (22) 64 148 10 14 0.53 0.006 148 10 14 0.53 100.000 0 0 0 0 0.000 0.000 12.833
cbdata TunnelStateData (30) 116 74 9 17 2.87 0.005 74 9 17 2.87 100.000 0 0 0 0 0.000 0.000 0.746
MimeEntry 48 177 9 9 14.07 0.005 177 9 9 14.07 100.000 0 0 0 0 0.000 0.000 0.000
Auth::Ntlm::User 92 88 8 19 3.12 0.005 88 8 19 3.12 100.000 0 0 0 0 0.000 0.000 2.592
Auth::Ntlm::UserRequest 44 174 8 17 0.41 0.005 174 8 17 0.41 100.000 0 0 0 0 0.000 0.000 2.592
cbdata AuthUserIP (33) 60 78 5 12 3.09 0.003 78 5 12 3.09 100.000 0 0 0 0 0.000 0.000 0.000
cbdata store_client (28) 96 29 3 13 4.50 0.002 29 3 13 4.50 100.000 0 0 0 0 0.000 0.000 12.378
acl_proxy_auth_match_cache 20 125 3 5 3.09 0.001 125 3 5 3.09 100.000 0 0 0 0 0.000 0.000 0.139
fqdncache_entry 80 28 3 3 0.69 0.001 28 3 3 0.69 100.000 0 0 0 0 0.000 0.000 0.000
cbdata HttpStateData (35) 144 15 3 9 1.31 0.001 15 3 9 1.31 100.000 0 0 0 0 0.000 0.000 6.423
Acl::AndNode 104 17 2 2 14.07 0.001 17 2 2 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata FwdState (34) 88 16 2 8 4.49 0.001 16 2 8 4.49 100.000 0 0 0 0 0.000 0.000 6.183
acl_ip_data 88 13 2 2 14.07 0.001 13 2 2 14.07 100.000 0 0 1 6 0.000 0.000 0.000
cbdata Logfile (11) 1080 1 2 2 14.07 0.001 1 2 2 14.07 100.000 0 0 0 0 0.000 0.000 0.000
AuthUserHashPointer 12 77 1 3 3.09 0.001 77 1 3 3.09 100.000 0 0 0 0 0.000 0.000 0.000
cbdata Tree (3) 136 5 1 1 14.07 0.000 5 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata UFSStoreState (39) 104 6 1 4 0.48 0.000 6 1 4 0.48 100.000 0 0 0 0 0.000 0.000 3.085
ACLStrategised 104 5 1 1 14.07 0.000 5 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (1) 28 18 1 1 14.07 0.000 18 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 104 4 1 1 14.07 0.000 4 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ev_entry 32 12 1 1 3.73 0.000 12 1 1 3.73 100.000 0 0 0 0 0.000 0.000 1.770
ACLSourceIP 96 4 1 1 14.07 0.000 4 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata BlockingFile (40) 52 6 1 2 0.48 0.000 6 1 2 0.48 100.000 0 0 0 0 0.000 0.000 3.085
Acl::NotNode 104 3 1 1 14.07 0.000 3 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
RegexList 28 8 1 1 14.07 0.000 8 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata PortCfg (4) 100 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLProxyAuth 100 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLDestinationIP 96 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata BodyPipe (29) 92 2 1 2 4.24 0.000 2 1 2 4.24 100.000 0 0 0 0 0.000 0.000 0.202
cbdata StateData (25) 28 6 1 2 4.41 0.000 6 1 2 4.41 100.000 0 0 0 0 0.000 0.000 5.058
cbdata CbDataList (36) 52 3 1 3 1.31 0.000 3 1 3 1.31 100.000 0 0 0 0 0.000 0.000 24.428
ACLRegexData 8 18 1 1 14.07 0.000 18 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata statefulhelper (9) 136 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata generic_cbdata (24) 20 6 1 2 0.53 0.000 6 1 2 0.53 100.000 0 0 0 0 0.000 0.000 4.337
cbdata RemovalPolicy (6) 56 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata TcpAcceptor (15) 56 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 104 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLTimeData 20 5 1 1 14.07 0.000 5 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
helper_stateful_request 16 6 1 1 4.41 0.000 6 1 1 4.41 100.000 0 0 0 0 0.000 0.000 5.058
cbdata ConnOpener (32) 76 1 1 4 4.25 0.000 1 1 4 4.25 100.000 0 0 0 0 0.000 0.000 5.677
cbdata StoreSearchHashIndex (16) 60 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata AclAddress (5) 52 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLUserData 12 4 1 1 14.07 0.000 4 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLHTTPHeaderData 24 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLDomainData 8 5 1 1 14.07 0.000 5 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (2) 36 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLStringData 8 4 1 1 14.07 0.000 4 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData 28 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
CacheDigest 24 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
wordlist 8 3 1 2 14.07 0.000 3 1 2 14.07 100.000 0 0 2 14 0.000 0.000 0.000
ACLNoteData 20 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLASN 8 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLMethodData 8 2 1 1 14.07 0.000 2 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
ACLProtocolData 8 1 1 1 14.07 0.000 1 1 1 14.07 100.000 0 0 0 0 0.000 0.000 0.000
UFSStoreState::_queued_write 20 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 24.731
cbdata WriteRequest (41) 48 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 24.731
cbdata ReadRequest (42) 44 0 0 1 14.03 0.000 0 0 1 14.03 -1.000 0 0 0 0 0.000 0.000 0.215
StoreMetaVary 16 0 0 1 14.04 0.000 0 0 1 14.04 -1.000 0 0 0 0 0.000 0.000 0.101
HttpHdrRange 20 0 0 1 1.47 0.000 0 0 1 1.47 -1.000 0 0 0 0 0.000 0.000 0.000
HttpHdrRangeSpec 16 0 0 1 1.47 0.000 0 0 1 1.47 -1.000 0 0 0 0 0.000 0.000 0.000
cbdata RemovalPolicyWalker (43) 32 0 0 1 9.57 0.000 0 0 1 9.57 -1.000 0 0 0 0 0.000 0.000 0.000
HttpHdrSc 8 0 0 1 3.52 0.000 0 0 1 3.52 -1.000 0 0 0 0 0.000 0.000 0.000
4KB Strings 4096 0 0 136 2.22 0.000 0 0 136 2.22 -1.000 0 0 0 0 0.000 0.000 0.164
HttpHdrScTarget 48 0 0 1 3.52 0.000 0 0 1 3.52 -1.000 0 0 0 0 0.000 0.000 0.000
HttpHdrContRange 24 0 0 1 1.31 0.000 0 0 1 1.31 -1.000 0 0 0 0 0.000 0.000 0.000
FwdServer 12 0 0 1 5.03 0.000 0 0 1 5.03 -1.000 0 0 0 0 0.000 0.000 6.929
StoreSwapLogData 68 0 0 1 14.07 0.000 0 0 1 14.07 -1.000 0 0 0 0 0.000 0.000 5.803
dwrite_q 28 0 0 1 14.07 0.000 0 0 1 14.07 -1.000 0 0 0 0 0.000 0.000 30.535
dread_ctrl 32 0 0 1 14.03 0.000 0 0 1 14.03 -1.000 0 0 0 0 0.000 0.000 0.215
dlink_node 12 0 0 1 14.07 0.000 0 0 1 14.07 -1.000 0 0 0 0 0.000 0.000 0.000
link_list 8 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 24.731
64K Buffer 65536 0 0 320 1.44 0.000 0 0 320 1.44 -1.000 0 0 0 0 0.000 0.000 0.000
32K Buffer 32768 0 0 64 3.77 0.000 0 0 64 3.77 -1.000 0 0 0 0 0.000 0.000 0.000
16KB Strings 16384 0 0 64 6.04 0.000 0 0 64 6.04 -1.000 0 0 0 0 0.000 0.000 0.000
cbdata ErrorState (27) 140 0 0 2 4.27 0.000 0 0 2 4.27 -1.000 0 0 0 0 0.000 0.000 5.968
cbdata RebuildState (12) 636 0 0 1 14.07 0.000 0 0 1 14.07 -1.000 0 0 0 0 0.000 0.000 0.000
cbdata ps_state (31) 176 0 0 1 5.03 0.000 0 0 1 5.03 -1.000 0 0 0 0 0.000 0.000 6.929
cbdata idns_query (8) 8604 0 0 622 0.53 0.000 0 0 622 0.53 -1.000 0 0 0 0 0.000 0.000 1.353
cbdata RemovalPurgeWalker (37) 44 0 0 1 14.07 0.000 0 0 1 14.07 -1.000 0 0 0 0 0.000 0.000 0.923
StoreMetaMD5 16 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 3.072
StoreMetaSTDLFS 16 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 3.072
StoreMetaURL 16 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 3.072
StoreMetaObjSize 16 0 0 1 14.06 0.000 0 0 1 14.06 -1.000 0 0 0 0 0.000 0.000 3.072
Total 1 658258 165521 178358 0.53 100.000 658258 165521 178358 0.53 100.000 0 0 2 39 0.000 0.000 5872366.497
Cumulative allocated volume: 55.251 GB
Current overhead: 20695 bytes (0.012%)
Idle pool limit: 0.00 MB
Total Pools created: 122
Pools ever used: 113 (shown above)
Currently in use: 84
String Pool Impact
(%strings) (%volume)
Short Strings 69 39
Medium Strings 31 61
Long Strings 0 0
1KB Strings 0 0
4KB Strings 0 0
16KB Strings 0 0
Other Strings 0 0
Large buffers: 0 (0 KB)
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:sudakov at sibptus.tomsk.ru
More information about the squid-users
mailing list