[squid-users] squid-users Digest, Vol 22, Issue 62
Nilesh Gavali
nilesh.gavali at tcs.com
Tue Jun 14 12:25:46 UTC 2016
Hello Antony;
I have setup like below :-
end user >> LinuxProxy(3.1.10) >> External Proxy(3.4)>> Internet
when we configure external proxy ip in end user's IE, all HTTP & HTTPS
site work properly.
when we configure Linux proxy ip in end user's IE, HTTP works fine but
none of the HTTPS site open.
when we access https site from LinuxProxy, below logs are appearing in
access.log
TCP_DENIED/407 CONNECT sitename:443 - NONE/ text/html
TCP_MISS/503 0 CONNECT sitename:443 username at MYDOMAIN.COM DIRECT /
- -
TCP_MISS/200 23456 GET http://www.anysite.com
username at MYDOMAIN.COM DEFAULT_PARENT/10.10.x.x text/html
what I making out from log is ( I might be wrong) - HTTPS request are
going directly instead .
attached is my Linux Proxy config-
=================
#
# Recommended minimum configuration:
#
#auth_param negotiate program /usr/lib64/squid/squid_kerb_auth -d -s
GSS_C_NO_NAME
auth_param negotiate program /usr/lib64/squid/squid_kerb_auth -s
HTTP/proxy02.abcd.com at ABCD.COM -d
auth_param negotiate children 10
auth_param negotiate keep_alive on
auth_param basic credentialsttl 2 hours
acl ad_auth proxy_auth REQUIRED
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
#
# Recommended minimum Access Permission configuration:
#
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
# Deny requests to certain unsafe ports
http_access deny !Safe_ports
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
#
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
#http_access allow localnet
#http_access allow localhost
http_access deny !ad_auth
http_access allow ad_auth
# And finally deny all other access to this proxy
http_access deny all
# Squid normally listens to port 3128
http_port 8080
cache_peer xx.xx.2.108 parent 8080 0 default
#dns_nameservers ABCDNS.ABCD.COM
dns_nameservers xx.xx.2.108
# We recommend you to use at least the following line.
#hierarchy_stoplist cgi-bin ?
# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/spool/squid 2048 16 256
# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid
# Log forwarding to SysLog
access_log syslog:local1.info
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
-======================================
Thanks & Regards
Nilesh Suresh Gavali
From: squid-users-request at lists.squid-cache.org
To: squid-users at lists.squid-cache.org
Date: 14/06/2016 13:00
Subject: squid-users Digest, Vol 22, Issue 62
Sent by: "squid-users" <squid-users-bounces at lists.squid-cache.org>
Send squid-users mailing list submissions to
squid-users at lists.squid-cache.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.squid-cache.org/listinfo/squid-users
or, via email, send a message with subject or body 'help' to
squid-users-request at lists.squid-cache.org
You can reach the person managing the list at
squid-users-owner at lists.squid-cache.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of squid-users digest..."
Today's Topics:
1. Re: Excessive TCP memory usage (Eliezer Croitoru)
2. Re: Squid not allowing HTTPS access (Nilesh Gavali)
3. Re: Squid not allowing HTTPS access (Antony Stone)
----------------------------------------------------------------------
Message: 1
Date: Tue, 14 Jun 2016 13:21:32 +0300
From: "Eliezer Croitoru" <eliezer at ngtech.co.il>
To: "'Deniz Eren'" <denizlist at denizeren.net>,
<squid-users at lists.squid-cache.org>
Subject: Re: [squid-users] Excessive TCP memory usage
Message-ID: <05c801d1c626$85a97ff0$90fc7fd0$@ngtech.co.il>
Content-Type: text/plain; charset="utf-8"
Hey,
Steps to reproduce are not exactly everything since squid works fine in
many other scenarios.
I do not know this specific system but if you are talking about 1-4k open
connections it should not be a big problem for many servers.
The issue in hands is a bit different.
Have you tried tuning the ipv4\net using sysctl to see if it affects
anything?
What I can offer is to build a tiny ICAP service that will use a 204 on
every request and then moves on.
If the same happens with the dummy service it's probably a very bad
scenario and if not then we can try to think
if there is something unique about your setup.
I have not seen this issue in my current testing setup which includes
3.5.19 + ICAP url filtering service.
Eliezer
----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il
-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
Behalf Of Deniz Eren
Sent: Tuesday, June 14, 2016 11:07 AM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Excessive TCP memory usage
Little bump :)
I have posted bug report with steps to reproduce. The problem still exists
and I am curious whether anyone else is having the same problem, too.
http://bugs.squid-cache.org/show_bug.cgi?id=4526
On Wed, May 25, 2016 at 1:18 PM, Deniz Eren <denizlist at denizeren.net>
wrote:
> When I listen to connections between squid and icap using tcpdump I
> saw that after a while icap closes the connection but squid does not
> close, so connection stays in CLOSE_WAIT state:
>
> [root at test ~]# tcpdump -i any -n port 34693
> tcpdump: WARNING: Promiscuous mode not supported on the "any" device
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode listening on any, link-type LINUX_SLL (Linux cooked), capture
> size 96 bytes
> 13:07:31.802238 IP 127.0.0.1.icap > 127.0.0.1.34693: F
> 2207817997:2207817997(0) ack 710772005 win 395 <nop,nop,timestamp
> 104616992 104016968>
> 13:07:31.842186 IP 127.0.0.1.34693 > 127.0.0.1.icap: . ack 1 win 3186
> <nop,nop,timestamp 104617032 104616992>
>
> [root at test ~]# netstat -tulnap|grep 34693
> tcp 215688 0 127.0.0.1:34693 127.0.0.1:1344
> CLOSE_WAIT 19740/(squid-1)
>
> These CLOSE_WAIT connections do not timeout and stay until squid
> process is killed.
>
> 2016-05-25 10:37 GMT+03:00 Deniz Eren <denizlist at denizeren.net>:
>> 2016-05-24 21:47 GMT+03:00 Amos Jeffries <squid3 at treenet.co.nz>:
>>> On 25/05/2016 5:50 a.m., Deniz Eren wrote:
>>>> Hi,
>>>>
>>>> After upgrading to squid 3.5.16 I realized that squid started using
>>>> much of kernel's TCP memory.
>>>
>>> Upgrade from which version?
>>>
>> Upgrading from squid 3.1.14. I started using c-icap and ssl-bump.
>>
>>>>
>>>> When squid was running for a long time TCP memory usage is like
below:
>>>> test at test:~$ cat /proc/net/sockstat
>>>> sockets: used *
>>>> TCP: inuse * orphan * tw * alloc * mem 200000
>>>> UDP: inuse * mem *
>>>> UDPLITE: inuse *
>>>> RAW: inuse *
>>>> FRAG: inuse * memory *
>>>>
>>>> When I restart squid the memory usage drops dramatically:
>>>
>>> Of course it does. By restarting you just erased all of the
>>> operational state for an unknown but large number of active network
connections.
>>>
>> That's true but what I mean was squid's CLOSE_WAIT connections are
>> using too much memory and they are not timing out.
>>
>>> Whether many of those should have been still active or not is a
>>> different question. the answer to which depends on how you have your
>>> Squid configured, and what the traffic through it has been doing.
>>>
>>>
>>>> test at test:~$ cat /proc/net/sockstat
>>>> sockets: used *
>>>> TCP: inuse * orphan * tw * alloc * mem 10
>>>> UDP: inuse * mem *
>>>> UDPLITE: inuse *
>>>> RAW: inuse *
>>>> FRAG: inuse * memory *
>>>>
>>>
>>> The numbers you replaced with "*" are rather important for context.
>>>
>>>
>> Today again I saw the problem:
>>
>> test at test:~$ cat /proc/net/sockstat
>> sockets: used 1304
>> TCP: inuse 876 orphan 81 tw 17 alloc 906 mem 29726
>> UDP: inuse 17 mem 8
>> UDPLITE: inuse 0
>> RAW: inuse 1
>> FRAG: inuse 0 memory 0
>>
>>>> I'm using Squid 3.5.16.
>>>>
>>>
>>> Please upgrade to 3.5.19. Some important issues have been resolved.
>>> Some of them may be related to your TCP memory problem.
>>>
>>>
>> I have upgraded now and problem still exists.
>>
>>>> When I look with "netstat" and "ss" I see lots of CLOSE_WAIT
>>>> connections from squid to ICAP or from squid to upstream server.
>>>>
>>>> Do you have any idea about this problem?
>>>
>>> Memory use by the TCP system of your kernel has very little to do
>>> with Squid. Number of sockets in CLOSE_WAIT does have some relation
>>> to Squid or at least to how the traffic going through it is handled.
>>>
>>> If you have disabled persistent connections in squid.conf then lots
>>> of closed sockets and FD are to be expected.
>>>
>>> If you have persistent connections enabled, then fewer closures
>>> should happen. But some will so expectations depends on how high the
>>> traffic load is.
>>>
>> Persistent connection parameters are enabled in my conf, the problem
>> occurs especially with connections to c-icap service.
>>
>> My netstat output is like this:
>> netstat -tulnap|grep squid|grep CLOSE
>>
>> tcp 211742 0 127.0.0.1:55751 127.0.0.1:1344
>> CLOSE_WAIT 17076/(squid-1)
>> tcp 215700 0 127.0.0.1:55679 127.0.0.1:1344
>> CLOSE_WAIT 17076/(squid-1)
>> tcp 215704 0 127.0.0.1:55683 127.0.0.1:1344
>> CLOSE_WAIT 17076/(squid-1)
>> ...(hundreds)
>> Above ones are connections to c-icap service.
>>
>> netstat -tulnap|grep squid|grep CLOSE Active Internet connections
>> (servers and established)
>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State PID/Program name
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:45182
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.2.177:50020
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.2.172:60028
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:44049
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:55054
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.2.137:52177
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:43542
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.155:39489
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.147:38939
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:38754
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.164:39602
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.147:54114
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:57857
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.156:43482
>> CLOSE_WAIT 15245/(squid-1)
>> ...(about 50)
>> Above ones are connections from https port to client.
>>
>> As you can see recv-q for icap connections allocate more memory but
>> connections from https_port to upstream server connections allocate
>> only one byte.
>>
>> What can be done to close these unused connections?
>>
>> The problem in this thread seems similar:
>> http://www.squid-cache.org/mail-archive/squid-users/201301/0092.html
>>
>>> Amos
>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
------------------------------
Message: 2
Date: Tue, 14 Jun 2016 11:36:30 +0100
From: Nilesh Gavali <nilesh.gavali at tcs.com>
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Squid not allowing HTTPS access
Message-ID:
<OF83BE7BF7.92EBA289-ON80257FD2.003A3940-80257FD2.003A455E at tcs.com>
Content-Type: text/plain; charset="utf-8"
Team;
kindly help on below issue.
Thanks & Regards
Nilesh Suresh Gavali
From: Nilesh Gavali/MUM/TCS
To: squid-users at lists.squid-cache.org
Date: 13/06/2016 14:00
Subject: Squid not allowing HTTPS access
Hello All;
Facing issue while accessing HTTPS via squid, normal http traffic working
fine. I have squid 3.1.10 on RHEL.6.0
attached is my squid .conf for your reference,..help will be much
appreciated.
Thanks & Regards
Nilesh Suresh Gavali
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.squid-cache.org/pipermail/squid-users/attachments/20160614/79185c58/attachment-0001.html
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: squid.conf
Type: application/octet-stream
Size: 3149 bytes
Desc: not available
URL: <
http://lists.squid-cache.org/pipermail/squid-users/attachments/20160614/79185c58/attachment-0001.obj
>
------------------------------
Message: 3
Date: Tue, 14 Jun 2016 12:40:14 +0200
From: Antony Stone <Antony.Stone at squid.open.source.it>
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Squid not allowing HTTPS access
Message-ID: <201606141240.15034.Antony.Stone at squid.open.source.it>
Content-Type: Text/Plain; charset="iso-8859-15"
On Tuesday 14 June 2016 at 12:36:30, Nilesh Gavali wrote:
> Team;
> kindly help on below issue.
Nilesh:
kindly respond to my reply below.
On Monday 13 June 2016 at 15:26:22, Antony Stone wrote:
> On Monday 13 June 2016 at 15:01:02, Nilesh Gavali wrote:
> > Facing issue while accessing HTTPS via squid, normal http traffic
working
> > fine.
>
> Please define "issue", with as much detail as possible:
>
> - what exactly are you trying to do when a problem occurs?
>
> - have you previously been able to do this without the problem
occurring?
> If so, what has changed between then and now (different squid config,
> different squid version, different browser...)?
>
> - what is the actual problem (what error message is displayed, if there
is
> one)?
>
> - what appears in your access.log when the problem occurs?
>
> - any other information you think might be relevant to us in working
out
> what's happening on your network?
>
> > I have squid 3.1.10 on RHEL.6.0
>
> For HTTPS traffic in particular you are strongly advised to upgrade.
>
>
> Regards,
>
> Antony.
--
All generalisations are inaccurate.
Please reply to the
list;
please *don't* CC
me.
------------------------------
Subject: Digest Footer
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
------------------------------
End of squid-users Digest, Vol 22, Issue 62
*******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160614/91af202c/attachment-0001.html>
More information about the squid-users
mailing list