[squid-users] slow TCP_TUNNEL [SOLVED]

Wed Jul 27 07:22:19 UTC 2022

Hey Eliezer,

Thank you. Now I am ashamed for I have sent unintentionally [solved]
answer only to Alex yesterday.

I tried wireshark, disabled client persistent connections.
but the problem was system run out of filedescriptors.
4096 was not enough . I didn't notice it in the cache.log before

So I put into 

/etc/security/limits.conf

line 

* - nofile 65535

and into squid.conf:

max_filedescriptors 65535

systemctl restart squid

Now it is working all right.

Thank you for your effort,
now see I have a lot to learn.

I have to figure out the difference between  forward and intercept
proxy,
what the pinger daemon is and what workers are.

Two proxies makes faster performance for us, 

On Debian 11 there is version Version 4.13
load balance is done by bind - two IPs have the name in A record

I will remove the  `dns_v4_first` 

Thank you all again for your support,

Katerina

>>> <ngtech1ltd at gmail.com> 07/26/22 9:40 PM >>>
Hey Katerina,

Let’s try to understand the issue first.
The CentOS 6 squid version is what exactly? (squid -v )
What do you require from this proxy to do else then squidguard?
>From what I understand it’s a simple forward proxy and not an intercept
proxy.
It means that all DNS resolution is done on the proxy.
What localhost DNS daemon are you using? (Bind? unbound? other?)
What version of Squid are you running on the Debian 11 machine (squid
-v)

The `dns_v4_first` configuration is obsolete.
The first thing to check is that if the pinger is up and running.
>From what I remember Debian didn’t liked the pinger daemon.
Are you using workers in your configuration?
If you can share your current squid version and squid.conf we might be
able to help you try to understand what’s going on.
The latest version of squid I have published for CentOS 6 was:
squid-3.5.28-1.el6.x86_64.rpm

For a simple forward proxy you won’t need too much hardware but if you
won’t be using workers this would be the expected behavior.
I believe that if you do not require special configurations we can
compare another simple forward proxy just to make sure that
the core of the issue is something with squid.
(it’s possible to find the issue without using a comparison if enough
details would be published)

How exactly is the proxy configured on the clients side?
Also is there any form of authentication happening or plain connections?
How do you load balance between the proxies?

>From my experience 700 clients are OK on a single proxy however…
only if the load is not too high in the network level.
It can be that the maximum limit of open file descriptors is reached or
couple other reasons.
Can you share couple pages of your cache manager output?
The next script:
https://gist.githubusercontent.com/elico/8790bdc835d8e9ecbc57e72fc31effc0/raw/60d140b0e772fa4f418779bfc27e4804a345ce23/dump-cache-mgr-to-file.sh

should dump all pages to stdout but don’t send the output on the public
list please.
I will try to analyze the pages and see if there is something specific I
can understand from them.
If the above output would not be sufficient I have another more in-depth
script that should help us
to understand what is the core issue with this VM.

I am waiting for more details so I would be able to try and help you.

Eliezer

----
Eliezer Croitoru
NgTech, Tech Support
Mobile: +972-5-28704261
Email: mailto:ngtech1ltd at gmail.com
Web: https://ngtech.co.il/
My-Tube: https://tube.ngtech.co.il/

From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf
Of Katerina Bubenickova
Sent: Monday, 25 July 2022 11:40
To: squid-users at lists.squid-cache.org
Subject: [squid-users] slow TCP_TUNNEL

Hi,
We have 2 squid proxies running on Centos 6 which is very old (let's
call them C1, C2) in DMZ.

I have installed Debian 11 bullseye and squid +squidguard, trying to use
the same configuration (let's call it D1).
If I use this proxy for 1 pc station, all is ok.
If Iadded the proxy toI tryed to fix it without success:
I added directive url_rewrite_children 200
I changed DNS from 8.8.8.8 to localhost
I turned off squid cache
I commented out squidguard
I migrated from one vmware server to another, better,
I added memory (16 GB) and CPU (6) ,
I added directive dns_v4_first on
There are no rules in D1 firewall

We have about 700 pc and the problem starts after 15-20 minutes after I
add D1 into dns as proxy.
I can see it on our monitor Flowmon, where I can see value NPM response
time. In normal state it is about 0.1s, when the problem arises it is up
to 3s.
If I remove D1 from dns, all is ok after a while.

I  tried to set IP of D1 the same as IP of C2 (of course C2 was power
off) to test, whether the problem could be caused by some firewall
between user pc and proxy, it doesn.t help

access.log of D1
1658483765.546 1622444 172.19.11.101 TCP_TUNNEL/200 3635 CONNECT
epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483765.546 900188 172.19.14.73 TCP_TUNNEL/200 4738 CONNECT
prg.smartadserver.com:443 - HIER_DIRECT/185.86.138.16 -
1658483765.546 900246 172.19.14.73 TCP_TUNNEL/200 4102 CONNECT
bidder.criteo.com:443 - HIER_DIRECT/178.250.0.165 -
1658483765.546 1008081 172.19.13.171 TCP_TUNNEL/200 12749428 CONNECT
rr1---sn-jxnoxu-2gbl.googlevideo.com:443 - HIER_DIRECT/195.113.214.204 -
1658483765.546 901791 172.19.14.95 TCP_TUNNEL/200 2176 CONNECT
pcb.kb.cz:443 - HIER_DIRECT/194.50.226.158 -
1658483765.546 900569 172.19.14.73 TCP_TUNNEL/200 7260 CONNECT
adx.adform.net:443 - HIER_DIRECT/37.157.2.237 -
1658483766.547 935767 172.19.13.190 TCP_TUNNEL/200 110961 CONNECT
http://www.seznam.cz:443 - HIER_DIRECT/77.75.77.222 -
1658483766.547 1620846 172.19.15.40 TCP_TUNNEL/200 3520 CONNECT
epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483766.547 900183 172.19.14.73 TCP_TUNNEL/200 4296 CONNECT
etarget-emea.adnxs.com:443 - HIER_DIRECT/37.252.173.214 -
1658483767.548 1028965 172.19.15.2 TCP_TUNNEL/200 20796 CONNECT
http://www.seznam.cz:443 - HIER_DIRECT/77.75.79.222 -
1658483767.548 959986 172.19.15.199 TCP_TUNNEL/200 7269 CONNECT
fonts.googleapis.com:443 - HIER_DIRECT/142.251.36.106 -
1658483767.548 900027 172.19.14.73 TCP_TUNNEL/200 10544 CONNECT
fastlane.rubiconproject.com:443 - HIER_DIRECT/69.173.144.141 -
1658483768.549 1625800 172.19.14.204 TCP_TUNNEL/200 3635 CONNECT
epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483768.549 903754 172.19.14.73 TCP_TUNNEL/200 5891 CONNECT
region1.google-analytics.com:443 - HIER_DIRECT/216.239.34.36 -
1658483769.551 1017091 172.19.14.19 TCP_TUNNEL/200 4702 CONNECT
secure-it.imrworldwide.com:443 - HIER_DIRECT/54.246.16.130 -
1658483769.551 976878 172.19.14.69 TCP_TUNNEL/200 1120200 CONNECT
rr2---sn-jxnoxu-2gbl.googlevideo.com:443 - HIER_DIRECT/195.113.214.205 -
1658483769.551 1626332 172.19.15.189 TCP_TUNNEL/200 3635 CONNECT
epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483769.551 986881 172.19.14.69 TCP_TUNNEL/200 11983552 CONNECT
rr5---sn-2gb7sn7r.googlevideo.com:443 - HIER_DIRECT/172.217.130.74 -

in access.log of C1 are the time numbers a lot lower:

1658491430.020      7 172.19.14.191 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/31/eid/108/lid/108
- DIRECT/91.228.166.45 application/octet-stream
1658491430.025      6 172.19.15.72 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/31/eid/108/lid/108
- DIRECT/91.228.166.45 application/octet-stream
1658491430.031      6 172.19.14.191 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/32/eid/117876/lid/117876
- DIRECT/91.228.166.45 application/octet-stream
1658491430.038    430 172.19.14.71 TCP_MISS/200 4906 CONNECT
v10.events.data.microsoft.com:443 - DIRECT/20.42.65.90 -
1658491430.076     40 172.19.14.191 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/33/eid/10185/lid/10185
- DIRECT/91.228.166.45 application/octet-stream
1658491430.112     55 172.19.15.72 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADB- DIRECT/91.228.166.45 application/octet-stream
1658491430.155  30017 172.19.14.186 TCP_MISS/200 62553 CONNECT
armmf.adobe.com:443 - DIRECT/2.23.8.158 -
1658491430.180     43 172.19.14.191 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/34/eid/8576/lid/8576
- DIRECT/91.228.166.45 application/octet-stream
Tato e-mailová zpráva má pouze informativní charakter a vychází z 
podkladů, které byly odesílateli předány nebo zaslány v minulosti. Obsah
 této e-mailové zprávy, resp. jakékoliv případné právní jednání 
vyplývající z této e-mailové zprávy, se nepovažuje za návrh na uzavření 
smlouvy ve smyslu ustanovení § 1731 zákona č. 89/2012 Sb., občanského 
zákoníku, v platném znění (dále jen „občanský zákoník“) nebo akceptaci 
návrhu na uzavření smlouvy ve smyslu § 1740 občanského zákoníku, popř. 
potvrzení uzavření smluvního vztahu, není-li odesílatelem uvedeno 
výslovně jinak. Tato e-mailová zpráva je určena pouze pro osobní a 
důvěrné užití určenými adresáty, tedy osobou (osobami), které (kterým) 
je obsahově určena. Nejste-li osobou, které je tato zpráva určena, 
upozorňujeme Vás, že jakékoliv šíření či kopírování této e-mailové 
zprávy, či informace z ní, je zakázáno. Jestliže tuto e-mailovou zprávu 
obdržíte nedopatřením, prosíme, oznamte tuto skutečnost odesílateli a 
odstraňte zprávu samotnou i všechny její přílohy ze svého systému. 

1658491430.227    235 172.19.15.144 TCP_MISS/200 3793 CONNECT
ts.eset.com:443 - DIRECT/91.228.166.148 -
1658491430.276     73 172.19.15.72 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/33/eid/10185/lid/10185
- DIRECT/91.228.166.45 application/octet-stream
1658491430.404     90 172.19.14.191 TCP_MISS/200 473 GET
http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/35/eid/7351/lid/7351
- DIRECT/91.228.166.45 application/octet-stream

Now I am out of ideas what to test.

Thank you very much for your help,
Katerina

}

Tato e-mailová zpráva má pouze informativní charakter a vychází z
podkladů, které byly odesílateli předány nebo zaslány v minulosti. Obsah
této e-mailové zprávy, resp. jakékoliv případné právní jednání
vyplývající z této e-mailové zprávy, se nepovažuje za návrh na
uzavření smlouvy ve smyslu ustanovení § 1731 zákona č. 89/2012 Sb.,
občanského zákoníku, v platném znění (dále jen „občanský zákoník“)
nebo akceptaci návrhu na uzavření smlouvy ve smyslu § 1740 občanského
zákoníku, popř. potvrzení uzavření smluvního vztahu, není-li
odesílatelem uvedeno výslovně jinak. Tato e-mailová zpráva je určena
pouze pro osobní a důvěrné užití určenými adresáty, tedy osobou
(osobami), které (kterým) je obsahově určena. Nejste-li osobou, které je
tato zpráva určena, upozorňujeme Vás, že jakékoliv šíření či kopírování
této e-mailové zprávy, či informace z ní, je zakázáno. Jestliže tuto
e-mailovou zprávu obdržíte nedopatřením, prosíme, oznamte tuto
skutečnost odesílateli a odstraňte zprávu samotnou i všechny její
přílohy ze svého systému.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220727/b10ebe84/attachment-0001.htm>