[squid-users] slow TCP_TUNNEL [SOLVED]

Thu Jul 28 05:20:11 UTC 2022

Great!

I’m happy you were able to resolve the issue easily.

All The Bests,
Eliezer

----
Eliezer Croitoru
NgTech, Tech Support
Mobile: +972-5-28704261
Email:  <mailto:ngtech1ltd at gmail.com> ngtech1ltd at gmail.com
Web:  <https://ngtech.co.il/> https://ngtech.co.il/
My-Tube:  <https://tube.ngtech.co.il/> https://tube.ngtech.co.il/

From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Katerina Bubenickova
Sent: Wednesday, 27 July 2022 10:22
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] slow TCP_TUNNEL [SOLVED]

Hey Eliezer,

Thank you. Now I am ashamed for I have sent unintentionally [solved] answer only to Alex yesterday.

I tried wireshark, disabled client persistent connections.
but the problem was system run out of filedescriptors.
4096 was not enough . I didn't notice it in the cache.log before

So I put into 
/etc/security/limits.conf

line 
* - nofile 65535

and into squid.conf:
max_filedescriptors 65535

systemctl restart squid
Now it is working all right.

Thank you for your effort,
now see I have a lot to learn.
I have to figure out the difference between  forward and intercept proxy,
what the pinger daemon is and what workers are.

Two proxies makes faster performance for us, 
On Debian 11 there is version Version 4.13
load balance is done by bind - two IPs have the name in A record

I will remove the `dns_v4_first` 
Thank you all again for your support,

Katerina

>>> <ngtech1ltd at gmail.com <mailto:ngtech1ltd at gmail.com> > 07/26/22 9:40 PM >>>
Hey Katerina,

Let’s try to understand the issue first.
The CentOS 6 squid version is what exactly? (squid -v )
What do you require from this proxy to do else then squidguard?
>From what I understand it’s a simple forward proxy and not an intercept proxy.
It means that all DNS resolution is done on the proxy.
What localhost DNS daemon are you using? (Bind? unbound? other?)
What version of Squid are you running on the Debian 11 machine (squid -v)

The `dns_v4_first` configuration is obsolete.
The first thing to check is that if the pinger is up and running.
>From what I remember Debian didn’t liked the pinger daemon.
Are you using workers in your configuration?
If you can share your current squid version and squid.conf we might be able to help you try to understand what’s going on.
The latest version of squid I have published for CentOS 6 was: squid-3.5.28-1.el6.x86_64.rpm

For a simple forward proxy you won’t need too much hardware but if you won’t be using workers this would be the expected behavior.
I believe that if you do not require special configurations we can compare another simple forward proxy just to make sure that
the core of the issue is something with squid.
(it’s possible to find the issue without using a comparison if enough details would be published)

How exactly is the proxy configured on the clients side?
Also is there any form of authentication happening or plain connections?
How do you load balance between the proxies?

>From my experience 700 clients are OK on a single proxy however…
only if the load is not too high in the network level.
It can be that the maximum limit of open file descriptors is reached or couple other reasons.
Can you share couple pages of your cache manager output?
The next script:
https://gist.githubusercontent.com/elico/8790bdc835d8e9ecbc57e72fc31effc0/raw/60d140b0e772fa4f418779bfc27e4804a345ce23/dump-cache-mgr-to-file.sh

should dump all pages to stdout but don’t send the output on the public list please.
I will try to analyze the pages and see if there is something specific I can understand from them.
If the above output would not be sufficient I have another more in-depth script that should help us
to understand what is the core issue with this VM.

I am waiting for more details so I would be able to try and help you.

Eliezer

----
Eliezer Croitoru
NgTech, Tech Support
Mobile: +972-5-28704261
Email: mailto:ngtech1ltd at gmail.com
Web: https://ngtech.co.il/
My-Tube: https://tube.ngtech.co.il/

From: squid-users <squid-users-bounces at lists.squid-cache.org <mailto:squid-users-bounces at lists.squid-cache.org> > On Behalf Of Katerina Bubenickova
Sent: Monday, 25 July 2022 11:40
To: squid-users at lists.squid-cache.org <mailto:squid-users at lists.squid-cache.org> 
Subject: [squid-users] slow TCP_TUNNEL

Hi,
We have 2 squid proxies running on Centos 6 which is very old (let's call them C1, C2) in DMZ.

I have installed Debian 11 bullseye and squid +squidguard, trying to use the same configuration (let's call it D1).
If I use this proxy for 1 pc station, all is ok.
If Iadded the proxy to dns as third proxy (C1+C2 + D1) or use one old proxy and the new one (C1+D1) the response of internet was very slow, unusable.

I tryed to fix it without success:
I added directive url_rewrite_children 200
I changed DNS from 8.8.8.8 to localhost
I turned off squid cache
I commented out squidguard
I migrated from one vmware server to another, better,
I added memory (16 GB) and CPU (6) ,
I added directive dns_v4_first on
There are no rules in D1 firewall

We have about 700 pc and the problem starts after 15-20 minutes after I add D1 into dns as proxy.
I can see it on our monitor Flowmon, where I can see value NPM response time. In normal state it is about 0.1s, when the problem arises it is up to 3s.
If I remove D1 from dns, all is ok after a while.

I tried to set IP of D1 the same as IP of C2 (of course C2 was power off) to test, whether the problem could be caused by some firewall between user pc and proxy, it doesn.t help

access.log of D1
1658483765.546 1622444 172.19.11.101 TCP_TUNNEL/200 3635 CONNECT epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483765.546 900188 172.19.14.73 TCP_TUNNEL/200 4738 CONNECT prg.smartadserver.com:443 - HIER_DIRECT/185.86.138.16 -
1658483765.546 900246 172.19.14.73 TCP_TUNNEL/200 4102 CONNECT bidder.criteo.com:443 - HIER_DIRECT/178.250.0.165 -
1658483765.546 1008081 172.19.13.171 TCP_TUNNEL/200 12749428 CONNECT rr1---sn-jxnoxu-2gbl.googlevideo.com:443 - HIER_DIRECT/195.113.214.204 -
1658483765.546 901791 172.19.14.95 TCP_TUNNEL/200 2176 CONNECT pcb.kb.cz:443 - HIER_DIRECT/194.50.226.158 -
1658483765.546 900569 172.19.14.73 TCP_TUNNEL/200 7260 CONNECT adx.adform.net:443 - HIER_DIRECT/37.157.2.237 -
1658483766.547 935767 172.19.13.190 TCP_TUNNEL/200 110961 CONNECT http://www.seznam.cz:443 - HIER_DIRECT/77.75.77.222 -
1658483766.547 1620846 172.19.15.40 TCP_TUNNEL/200 3520 CONNECT epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483766.547 900183 172.19.14.73 TCP_TUNNEL/200 4296 CONNECT etarget-emea.adnxs.com:443 - HIER_DIRECT/37.252.173.214 -
1658483767.548 1028965 172.19.15.2 TCP_TUNNEL/200 20796 CONNECT http://www.seznam.cz:443 - HIER_DIRECT/77.75.79.222 -
1658483767.548 959986 172.19.15.199 TCP_TUNNEL/200 7269 CONNECT fonts.googleapis.com:443 - HIER_DIRECT/142.251.36.106 -
1658483767.548 900027 172.19.14.73 TCP_TUNNEL/200 10544 CONNECT fastlane.rubiconproject.com:443 - HIER_DIRECT/69.173.144.141 -
1658483768.549 1625800 172.19.14.204 TCP_TUNNEL/200 3635 CONNECT epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483768.549 903754 172.19.14.73 TCP_TUNNEL/200 5891 CONNECT region1.google-analytics.com:443 - HIER_DIRECT/216.239.34.36 -
1658483769.551 1017091 172.19.14.19 TCP_TUNNEL/200 4702 CONNECT secure-it.imrworldwide.com:443 - HIER_DIRECT/54.246.16.130 -
1658483769.551 976878 172.19.14.69 TCP_TUNNEL/200 1120200 CONNECT rr2---sn-jxnoxu-2gbl.googlevideo.com:443 - HIER_DIRECT/195.113.214.205 -
1658483769.551 1626332 172.19.15.189 TCP_TUNNEL/200 3635 CONNECT epns.eset.com:443 - HIER_DIRECT/91.228.167.192 -
1658483769.551 986881 172.19.14.69 TCP_TUNNEL/200 11983552 CONNECT rr5---sn-2gb7sn7r.googlevideo.com:443 - HIER_DIRECT/172.217.130.74 -

in access.log of C1 are the time numbers a lot lower:

1658491430.020 7 172.19.14.191 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/31/eid/108/lid/108 - DIRECT/91.228.166.45 application/octet-stream
1658491430.025 6 172.19.15.72 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/31/eid/108/lid/108 - DIRECT/91.228.166.45 application/octet-stream
1658491430.031 6 172.19.14.191 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/32/eid/117876/lid/117876 - DIRECT/91.228.166.45 application/octet-stream
1658491430.038 430 172.19.14.71 TCP_MISS/200 4906 CONNECT v10.events.data.microsoft.com:443 - DIRECT/20.42.65.90 -
1658491430.076 40 172.19.14.191 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/33/eid/10185/lid/10185 - DIRECT/91.228.166.45 application/octet-stream
1658491430.112 55 172.19.15.72 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/32/eid/117876/lid/117876 - DIRECT/91.228.166.45 application/octet-stream
1658491430.155 30017 172.19.14.186 TCP_MISS/200 62553 CONNECT armmf.adobe.com:443 - DIRECT/2.23.8.158 -
1658491430.180 43 172.19.14.191 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/34/eid/8576/lid/8576 - DIRECT/91.228.166.45 application/octet-stream
Tato e-mailová zpráva má pouze informativní charakter a vychází z podkladů, které byly odesílateli předány nebo zaslány v minulosti. Obsah této e-mailové zprávy, resp. jakékoliv případné právní jednání vyplývající z této e-mailové zprávy, se nepovažuje za návrh na uzavření smlouvy ve smyslu ustanovení § 1731 zákona č. 89/2012 Sb., občanského zákoníku, v platném znění (dále jen „občanský zákoník“) nebo akceptaci návrhu na uzavření smlouvy ve smyslu § 1740 občanského zákoníku, popř. potvrzení uzavření smluvního vztahu, není-li odesílatelem uvedeno výslovně jinak. Tato e-mailová zpráva je určena pouze pro osobní a důvěrné užití určenými adresáty, tedy osobou (osobami), které (kterým) je obsahově určena. Nejste-li osobou, které je tato zpráva určena, upozorňujeme Vás, že jakékoliv šíření či kopírování této e-mailové zprávy, či informace z ní, je zakázáno. Jestliže tuto e-mailovou zprávu obdržíte nedopatřením, prosíme, oznamte tuto skutečnost odesílateli a odstraňte zprávu samotnou i všechny její přílohy ze svého systému. 

1658491430.227 235 172.19.15.144 TCP_MISS/200 3793 CONNECT ts.eset.com:443 - DIRECT/91.228.166.148 -
1658491430.276 73 172.19.15.72 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/33/eid/10185/lid/10185 - DIRECT/91.228.166.45 application/octet-stream
1658491430.404 90 172.19.14.191 TCP_MISS/200 473 GET http://i5.c.eset.com/v1/auth/60064ADBC289880E8D77/updlist/35/eid/7351/lid/7351 - DIRECT/91.228.166.45 application/octet-stream

Now I am out of ideas what to test.

Thank you very much for your help,
Katerina

}

Tato e-mailová zpráva má pouze informativní charakter a vychází z podkladů, které byly odesílateli předány nebo zaslány v minulosti. Obsah této e-mailové zprávy, resp. jakékoliv případné právní jednání vyplývající z této e-mailové zprávy, se nepovažuje za návrh na uzavření smlouvy ve smyslu ustanovení § 1731 zákona č. 89/2012 Sb., občanského zákoníku, v platném znění (dále jen „občanský zákoník“) nebo akceptaci návrhu na uzavření smlouvy ve smyslu § 1740 občanského zákoníku, popř. potvrzení uzavření smluvního vztahu, není-li odesílatelem uvedeno výslovně jinak. Tato e-mailová zpráva je určena pouze pro osobní a důvěrné užití určenými adresáty, tedy osobou (osobami), které (kterým) je obsahově určena. Nejste-li osobou, které je tato zpráva určena, upozorňujeme Vás, že jakékoliv šíření či kopírování této e-mailové zprávy, či informace z ní, je zakázáno. Jestliže tuto e-mailovou zprávu obdržíte nedopatřením, prosíme, oznamte tuto skutečnost odesílateli a odstraňte zprávu samotnou i všechny její přílohy ze svého systému. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220728/74620c1c/attachment-0001.htm>