<div dir="rtl"><div dir="ltr">By the help of God.</div><div dir="ltr"><br></div><div dir="ltr">I think I can pinpoint the problem but I'm not sure, It looks like the issue is between c-icap and squid communication.</div><div dir="ltr">From squid logs I can see lots of traces: </div><div dir="ltr">2023/10/22 12:31:50.654 kid1| 93,3| ModXact.cc(556) readMore: not reading because ICAP reply pipe is full<br>2023/10/22 12:31:50.654 kid1| 93,3| ModXact.cc(556) readMore: not reading because ICAP reply pipe is full<br>2023/10/22 12:31:50.654 kid1| 93,3| AsyncJob.cc(161) callEnd: Adaptation::Icap::ModXact status out: [FD 319;rp(2)S(2)/RwP(ieof) job2845<br></div><div dir="ltr"><br></div><div dir="ltr">And on c-icap logs I can see that squid closing connection:</div><div dir="ltr">Sun Oct 22 12:38:07 2023, 709785/139749352670784, Error while sending rest responce or client closed the connection<br></div><div dir="ltr"><br></div><div dir="ltr">Those traces are coming from a relatively high debug level so I'm not sure if it's pointing to an issue but it is worth thinking about.</div><div dir="ltr">To me it sounds not good if c-icap needs to send back data to squid and while it's sending the session is closed.</div><div dir="ltr">What do you think?</div><div dir="ltr"><br></div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="rtl" class="gmail_attr">בתאריך שבת, 21 באוק׳ 2023 ב-22:35 מאת Ben Goz <<a href="mailto:ben.goz87@gmail.com" target="_blank">ben.goz87@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="rtl"><div dir="ltr">By the help of God.</div><div dir="ltr"><br></div><div dir="ltr">I'll clarify more when I'm looking at chrome's Internet Tools network traffic of one of the users that several</div><div dir="ltr">Requests are timed out and at the same time I see in Squid's access log a trace that according to your explanations</div><div dir="ltr">was opened for more that 60 secs for the timed out url at client side. This issue is observed only for certain urls while the other urls are working as expected.</div><div dir="ltr"><br></div><div dir="ltr">I asked the question very generally but I think I should say that the question is regarding TPROXY setup with ssl-bump</div><div dir="ltr">And eventually an icap service. The weird thing here is that for many other URLs this setup works fine.</div><div dir="ltr">So currently I don't have any idea how to work it out.</div><div dir="ltr"><br></div><div dir="ltr">If more information about my setup is needed please let me know.</div><div dir="ltr"><br></div><div dir="ltr">Thanks,</div><div dir="ltr">Ben</div><div dir="ltr"><br></div><div dir="ltr"><br></div></div><br><div class="gmail_quote"><div dir="rtl" class="gmail_attr">בתאריך יום ו׳, 20 באוק׳ 2023 ב-6:27 מאת Amos Jeffries <<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 19/10/23 01:21, Ben Goz wrote:<br>
> By the help of God.<br>
> <br>
> Hi,<br>
> I saw in my access log a traces that shows that spliced URLs tunneling <br>
> is very slowly:<br>
<br>
Please clarify what you mean by "slow" ?<br>
<br>
How have you determined speed ?<br>
What speed are you expecting / would you call non-slow ?<br>
<br>
<br>
<br>
FYI, Several things to be aware of:<br>
<br>
1) CONNECT tunnel is not a simple thing with a constant "speed" of <br>
transfer. It represents and entire set of tunneled messages (or other <br>
opaque data) over indefinite timespan. Each of those messages has its <br>
own "speed" of transfer, with possible empty periods of 0 bytes <br>
transferred between.<br>
<br>
2) the SSL-Bump procedure may pause a CONNECT tunnel during TLS <br>
handshake and/or validation process to asynchronously fetch missing <br>
certificate details, and/or validate other data with ACLs, etc.<br>
Each of these subsidiary transactions may add indefinite effects on <br>
timing of the 'bumped' CONNECT tunnel.<br>
<br>
3) modern networking systems utilize "Happy Eyeballs" algorithms <br>
wherein they may open multiple TCP connections to various (or same) <br>
services in parallel and only utilize the fastest to connect. This can <br>
result in CONNECT tunnel being initiated and unused - either closed <br>
immediately or left open waiting activity for long periods.<br>
<br>
<br>
So, as you should be able to see the log snippet shows some details <br>
about tunnels duration of use, you cannot tell "speed" from these logs.<br>
<br>
<br>
For example:<br>
> <br>
> 18/Oct/2023:15:18:50 +0300 240841 192.168.3.98 TCP_TUNNEL/200 6225 <br>
> CONNECT <a href="http://beacons2.gvt2.com:443" rel="noreferrer" target="_blank">beacons2.gvt2.com:443</a> <<a href="http://beacons2.gvt2.com:443" rel="noreferrer" target="_blank">http://beacons2.gvt2.com:443</a>> - <br>
> HIER_DIRECT/<a href="http://172.217.0.67" rel="noreferrer" target="_blank">172.217.0.67</a> <<a href="http://172.217.0.67" rel="noreferrer" target="_blank">http://172.217.0.67</a>> - <a href="http://beacons2.gvt2.com" rel="noreferrer" target="_blank">beacons2.gvt2.com</a> <br>
> <<a href="http://beacons2.gvt2.com" rel="noreferrer" target="_blank">http://beacons2.gvt2.com</a>> - splice -<br>
<br>
<br>
Tunnel was _open_ for 240 seconds. 6225 bytes transferred.<br>
<br>
Those bytes may have been transferred in the first 1 milliseconds of the <br>
tunnel being open. Then Squid leaving it open waiting for further uses <br>
which never came.<br>
<br>
... "slow" at 1.4 GB/sec.<br>
<br>
... or it could have been "slow" at 10 bytes/sec the whole time. One <br>
cannot tell.<br>
<br>
<br>
> 18/Oct/2023:15:18:50 +0300 680 192.168.3.173 TCP_TUNNEL/500 4977 <br>
> CONNECT <a href="http://mobile.events.data.microsoft.com:443" rel="noreferrer" target="_blank">mobile.events.data.microsoft.com:443</a> <br>
> <<a href="http://mobile.events.data.microsoft.com:443" rel="noreferrer" target="_blank">http://mobile.events.data.microsoft.com:443</a>> - HIER_DIRECT/<a href="http://13.89.178.26" rel="noreferrer" target="_blank">13.89.178.26</a> <br>
> <<a href="http://13.89.178.26" rel="noreferrer" target="_blank">http://13.89.178.26</a>> - <a href="http://mobile.events.data.microsoft.com" rel="noreferrer" target="_blank">mobile.events.data.microsoft.com</a> <br>
> <<a href="http://mobile.events.data.microsoft.com" rel="noreferrer" target="_blank">http://mobile.events.data.microsoft.com</a>> - splice -<br>
<br>
This tunnel was never open at all. It was *rejected*.<br>
"speed" in that case was 3 KB/sec.<br>
<br>
<br>
> 18/Oct/2023:15:18:51 +0300 127307 192.168.3.97 TCP_TUNNEL/500 3101 <br>
> CONNECT <a href="http://array612.prod.do.dsp.mp.microsoft.com:443" rel="noreferrer" target="_blank">array612.prod.do.dsp.mp.microsoft.com:443</a> <br>
> <<a href="http://array612.prod.do.dsp.mp.microsoft.com:443" rel="noreferrer" target="_blank">http://array612.prod.do.dsp.mp.microsoft.com:443</a>> - <br>
> HIER_DIRECT/<a href="http://20.54.24.148" rel="noreferrer" target="_blank">20.54.24.148</a> <<a href="http://20.54.24.148" rel="noreferrer" target="_blank">http://20.54.24.148</a>> - <br>
> <a href="http://array612.prod.do.dsp.mp.microsoft.com" rel="noreferrer" target="_blank">array612.prod.do.dsp.mp.microsoft.com</a> <br>
> <<a href="http://array612.prod.do.dsp.mp.microsoft.com" rel="noreferrer" target="_blank">http://array612.prod.do.dsp.mp.microsoft.com</a>> - splice -<br>
<br>
Tunnel was _open_ for 127 seconds. 3101 bytes transferred.<br>
<br>
Those bytes may have been transferred in the first 1 milliseconds of the <br>
tunnel being open. Then Squid leaving it open waiting for further uses <br>
which never came.<br>
<br>
... "slow" at 376 MB/sec.<br>
<br>
... or it could have been "slow" at 25 bytes/sec the whole time. One <br>
cannot tell.<br>
<br>
<br>
<br>
> This is my squid configurations:<br>
> <br>
> acl NoSSLInterceptRegexp_always ssl::server_name --client-requested <br>
> "/usr/local/squid/etc/splice.list"<br>
<br>
> acl alwaysBump ssl::server_name --client-requested <br>
> <a href="http://storage.googleapis.com" rel="noreferrer" target="_blank">storage.googleapis.com</a> <<a href="http://storage.googleapis.com" rel="noreferrer" target="_blank">http://storage.googleapis.com</a>> <br>
> <a href="http://youtubei.googleapis.com" rel="noreferrer" target="_blank">youtubei.googleapis.com</a> <<a href="http://youtubei.googleapis.com" rel="noreferrer" target="_blank">http://youtubei.googleapis.com</a>> <a href="http://www.eset.com" rel="noreferrer" target="_blank">www.eset.com</a> <br>
> <<a href="http://www.eset.com" rel="noreferrer" target="_blank">http://www.eset.com</a>> <a href="http://eset.com" rel="noreferrer" target="_blank">eset.com</a> <<a href="http://eset.com" rel="noreferrer" target="_blank">http://eset.com</a>> <br>
> <a href="http://safebrowsing.googleapis.com" rel="noreferrer" target="_blank">safebrowsing.googleapis.com</a> <<a href="http://safebrowsing.googleapis.com" rel="noreferrer" target="_blank">http://safebrowsing.googleapis.com</a>> <br>
> <a href="http://play.google.com" rel="noreferrer" target="_blank">play.google.com</a> <<a href="http://play.google.com" rel="noreferrer" target="_blank">http://play.google.com</a>><br>
> on_unsupported_protocol tunnel<br>
> acl DiscoverSNIHost at_step SslBump1<br>
> ssl_bump peek DiscoverSNIHost<br>
> ssl_bump bump alwaysBump - Used to bumd certain subdomains before the <br>
> whole domain is bumped.<br>
> ssl_bump splice NoSSLInterceptRegexp_always<br>
> ssl_bump stare all<br>
> <br>
> <br>
> <br>
> Other CONNECT requests are served noramly.<br>
> Is this issue could be a root cause for the generally slow internet?<br>
> <br>
> Thanks,<br>
> Ben<br>
> <br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div>
</blockquote></div></div>