[squid-users] client->Squid: TCP [RST] and [RST,ACK]

Alex Rousskov rousskov at measurement-factory.com
Tue Feb 28 19:06:11 UTC 2023 

On 2/28/23 13:07, Maciej Leks wrote:
>>Does child Squid bump the TLS client connection, tunnel it, or
>>terminates it (i.e. the child works as a reverse proxy)?

> * client connects to child Squid's http_port without "ssl-bump".
> * client sends a plain text CONNECT request to child Squid.
> * child Squid connects to parent using cache_peer with mTLS.

I assume that, by mTLS, you mean a cache_peer directive with "tls" 
option _and_ with "sslcert" (or equivalent) option to specify the TLS 
client certificate to be sent to the cache peer.

I assume that the packets you pasted earlier were between a child Squid 
and its parent.


>> What are those TLS alerts?
> Code 21 - decryption_failed_RESERVED(21).

I would focus on this error and ignore RST packets for now. Most likely, 
this error is what triggers those packets.


HTH,

Alex.



>>If those TLS alerts are close_notify alerts, then the RST packets you
>>are seeing are probably triggered by the other side doing clean TLS
>>shutdown (i.e. sending a TLS close_notify alert before closing the
>>connection), either after receiving a FIN packet (unlikely with
>>half_closed_clients off?) or while that FIN packet is being generated or
>>is in-flight. When Such RST packets would be sent by the TCP stack
>>rather than Squid code itself. They may be an indication of a benign
>>race condition, a bug, or some other deficiency.
> 
> half_closed_clients is off
> 
>>Higher-level information (who initiates closure and why) may be needed
>>to figure this out. I recommend sharing a link to a compressed ALL,9
>>cache.log reproducing the problem with a single transaction _combined_
>>with a matching packet trace file in libpcap format.
> 
> There is no other way than turning debug on.
> 
> Maciek
> 
> wt., 28 lut 2023 o 15:15 Alex Rousskov <rousskov at measurement-factory.com 
> <mailto:rousskov at measurement-factory.com>> napisał(a):
> 
>     On 2/28/23 08:35, Maciej Leks wrote:
>      > Am I right in saying that RST it's a design intent of squid to end
>      > connections quickly? I've started digging into the squid code and see
>      > SO_LINGER and timeout set to 0, which means that it's done on purpose
>      > not to hang on connections in TIME_WAIT state?
> 
>     Does child Squid bump the TLS client connection, tunnel it, or
>     terminates it (i.e. the child works as a reverse proxy)?
> 
>     What are those TLS alerts?
> 
>     If those TLS alerts are close_notify alerts, then the RST packets you
>     are seeing are probably triggered by the other side doing clean TLS
>     shutdown (i.e. sending a TLS close_notify alert before closing the
>     connection), either after receiving a FIN packet (unlikely with
>     half_closed_clients off?) or while that FIN packet is being
>     generated or
>     is in-flight. When Such RST packets would be sent by the TCP stack
>     rather than Squid code itself. They may be an indication of a benign
>     race condition, a bug, or some other deficiency.
> 
>     Higher-level information (who initiates closure and why) may be needed
>     to figure this out. I recommend sharing a link to a compressed ALL,9
>     cache.log reproducing the problem with a single transaction _combined_
>     with a matching packet trace file in libpcap format.
> 
>     https://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction <https://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction>
> 
>     HTH,
> 
>     Alex.
> 
> 
>      > wt., 28 lut 2023 o 08:12 Maciej Leks <maciej.leks at gmail.com
>     <mailto:maciej.leks at gmail.com>> napisał(a):
>      >>
>      >> For a couple of days we've  been encountering many ECONNRESET error
>      >> messages in our nodejs client to Salesforce. Even if access.log
>     shows
>      >> only /200 tshark shows a messy communication between client->squid
>      >> (child squid).
>      >>
>      >> The architecture: nodejs client->3..* child squid->2*parent
>      >> squids->cloud Salesforce
>      >>
>      >> Some examples ([RST]]: 46 1.015513576 100.121.10.169 → 100.113.27.73
>      >> TCP 66 46098 → 3128 [ACK] Seq=1721 Ack=140135 Win=214656 Len=0
>      >> TSval=2672547296 TSecr=1443424287
>      >>
>      >>     47 1.016152326 100.113.27.73 → 100.121.10.169 TCP 66 3128 →
>     46098
>      >> [FIN, ACK] Seq=140135 Ack=1721 Win=42368 Len=0 TSval=1443424288
>      >> TSecr=2672547296
>      >>     48 1.017856001 100.121.10.169 → 100.113.27.73 TLSv1.2 97
>     Encrypted Alert
>      >>     49 1.017893411 100.121.10.169 → 100.113.27.73 TCP 66 46098 →
>     3128
>      >> [FIN, ACK] Seq=1752 Ack=140136 Win=214656 Len=0 TSval=2672547298
>      >> TSecr=1443424288
>      >>     50 1.018002285 100.113.27.73 → 100.121.10.169 TCP 54 3128 →
>     46098
>      >> [RST] Seq=140136 Win=0 Len=0
>      >>     51 1.018019806 100.113.27.73 → 100.121.10.169 TCP 54 3128 →
>     46098
>      >> [RST] Seq=140136 Win=0 Len=0
>      >>
>      >> [RST,ACK]:
>      >> 592 67.664585034 100.121.10.169 → 100.113.27.73 TLSv1.2 97
>     Encrypted Alert
>      >>    593 67.664737552 100.113.27.73 → 100.121.10.169 TCP 66 3128 →
>     52202
>      >> [ACK] Seq=7973 Ack=1129 Win=42752 Len=0 TSval=1443490937
>      >> TSecr=2672613945
>      >>    594 67.664841613 100.121.10.169 → 100.113.27.73 TCP 66 52202
>     → 3128
>      >> [FIN, ACK] Seq=1129 Ack=7973 Win=42368 Len=0 TSval=2672613945
>      >> TSecr=1443490937
>      >>    595 67.664895660 100.113.27.73 → 100.121.10.169 TCP 66 3128 →
>     52202
>      >> [RST, ACK] Seq=7973 Ack=1129 Win=42752 Len=0 TSval=1443490937
>      >> TSecr=2672613945
>      >>    596 67.664936264 100.113.27.73 → 100.121.10.169 TCP 54 3128 →
>     52202
>      >> [RST] Seq=7973 Win=0 Len=0
>      >>
>      >> I'm wondering how to debug it (what exactly) and whether the reason
>      >> may be on the squid side (specific configuration?).
>      >>
>      >> env: GKE/k8s
>      >> client container: alpine linux
>      >> child squid container: alpine linux
>      >> version: 5.7
>      >>
>      >> Cheers,
>      >> Maciek Leks
>      > _______________________________________________
>      > squid-users mailing list
>      > squid-users at lists.squid-cache.org
>     <mailto:squid-users at lists.squid-cache.org>
>      > http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
> 
>     _______________________________________________
>     squid-users mailing list
>     squid-users at lists.squid-cache.org
>     <mailto:squid-users at lists.squid-cache.org>
>     http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
> 
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

More information about the squid-users mailing list