[squid-users] Proxy Server closes the connection to http server before transferring all application data to http client

Mon Dec 21 09:11:30 UTC 2020

Thanks Alex! 
I will contact curl project to take a look.

Thanks
Lily

-----Original Message-----
From: Alex Rousskov <rousskov at measurement-factory.com> 
Sent: Friday, December 18, 2020 11:44 PM
To: Zhang, Lily (USD); squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Proxy Server closes the connection to http server before transferring all application data to http client

[EXTERNAL EMAIL] 

On 12/18/20 6:10 AM, Zhang, Lily (USD) wrote:

> Http client use libcurl,  curl_easy_recv API returns CURLE_UNSUPPORTED_PROTOCOL,    /* 1 */
> This meets the description https://curl.se/mail/lib-2013-01/0246.html , CURLE_UNSUPPORTED_PROTOCOL  means that connection was shutdown.
> 
> 10.105.8.55 is http server ip, 10.245.166.20 is the http client ip, 10.244.102.133 is proxy server's ip. 

The test transaction is an HTTP CONNECT transaction (that is not going through SslBump decoding in Squid). Thus, after the initial CONNECT headers exchange, Squid plays a role of a TCP tunnel. I am guessing that the tunnel contains HTTPS transactions, but Squid does not "see" them -- Squid only sees opaque TCP payload bytes, not TLS- and not HTTP-layer information. Whatever settings you apply to those HTTPS transactions, including HTTP Connection headers are _invisible_ to your Squid.

I see several reads/writes from/to the client and several reads/writes from/to the server. The numbers appear to match -- after the CONNECT exchange, everything read from one side is written to the other, in both directions. This observation contradicts the statement in the Subject line of this thread.

If you disagree, please double check the I/O sizes in the logged HandleRead and HandleWrite lines for FD 13 (client) and FD 15 (server).
The sizes should match after the CONNECT exchange, which ends with the
39 "HTTP/1.1 200 Connection established\r\n\r\n" bytes written to FD 13 at 10:54:01.882:

    2020/12/18 10:54:01.882 kid1| 5,5| Write.cc(66) HandleWrite:
local=10.244.102.133:3128 remote=10.245.166.20:39906 FD 13 flags=1: off 0, sz 39.

A few exchanges later, Squid reads EOF from the origin server at
10:54:03.944:

    2020/12/18 10:54:03.944 kid1| 5,3| Read.cc(145) HandleRead: FD 15, size 65535, retval 0, errno 0

At that time, Squid has already written everything (it previously read from the server) to the client. Squid tunnels do not support half-closed connections, so Squid closes the server and the client connections. End of story as far as Squid is concerned. This is normal/expected behavior.

If your client is unhappy about something, and changing HTTP headers (that Squid does not see!) changes the client or server behavior, then your problem lies in the client or server, not Squid.

Squid could be enhanced to support half-closed connections (when tunneling traffic), but there is currently no indication that your problem is caused by the lack of that support and, IIRC, HTTP CONNECT does not require such support, so relying on it is a bad idea in general.

Your next step is to understand what the client is unhappy about. While Factory has done some serious Curl development, I am not familiar with CURLE_UNSUPPORTED_PROTOCOL specific. If you cannot figure it out on your own, you may want to contact the Curl project for support, especially if you found some relevant posts on its mailing lists. I have _not_ studied that email thread, but if it was a curl bug, make sure you are using the fixed version of their library (if available).

HTH,

Alex.

> -----Original Message-----
> From: Alex Rousskov <rousskov at measurement-factory.com>
> Sent: Friday, December 18, 2020 1:33 AM
> To: Zhang, Lily (USD); squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] Proxy Server closes the connection to http 
> server before transferring all application data to http client
> 
> 
> [EXTERNAL EMAIL]
> 
> On 12/17/20 3:31 AM, Zhang, Lily (USD) wrote:
> 
>>> If you want to work on that, respond to this message and somebody will send you more detailed instructions.
> 
>> Yes, please tell me how to do this.  
> 
> Please try the procedure described at
> https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_
> transaction
> 
> In your case, if possible, use a single problematic master transaction (including client-Squid and Squid-server HTTP transactions) -- the one that results in response truncation. I recommend "ALL,9" debugging for this case. Please share a link to the _compressed_ log.
> 
> 
>> I am curious of
> 
>> 1. When http client uses close, and server uses keep-alive, when 
>> connection between proxy server and http server is closed, it still 
>> can transfer data between client and proxy server,
> 
> Yes, Squid should send the rest of the response to the client after the server sent the entire response to Squid.
> 
> Whether the server closed the connection after sending the entire response is irrelevant (except for special cases where the end of response is determined by the connection closure).
> 
> 
>> and data transfer to http client is not completed sometimes.
> 
> That is a mystery you want to solve.
> 
> 
>> 2. When http client uses close, and server uses closes, no data 
>> transfer from http client and proxy server after http server closes 
>> the connection.
> 
> I assume that by "from http client and proxy" you meant "to http client from proxy".
> 
> That is another mystery you want to solve.
> 
> Hopefully, debugging logs will help us solve both mysteries.
> 
> Alex.
> 
> 
>> -----Original Message-----
>> From: Alex Rousskov <rousskov at measurement-factory.com>
>> Sent: Wednesday, December 16, 2020 10:50 PM
>> To: Zhang, Lily (USD); squid-users at lists.squid-cache.org
>> Subject: Re: [squid-users] Proxy Server closes the connection to http 
>> server before transferring all application data to http client
>>
>>
>> [EXTERNAL EMAIL]
>>
>> On 12/16/20 4:02 AM, Zhang, Lily (USD) wrote:
>>
>>> Issue still can be reproduced when http client using connection: close, and http server uses connection: keep-alive.
>>> I can't reproduce this issue when both side using keep-alive.
>>
>> When everything works as it should, the HTTP Connection header has no effect on the data transfer during the transaction that header belongs to. Both "close" and "keep-alive" instructions tell the agents what to do _after_ the entire response body is delivered from the server to the client. This is just how HTTP works... Thus, what you are describing suggests a bug in client software, server software, Squid, and/or the test.
>>
>> Going forward:
>>
>> * If you do not need further assistance and do not want to work to discover the true cause of the problem, then just ignore this message. I am glad you found a workaround, and we can leave it at that.
>>
>> * Otherwise, the best next step may be to share Squid debugging logs while reproducing the problem in the lab. If you want to work on that, respond to this message and somebody will send you more detailed instructions.
>>
>>
>> Cheers,
>>
>> Alex.
>>
>>
>>> -----Original Message-----
>>> From: Zhang, Lily (USD)
>>> Sent: Wednesday, December 16, 2020 3:29 PM
>>> To: Alex Rousskov; squid-users at lists.squid-cache.org
>>> Subject: RE: [squid-users] Proxy Server closes the connection to 
>>> http server before transferring all application data to http client
>>>
>>> Hi Alex,
>>> Thanks for your reply.
>>>
>>> At first, http server turns off  keep-alive, then http client can't get the full response sometimes (example 20 0000 char, get 17 0000 chars).
>>>
>>> I did a test in the lab, found
>>>
>>> 1. If http client uses connection: close, and http server uses
>>> connection: keep-alive, then still can transfer data after tcp 
>>> connection between http server and proxy server is closed.  (don't 
>>> know how this happens)
>>>
>>> 2. If http client uses connection: keep-alive, and http server uses connection: keep-alive, then connection between http server and proxy server won't closed first. It is closed after client closes the connection.
>>>
>>> Thanks
>>> Lily
>>> -----Original Message-----
>>> From: Alex Rousskov <rousskov at measurement-factory.com>
>>> Sent: Tuesday, December 15, 2020 11:53 PM
>>> To: squid-users at lists.squid-cache.org
>>> Cc: Zhang, Lily (USD)
>>> Subject: Re: [squid-users] Proxy Server closes the connection to 
>>> http server before transferring all application data to http client
>>>
>>>
>>> [EXTERNAL EMAIL]
>>>
>>> On 12/15/20 2:52 AM, Zhang, Lily (USD) wrote:
>>>
>>>> Looks we can use "Connection: keep-alive" in http request to solve this issue.
>>>
>>> That client request header will not change Squid's reaction to Squid-server connection closure by the server. It also should not change the request that Squid sends to the server (if it does, it is a Squid bug). In HTTP proxying, client-Squid and Squid-server connections/negotiations are mostly independent.
>>>
>>> If changing the client to send Connection:keep-alive to Squid fixes your problem, then the problem was most likely unrelated to the behavior you asked about earlier.
>>>
>>> If you can configure the HTTP server to keep the connection open/persistent after delivering the response (and, hence, to send a Connection:keep-alive _response_ header or otherwise signal HTTP connection persistency), then, yes, Squid may pool that _open_ Squid-server connection for future reuse (again, regardless of what headers the HTTP client is using).
>>>
>>> Alex.
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: squid-users <squid-users-bounces at lists.squid-cache.org> On 
>>>> Behalf Of Amos Jeffries
>>>> Sent: Tuesday, December 15, 2020 3:47 PM
>>>> To: squid-users at lists.squid-cache.org
>>>> Subject: Re: [squid-users] Proxy Server closes the connection to 
>>>> http server before transferring all application data to http client
>>>>
>>>>
>>>> [EXTERNAL EMAIL]
>>>>
>>>> On 15/12/20 4:21 pm, Zhang, Lily (USD) wrote:
>>>>> Hi
>>>>>
>>>>> I installed 4.13 squid proxy server. See attachment, http server
>>>>> (10.250.16.46) sends FIN, ACK to tells that response is finished. 
>>>>> Proxy server (10.244.102.133) sends FIN, ACK back to http server
>>>>> (10.250.16.46) before "Application Data" is transferred to http 
>>>>> client (10.245.166.20).
>>>>>
>>>>> Would you please help me on questions below:
>>>>>
>>>>>  1. Is it normal that proxy server sends FIN, ACK to http server before
>>>>>     http client finishes receiving  all the “Application Data” ?
>>>>
>>>> That depends on the transaction which is being performed.
>>>>
>>>> If it is a normal HTTP request-response transaction then yes. Once Squid has the response the server connection is done with - it may be closed or re-used for other transactions.
>>>>
>>>> If it is a tunnel containing non-HTTP traffic then no. Squid should only close the server connection when the client closes its end of the tunnel.
>>>>
>>>> Also, note that FIN+ACK in a single packet is a *response* to a FIN having come from the server itself. Not initiated by the proxy.
>>>>
>>>>
>>>>
>>>>>  2. Does proxy server have option to stop item 1?
>>>>>
>>>>
>>>> In general: No. This is something that is supposed to happen (or
>>>> not) according to the relevant protocol requirements.
>>>>
>>>> In specific, there may be some options that can be configured to 
>>>> change how the protocol behaves. Preventing it needing a close. You 
>>>> will have to find out what the cause actually is to determine where 
>>>> to look for solutions.
>>>>
>>>>
>>>> Amos
>>>>
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>>>
>>
> 