[squid-users] Big HTTP-POST file uploads not working

[Alex Rousskov]

On 1/29/19 12:05 AM, Matthias Weigel wrote:

> Any idea, why the IIS server behaves differently with Apache as the
> Reverse Proxy (anything else being the same)?

Given Amos' analysis, if you want to get to the bottom of this problem,
you would need to figure out why the origin server drops the connection.
This may take a while, but is usually possible by enabling origin server
debugging, analyzing packet dumps, comparing working cases with
non-working ones, and/or, in the extreme cases, playing with how Squid
sends data. It may take a pro a few hours or a few days, but it is
usually doable.

One interesting experiment would be to test Squid-Apache-IIS chain.

The analysis can either confirm a Squid bug (something we can fix),
detect a hardware problem (something you can fix), or provide enough
information to develop a workaround for the IIS bug (other than enabling
ALL,9 debugging in Squid :-).

Since ALL,9 debugging has an effect, it is possible that rate-limiting
Squid-IIS and/or client-Squid connection would help work around the problem.

Alex.


> Am 29.01.2019 um 05:01 schrieb Amos Jeffries:
>> On 29/01/19 8:50 am, Matthias Weigel wrote:
>>> Hi Alex,
>>>
>>> this problem just got a lot weirder:
>>> - Upload works fine with debug_options ALL,7 ALL,8 or ALL,9.
>>> - Upload fails with debug_options ALL,6 and lower.
>>>
>>> I created a debug_options ALL,6 cache.log.
>>> http://94.16.117.186/squid/cache.log.3b.gz
>>>
>>> During this test, the following happened at the client:
>>>
>>> Stall at 20:10:38
>>> Recover at 20:12:47
>>> Stall at 20:14:11
>>> Failure at 20:16:17
>>>
>>> Any ideas?
>>>
>>
>> The FD 18 client->Squid I/O stops happening at 20:10:35.923 with input
>> buffer full and more data apparently waiting to arrive.
>>
>> But the FD 20 Squid->server I/O halted earlier at 20:10:35.437 with a
>> 4KB packet being sent and still waiting for the signal to proceed
>> sending more.
>>
>> At 20:12:47.068 a signal is received finally, but it indicates the
>> server connection has gone away with no reason given. Squid generates a
>> 502 error response and delivers that to the client.
>>
>> The client opens a new connection, does all its TLS handshakes again and
>> re-sends the same very large POST request. Which encounters the same
>> sequence of server stops responding, client buffer fills and "stall"
>> until the server FD is closed. Then again the 502 from Squid but the
>> client does not resume this time.
>>
>>
>> So whatever is going on it looks to me like it is the origin server
>> being a problem. Though you may need to look at the actual TCP packet
>> flow to confirm. I expect you will find that a packet from Squid to the
>> server does not get ACK'd, or a network timeout occurs (NAT or TCP
>> lifetimes are common), ...
>>
>>  Or maybe the server is itself dealing with a similar backlog from
>> unresponsive internal agent/component leaving its input buffer to
>> overflow just like Squids one from the client does.
>>
>>
>> Amos
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>