[squid-users] Getting the full file content on a range request, but not on EVERY get ...

[Garri Djavadyan]

On Sat, 2016-05-14 at 01:52 +1200, Amos Jeffries wrote:
> The default action should be to fetch each range request separately
> and
> in parallel. Not caching the results.
> 
> When admin has set only the range offset & quick-abort to force full
> object retrieval the behaviour Heiler mentions happens - lots of
> upstream bandwidth used for N copies.
>  The first one to complete starts to be used as a HIT for future
> reuqests. But as each of the initial transfers completes it replaces
> the
> previously cached object as the one being hit on.
> 
> So timing is critical, if Squid happens to delay any of the parallel
> requests just long enough in its TCP accept() queue, auth or ACL
> processing they could become HITs on an already finished object.
> 
> Amos

Yes, you was right! Timing is very critical. I've simulated two
concurrent transfers (without 'collapsed_forwarding on' and with
'range_offset_limit none') using this code:

---
#!/bin/bash

export http_proxy="127.0.0.1:3128"
curl --range $((1024 * 1024 * 1))-$((1024 * 1024 * 2)) http://mirror.co
mnet.uz/centos/7/os/x86_64/images/efiboot.img > /dev/null &
curl --range $((1024 * 1024 * 3))-$((1024 * 1024 * 4)) http://mirror.co
mnet.uz/centos/7/os/x86_64/images/efiboot.img > /dev/null
---

And got two MISS':

1463150025.340    987 127.0.0.1 TCP_MISS/206 1048943 GET http://mirror.
comnet.uz/centos/7/os/x86_64/images/efiboot.img -
HIER_DIRECT/91.196.76.102 application/octet-stream
1463150026.315   1963 127.0.0.1 TCP_MISS/206 1048943 GET http://mirror.
comnet.uz/centos/7/os/x86_64/images/efiboot.img -
HIER_DIRECT/91.196.76.102 application/octet-stream

Then, I purged the object repeated with 'collapsed_forwarding on' and
got MISS and HIT:

1463150169.010    370 127.0.0.1 TCP_MISS/206 1048943 GET http://mirror.
comnet.uz/centos/7/os/x86_64/images/efiboot.img -
HIER_DIRECT/91.196.76.102 application/octet-stream
1463150169.476    836 127.0.0.1 TCP_HIT/206 1048950 GET http://mirror.c
omnet.uz/centos/7/os/x86_64/images/efiboot.img - HIER_NONE/-
application/octet-stream


Amos, thank you very much for the detailed explanation.
collapsed_forwarding is really useful option for this situation.