[squid-users] range_offset_limit and idm

[Amos Jeffries]

On 27/10/2015 1:00 a.m., joe wrote:
> regarding range_offset_limit   no matter what i do idm or any download
> manager not working correctly
> 
> try to download using idm and it start downloading one file suppose to start
> downloading multiple of chunk range setting idm to download 16 chunk range
> at the same time only one start downloading once it finish downloading first
> part it start the second one after the other and that is bad  wen i pause or
> stop then resume downloading all the chunk range start normal downloading
> same time that is normal behavior 
> 
> only new file at the start it download one chunk insted of multiple until i
> hit stop then resume so squid respond only to one range wen the new file
> start 
> hitting stop then resume    squid working good with all the range
> downloading same time 
> and that big time issue since most of my customer using  idm or any download
> manager file not cached 
> i duno if i explain it right but over squid supost to respond to all the
> range as idm rquest at the begining of the download if i bypass   squid idm
> work correctly 

Sounds right. idm is sending multiple parallel Range requests.
Essentially trying to fake faster downloads by forcing as many resources
to be consumed by the one user as possible. This is a very unfriendly
client behaviour - stealing bandwidth and resources from other users who
need to share them, at all network layers from the CPE device outwards
to the server itself.


Squid is an optimizing proxy. Parts of its design is to baulk that type
of behaviour and make the network operate better for all. In particular
it optimizes bandwidth consumption when you set range_offset_limit by
fetching the entire object from the beginning of the download so that it
can cache and server followup requests from the cache without consuming
any server-facing resources at all regardless of number of users wanting
that URL.


If you dont use range_offset_limit then all fetches made by idm are
Range request and cannot be cached to improve the experience for other
clients needing the same URL/download. Each and every time a new object
gets fetched, but in small pieces and hogging some of the potentially
limited upstream bandwidth by the user with idm.

When range_offset_limit is configured...

* idm has made 5 Range requests in parallel.

What that means is that it has sent 5x requests to Squid for the same
object. But not the same data out of it, which makes them each a unique
fetch.
 So... Squid starts fetching 5x downloads of the file (ouch, you can
enable collapsed_forwarding to reduce that).

 Squid quickly reaches the start of the Range connection #1 requested
and starts delivering it. BUT, it has not reached the download positions
connection #2, #3, #4, #5 requested.

All idm can see is connection #1 receiving data and the rest waiting.

Then at the end of #1 connection Range Squid also hits the point where
it has received the start of connection #2's Range and starts serving
that up.

Meanwhile connection #3, #4, #5 have not yet been reached, and so it
continues until all the Ranges have been fetched and delivered or idm
goes away.


When using a proxy tools like idm are useless. Their abusive
"optimizations" do more harm than good when they do "work".

Amos