[squid-users] Header order in squid proxy

[Alex Rousskov]

On 06/22/2017 11:49 AM, Sonya Roy wrote:

> I noticed that squid changes the header order received from the client
> before sending it to the origin server.
> 
> I assume this is because squid parses the header data and adds some
> headers depending on the config file and then recreates the header data.

IIRC, modern Squids change a header field position when the received
field is deleted and then added back. This is typical for hop-by-hop
headers such as Connection, but there are other reasons for Squid to
delete and add a header field. When the value of the added field is the
same as the value of the removed field, such pointless "editing" looks
like mindless "reordering" to the outside observer.

The two actions (field deletion and addition) may happen in a single
piece of code or may be separated by lots of code and even time.
Preventing pointless editing in the former cases is straightforward, but
the latter cases are difficult to handle. Correct avoidance of pointless
editing may improve performance and, if it does, can be considered a
useful optimization on its own, regardless of your use case.


> Is there any way to prevent this?

Not without changing Squid code (or adding more proxies). However,
before we even talk about code changes, we should clarify the problem we
are dealing with. The questions below will guide you.

It is probably much easier to ensure some fixed field send order
(regardless of the received order) than to preserve the received order.
Will a fixed order (e.g., always alphabetical) address your use case?
This feature will hurt performance, but you might be able to convince
others to accept it if you have a very compelling/specific/detailed use
case because it can be disabled by default.


> I am asking because some sites detect bots using the header order and
> they drop any such connection. So they unintentionally block squid
> proxies even if its not being used by a bot.

Are you implying that bots often change header field order between their
requests? Or that bots often use a different (fixed) header field order
than the (fixed) field order used by non-bots? Preserving received order
may help in the former case but not in the latter case.

Also, do those blocking sites pay attention to all headers or just
end-to-end headers?

Please note that there are many other ways to detect a proxy so if a
site wants to block proxies rather than bots, then it is probably
pointless to fight it (or, at least, the Squid Project should not).


HTH,

Alex.