[squid-users] Adding headers in ICAP server with no preview

[Alex Rousskov]

On 1/18/21 6:45 AM, Moti Berger wrote:

> If the ICAP server sets 'Preview: 0' in the OPTIONS it means that when
> the ICAP client sends a request, it should not contain the body.

The above summary may mislead many readers. I would describe the
protocol operation differently:

* Preview in an OPTIONS response indicates that the server supports
Preview in general and specifies the maximum Preview size the client
should use (e.g., Preview:0 limits Preview to HTTP headers).

* The Preview mode for a specific REQMOD or RESPMOD transaction is
signaled in the corresponding REQMOD or RESPMOD request (not a previous
OPTIONS response) by adding a Preview:N ICAP request header (Preview:0
specifies a headers-only Preview for the current transaction).

* The REQMOD or RESPMOD transaction with a Preview:0 request header is
split into two phases. During the first phase, the client must not send
the virgin body. During the second phase, if any, the client must send
the virgin body. Both phases comprise a single ICAP transaction, with a
single ICAP request and a single ICAP response. Thus, one cannot say
that this transaction (as a whole) "should not contain a body".


> This is the REQMOD request:
> 
>     F..n...DREQMOD icap://censor-req.proxy:14590/request ICAP/1.0
>     Host: censor-req.proxy:14590
>     Date: Mon, 18 Jan 2021 11:34:54 GMT
>     Encapsulated: req-hdr=0, req-body=222
>     Preview: 0
>     Allow: 204, trailers
>     X-custom-header: data
> 
>     POST http://www.dst-server.com:22222/v1/test HTTP/1.1
>     User-Agent: python-requests/2.25.1
>     Accept-Encoding: gzip, deflate
>     Accept: */*
>     Content-Length: 10
>     Content-Type: application/json
>     Host: www.dst-server.com:22222 <http://www.dst-server.com:22222>

> The ICAP 'Encapsulated' header has a req-body even though no 'body'
> should be in this request.

Not exactly. The request may not be over at this point. Please see my
third bullet above for details.


> The ICAP server has no way to encapsulate the HTTP request body if it
> didn't get it.

To get the request body in Preview:0 mode, the ICAP server must respond
with ICAP 100 (Continue).


> I want to avoid sending the body because the adaptation is body agnostic.

Yes, I know, but you have to work within the ICAP protocol boundaries.
ICAP simply does not optimize your use case! After you have the basics
working well, you can invest in implementing a use-original-body ICAP
extension[1] that, in _some_ cases, can prevent the body exchange while
adapting HTTP headers.

Alternatively, you can use an existing (extendible) ICAP server to do
the legwork for you [2]. Many individuals and companies have learned the
hard way that implementing an ICAP service correctly from scratch is
very difficult and often prohibitively expensive.

[1]
http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt

[2] https://wiki.squid-cache.org/Features/ICAP#ICAP_Servers


HTH,

Alex.



> On Sun, Jan 17, 2021 at 11:34 PM Alex Rousskov wrote:
> 
>     On 1/17/21 3:08 PM, Moti Berger wrote:
>     > What should the ICAP response look like?
> 
>     The vast majority off ICAP responses containing an HTTP POST message
>     will look like ICAP header + HTTP header + HTTP body. Please see RFC
>     3507 and its errata for examples of and discussion about those three
>     components. It should help avoid guessing and developing by examples
>     (which usually leads to bugs, especially where ICAP is involved).
> 
> 
>     > What I do is to reply like this:
>     >
>     >     (dI./M..ICAP/1.0 200 OK
>     >     ISTag: "SjIzlRA4te41axxcDOoiSl6rBRg4ZK"
>     >     Date: Sun, 17 Jan 2021 19:34:12 GMT
>     >     Server: BaseICAP/1.0 Python/3.6.12
>     >     Encapsulated: req-hdr=0, req-body=360
>     >
>     >     POST http://www.dst-server.com:22222/v1/test HTTP/1.1
>     >     x-new-header: {"key": "value"}
>     >     user-agent: python-requests/2.25.1
>     >     accept-encoding: gzip, deflate
>     >     accept: */*
>     >     content-length: 16
>     >     content-type: application/json
>     >     host: www.dst-server.com:22222
>     <http://www.dst-server.com:22222> <http://www.dst-server.com:22222>
> 
> 
>     FYI: The above incomplete ICAP response promises an HTTP request body,
>     both on the ICAP level (req-body) and on the HTTP level (content-length:
>     16).
> 
> 
>     > As I said, I use 'Preview: 0' since I don't mind the body. The
>     question
>     > is whether declaring the body starts at X (req-body=X) is OK even
>     though
>     > I don't have a body to send?
> 
>     It is not OK not to send the body. Encapsulated:req-body does more than
>     declaring where the encapsulated headers end. It also promises an
>     embedded HTTP body after those headers. You must encapsulate the body if
>     the HTTP message should have one. You cannot adapt the header of an HTTP
>     message with a body without also sending the HTTP body (virgin or
>     adapted).
> 
>     Preview is pretty much irrelevant in this context -- the ICAP protocol
>     does not care how the ICAP service gets the HTTP body to include in the
>     ICAP response.
> 
>     There are unofficial ICAP extensions that make it possible to tell the
>     ICAP client to reuse the body it has buffered while adapting the header,
>     but you should get the baseline case working before bothering with those
>     extensions -- they are optimizations that are not applicable to some
>     transactions.
> 
> 
>     > I think having req-null=X is bad since it
>     > probably tells squid that I decided the adapted request should have no
>     > body, but that's only a guess.
> 
>     If you meant to say "null-body", then you guessed correctly -- null-body
>     means the adapted HTTP message has no body. That is not what you want to
>     say when adapting most HTTP POST messages.
> 
> 
>     HTH,
> 
>     Alex.
>