[squid-users] Adding headers in ICAP server with no preview

Wed Jan 20 07:52:10 UTC 2021

Thank you very much, that really helped!

On Mon, Jan 18, 2021 at 6:48 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 1/18/21 6:45 AM, Moti Berger wrote:
>
> > If the ICAP server sets 'Preview: 0' in the OPTIONS it means that when
> > the ICAP client sends a request, it should not contain the body.
>
> The above summary may mislead many readers. I would describe the
> protocol operation differently:
>
> * Preview in an OPTIONS response indicates that the server supports
> Preview in general and specifies the maximum Preview size the client
> should use (e.g., Preview:0 limits Preview to HTTP headers).
>
> * The Preview mode for a specific REQMOD or RESPMOD transaction is
> signaled in the corresponding REQMOD or RESPMOD request (not a previous
> OPTIONS response) by adding a Preview:N ICAP request header (Preview:0
> specifies a headers-only Preview for the current transaction).
>
> * The REQMOD or RESPMOD transaction with a Preview:0 request header is
> split into two phases. During the first phase, the client must not send
> the virgin body. During the second phase, if any, the client must send
> the virgin body. Both phases comprise a single ICAP transaction, with a
> single ICAP request and a single ICAP response. Thus, one cannot say
> that this transaction (as a whole) "should not contain a body".
>
>
> > This is the REQMOD request:
> >
> >     F..n...DREQMOD icap://censor-req.proxy:14590/request ICAP/1.0
> >     Host: censor-req.proxy:14590
> >     Date: Mon, 18 Jan 2021 11:34:54 GMT
> >     Encapsulated: req-hdr=0, req-body=222
> >     Preview: 0
> >     Allow: 204, trailers
> >     X-custom-header: data
> >
> >     POST http://www.dst-server.com:22222/v1/test HTTP/1.1
> >     User-Agent: python-requests/2.25.1
> >     Accept-Encoding: gzip, deflate
> >     Accept: */*
> >     Content-Length: 10
> >     Content-Type: application/json
> >     Host: www.dst-server.com:22222 <http://www.dst-server.com:22222>
>
> > The ICAP 'Encapsulated' header has a req-body even though no 'body'
> > should be in this request.
>
> Not exactly. The request may not be over at this point. Please see my
> third bullet above for details.
>
>
> > The ICAP server has no way to encapsulate the HTTP request body if it
> > didn't get it.
>
> To get the request body in Preview:0 mode, the ICAP server must respond
> with ICAP 100 (Continue).
>
>
> > I want to avoid sending the body because the adaptation is body agnostic.
>
> Yes, I know, but you have to work within the ICAP protocol boundaries.
> ICAP simply does not optimize your use case! After you have the basics
> working well, you can invest in implementing a use-original-body ICAP
> extension[1] that, in _some_ cases, can prevent the body exchange while
> adapting HTTP headers.
>
> Alternatively, you can use an existing (extendible) ICAP server to do
> the legwork for you [2]. Many individuals and companies have learned the
> hard way that implementing an ICAP service correctly from scratch is
> very difficult and often prohibitively expensive.
>
> [1]
>
> http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt
>
> [2] https://wiki.squid-cache.org/Features/ICAP#ICAP_Servers
>
>
> HTH,
>
> Alex.
>
>
>
> > On Sun, Jan 17, 2021 at 11:34 PM Alex Rousskov wrote:
> >
> >     On 1/17/21 3:08 PM, Moti Berger wrote:
> >     > What should the ICAP response look like?
> >
> >     The vast majority off ICAP responses containing an HTTP POST message
> >     will look like ICAP header + HTTP header + HTTP body. Please see RFC
> >     3507 and its errata for examples of and discussion about those three
> >     components. It should help avoid guessing and developing by examples
> >     (which usually leads to bugs, especially where ICAP is involved).
> >
> >
> >     > What I do is to reply like this:
> >     >
> >     >     (dI./M..ICAP/1.0 200 OK
> >     >     ISTag: "SjIzlRA4te41axxcDOoiSl6rBRg4ZK"
> >     >     Date: Sun, 17 Jan 2021 19:34:12 GMT
> >     >     Server: BaseICAP/1.0 Python/3.6.12
> >     >     Encapsulated: req-hdr=0, req-body=360
> >     >
> >     >     POST http://www.dst-server.com:22222/v1/test HTTP/1.1
> >     >     x-new-header: {"key": "value"}
> >     >     user-agent: python-requests/2.25.1
> >     >     accept-encoding: gzip, deflate
> >     >     accept: */*
> >     >     content-length: 16
> >     >     content-type: application/json
> >     >     host: www.dst-server.com:22222
> >     <http://www.dst-server.com:22222> <http://www.dst-server.com:22222>
> >
> >
> >     FYI: The above incomplete ICAP response promises an HTTP request
> body,
> >     both on the ICAP level (req-body) and on the HTTP level
> (content-length:
> >     16).
> >
> >
> >     > As I said, I use 'Preview: 0' since I don't mind the body. The
> >     question
> >     > is whether declaring the body starts at X (req-body=X) is OK even
> >     though
> >     > I don't have a body to send?
> >
> >     It is not OK not to send the body. Encapsulated:req-body does more
> than
> >     declaring where the encapsulated headers end. It also promises an
> >     embedded HTTP body after those headers. You must encapsulate the
> body if
> >     the HTTP message should have one. You cannot adapt the header of an
> HTTP
> >     message with a body without also sending the HTTP body (virgin or
> >     adapted).
> >
> >     Preview is pretty much irrelevant in this context -- the ICAP
> protocol
> >     does not care how the ICAP service gets the HTTP body to include in
> the
> >     ICAP response.
> >
> >     There are unofficial ICAP extensions that make it possible to tell
> the
> >     ICAP client to reuse the body it has buffered while adapting the
> header,
> >     but you should get the baseline case working before bothering with
> those
> >     extensions -- they are optimizations that are not applicable to some
> >     transactions.
> >
> >
> >     > I think having req-null=X is bad since it
> >     > probably tells squid that I decided the adapted request should
> have no
> >     > body, but that's only a guess.
> >
> >     If you meant to say "null-body", then you guessed correctly --
> null-body
> >     means the adapted HTTP message has no body. That is not what you
> want to
> >     say when adapting most HTTP POST messages.
> >
> >
> >     HTH,
> >
> >     Alex.
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20210120/9c446f8a/attachment.htm>