[squid-users] Icap preview size

Sat Mar 6 22:38:08 UTC 2021

Hi Alex,

Missed that in the RFC, I was hoping I could only send the headers across and modify those, without having to send over everything, including very large responses for performance reasons.
Thanks for following along.

Regards,
Niels

Sent from my mobile

On 6 Mar 2021, at 23:22, Alex Rousskov <rousskov at measurement-factory.com> wrote:

On 3/6/21 3:33 PM, Niels Hofmans wrote:

> I fixed a bug in the go-icap/icap library, seen here:

FYI: Whatever you were trying to share in this paragraph, did not come
through the mailing list. However, we should not be discussing some ICAP
library bugs on this mailing list, so perhaps it is not important.

> But this made me wonder about the following: <PastedGraphic-1.tiff>

In the future, please paste message _text_ and/or provide frame/stream
references to pieces of shared captures. Working with images is very
inefficient when it comes to protocol analysis.

> Here you can see squid issuing a RESPMOD to my ICAP server with a
> Preview of “0”,

Yes.

> but it does contain a HTTP response.

It contains an HTTP request header (shown in the PastedGraphic-1.tiff
image you have attached), probably an HTTP response header (not shown),
and probably no HTTP response body. The HTTP response does have a body,
but the ICAP client does not send it during the Preview:0 stage. This is
how ICAP Preview works. I see nothing wrong here.

> And since my ICAP service will not return a HTTP response since the
> Preview size is 0, my http client receives nothing from squid.

AFAICT, your ICAP service is butchering HTTP responses by responding
with ICAP 200 responses that encapsulate HTTP messages without bodies. A
standard ICAP client will _not_ re-assemble the HTTP message by
combining your header with client-cached body parts. The ICAP standard
just does not work that way. When the ICAP service sends an ICAP 200
response, the service must return the _entire_ HTTP message, headers and
body.

As a consequence, an ICAP service cannot use standard ICAP to adapt HTTP
headers without receiving and returning the HTTP body as well. This
applies to REQMOD and RESPMOD transactions. You are just lucky that your
test REQMOD transactions have requests without bodies.

> I checked but I don’t think I see a 100 continuation from ICAP in the
> dump anymore.

Yes, that is correct. However, the new trace shows another ICAP service
problem as detailed above.

HTH,

Alex.

> On 6 Mar 2021, at 19:29, Alex Rousskov wrote:
> 
> On 3/6/21 5:09 AM, Niels Hofmans wrote:
> 
>> I’ve uploaded a tcpdump sample of where I’m requesting an image through
>> squid + icap and where you can see the image response being posted fully
>> to the ICAP service.
> 
> AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
> zero-size Preview containing just the HTTP headers (e.g., see frame
> #44). In one case, Squid also sends the HTTP response body but only
> because your ICAP service explicitly requests that HTTP response body by
> responding to Squid Preview with ICAP 100 Continue control message
> (e.g., see frame #48)!
> 
> If your ICAP service does not want to see an HTTP body, then it should
> not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
> based on the Preview alone, without sending ICAP 100 Continue control
> message first.
> 
> 
> HTH,
> 
> Alex.
> 
> 
>> On 5 Mar 2021, at 23:32, Alex Rousskov wrote:
>> On 3/5/21 5:21 PM, Niels Hofmans wrote:
>>> I receive that large payload right after an OPTIONS call to my ICAP
>>> endpoint. It is the preview.
>> The timing of an ICAP request does not determine whether that request
>> has a Preview stage and whether the "unexpected" body bytes were sent
>> during that Preview stage. I am asking these questions because I want to
>> determine whether Squid increases Preview size beyond what your server
>> has requested or does not send Preview at all.
>> Please share the ICAP headers of the problematic REQMOD request and the
>> last chunk metadata of that request body. You can get the latter from a
>> packet capture if your ICAP server does not report it in a convenient
>> form. In fact, sharing (a pointer to) the packet capture of the whole
>> problematic ICAP request is probably a good idea!
>> Alex.
>>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>>> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>>>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>>>> nor RESPMOD.
>>>> For the REQMOD OPTIONS requests, I respond with:
>>>> ICAP/1.0 200 OK
>>>> Allow: 200,204
>>>> Connection: close
>>>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>>>> Encapsulated: null-body=0
>>>> Methods: REQMOD
>>>> Preview: 64000
>>>> However in my ICAP service, I often receive a request body over 64000
>>>> bytes, e.g. 1000000 request bytes.
>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>>>> service will still receive a complete HTTP REQMOD+REQRESP body.
>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>> Alex.
>>>> icap_enable on
>>>> icap_connect_timeout 1 second
>>>> icap_service_failure_limit -1
>>>> icap_send_client_ip    on
>>>> icap_preview_size 0
>>>> icap_persistent_connections on
>>>> icap_service service_req reqmod_precache bypass=0
>>>> icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request>>
>>>> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request>>>
>>>> icap_service service_res respmod_precache bypass=0
>>>> icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response>>
>>>> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response>>>
>>>> icap_preview_enable on
>>>> adaptation_access service_req allow all
>>>> adaptation_access service_res allow all