[squid-users] Icap preview size

Sat Mar 6 23:07:28 UTC 2021

On 3/6/21 5:38 PM, Niels Hofmans wrote:

> Missed that in the RFC, I was hoping I could only send the headers across and modify those, without having to send over everything, including very large responses for performance reasons.

There are ICAP extensions that allow header-only adaptation, but, IMO,
it is best to get the basics working before adding support for
experimental protocol extensions.

Alex.

> On 6 Mar 2021, at 23:22, Alex Rousskov <rousskov at measurement-factory.com> wrote:
> 
> On 3/6/21 3:33 PM, Niels Hofmans wrote:
> 
>> I fixed a bug in the go-icap/icap library, seen here:
> 
> FYI: Whatever you were trying to share in this paragraph, did not come
> through the mailing list. However, we should not be discussing some ICAP
> library bugs on this mailing list, so perhaps it is not important.
> 
> 
>> But this made me wonder about the following: <PastedGraphic-1.tiff>
> 
> In the future, please paste message _text_ and/or provide frame/stream
> references to pieces of shared captures. Working with images is very
> inefficient when it comes to protocol analysis.
> 
> 
>> Here you can see squid issuing a RESPMOD to my ICAP server with a
>> Preview of “0”,
> 
> Yes.
> 
> 
>> but it does contain a HTTP response.
> 
> It contains an HTTP request header (shown in the PastedGraphic-1.tiff
> image you have attached), probably an HTTP response header (not shown),
> and probably no HTTP response body. The HTTP response does have a body,
> but the ICAP client does not send it during the Preview:0 stage. This is
> how ICAP Preview works. I see nothing wrong here.
> 
> 
>> And since my ICAP service will not return a HTTP response since the
>> Preview size is 0, my http client receives nothing from squid.
> 
> AFAICT, your ICAP service is butchering HTTP responses by responding
> with ICAP 200 responses that encapsulate HTTP messages without bodies. A
> standard ICAP client will _not_ re-assemble the HTTP message by
> combining your header with client-cached body parts. The ICAP standard
> just does not work that way. When the ICAP service sends an ICAP 200
> response, the service must return the _entire_ HTTP message, headers and
> body.
> 
> As a consequence, an ICAP service cannot use standard ICAP to adapt HTTP
> headers without receiving and returning the HTTP body as well. This
> applies to REQMOD and RESPMOD transactions. You are just lucky that your
> test REQMOD transactions have requests without bodies.
> 
> 
>> I checked but I don’t think I see a 100 continuation from ICAP in the
>> dump anymore.
> 
> Yes, that is correct. However, the new trace shows another ICAP service
> problem as detailed above.
> 
> 
> HTH,
> 
> Alex.
> 
> 
>> On 6 Mar 2021, at 19:29, Alex Rousskov wrote:
>>
>> On 3/6/21 5:09 AM, Niels Hofmans wrote:
>>
>>> I’ve uploaded a tcpdump sample of where I’m requesting an image through
>>> squid + icap and where you can see the image response being posted fully
>>> to the ICAP service.
>>
>> AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
>> zero-size Preview containing just the HTTP headers (e.g., see frame
>> #44). In one case, Squid also sends the HTTP response body but only
>> because your ICAP service explicitly requests that HTTP response body by
>> responding to Squid Preview with ICAP 100 Continue control message
>> (e.g., see frame #48)!
>>
>> If your ICAP service does not want to see an HTTP body, then it should
>> not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
>> based on the Preview alone, without sending ICAP 100 Continue control
>> message first.
>>
>>
>> HTH,
>>
>> Alex.
>>
>>
>>> On 5 Mar 2021, at 23:32, Alex Rousskov wrote:
>>> On 3/5/21 5:21 PM, Niels Hofmans wrote:
>>>> I receive that large payload right after an OPTIONS call to my ICAP
>>>> endpoint. It is the preview.
>>> The timing of an ICAP request does not determine whether that request
>>> has a Preview stage and whether the "unexpected" body bytes were sent
>>> during that Preview stage. I am asking these questions because I want to
>>> determine whether Squid increases Preview size beyond what your server
>>> has requested or does not send Preview at all.
>>> Please share the ICAP headers of the problematic REQMOD request and the
>>> last chunk metadata of that request body. You can get the latter from a
>>> packet capture if your ICAP server does not report it in a convenient
>>> form. In fact, sharing (a pointer to) the packet capture of the whole
>>> problematic ICAP request is probably a good idea!
>>> Alex.
>>>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>>>> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>>>>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>>>>> nor RESPMOD.
>>>>> For the REQMOD OPTIONS requests, I respond with:
>>>>> ICAP/1.0 200 OK
>>>>> Allow: 200,204
>>>>> Connection: close
>>>>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>>>>> Encapsulated: null-body=0
>>>>> Methods: REQMOD
>>>>> Preview: 64000
>>>>> However in my ICAP service, I often receive a request body over 64000
>>>>> bytes, e.g. 1000000 request bytes.
>>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>>>>> service will still receive a complete HTTP REQMOD+REQRESP body.
>>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>> Alex.
>>>>> icap_enable on
>>>>> icap_connect_timeout 1 second
>>>>> icap_service_failure_limit -1
>>>>> icap_send_client_ip    on
>>>>> icap_preview_size 0
>>>>> icap_persistent_connections on
>>>>> icap_service service_req reqmod_precache bypass=0
>>>>> icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request>>
>>>>> <icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request>>>
>>>>> icap_service service_res respmod_precache bypass=0
>>>>> icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response>>
>>>>> <icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response>>>
>>>>> icap_preview_enable on
>>>>> adaptation_access service_req allow all
>>>>> adaptation_access service_res allow all