[squid-dev] Streaming ICAP previews

[Steve Hill]

This is probably best directed at the ICAP Forum, but www.i-cap.org 
seems to be dead.

There seems to be a limitation in the ICAP specification that I've just 
hit, and would be trivial to fix, so wanted to have some input into 
whether this has previously been discussed (I can't see any mention in 
the measurement-factory errata page).

RFC3507 section 4.5 says:
"100 Continue.  If the entire encapsulated HTTP body did not fit in the 
preview, the ICAP client MUST send the remainder of its ICAP message, 
starting from the first chunk after the preview.  If the entire message 
fit in the preview (detected by the "EOF" symbol explained below), then 
the ICAP server MUST NOT respond with 100 Continue."

The "MUST NOT" part prevents streaming of content during the preview, 
because until the end of the preview the ICAP server does not know 
whether a 100 Continue response will be allowed.

Take the following scenario, with the ICAP server expecting 4096 byte 
previews:

The HTTP client makes a request via the proxy, and the server responds 
with an HTTP chunked request that slowly streams 16 byte chunks of data 
back at a rate of 1 chunk per second.

The proxy sends an ICAP RESPMOD request to the ICAP server, starts a 
4096 byte preview, and as it receives the chunked data from the HTTP 
server it streams that data to the ICAP server.  At 16 bytes per second, 
the preview will take a little over 4 minutes to complete.  During that 
4 minutes, the ICAP server can respond with "204 No Content" to allow 
the content, or it can produce an entirely new HTTP response of its own 
to replace the original response.

However, if the ICAP server wants to stream the original response back 
to the client (with or without any modifications), it can't start doing 
this until after the preview is complete because it has to wait until 
the end of the preview to find out whether or not to send a 100 Continue.

In our example, this means the client doesn't start seeing the response 
from the server for over 4 minutes.

This fix for this seems obvious - in my opinion, the RFC should read:
"100 Continue.  If the entire encapsulated HTTP body did not fit in the 
preview, the ICAP client MUST send the remainder of its ICAP message, 
starting from the first chunk after the preview.  If the entire message 
fit in the preview (detected by the "EOF" symbol explained below), then 
the ICAP client MUST ignore a 100 Continue."

This change should be straight forward to implement on the client side, 
and allows the server to start streaming at any point during the preview 
by blindly sending a "100 Continue" followed by the ICAP reqmod/respmod 
response:
  - If the HTTP response does not fit in the preview, the client would 
see the 100 Continue and continue to send the whole response to the ICAP 
server after the preview is complete, then wait for the reqmod/respmod 
response.
  - If the HTTP response does not fit in the preview, the client would 
ignore the 100 Continue and wait for the reqmod/respmod response.

In either case, the end of the preview is still marked by the chunk body 
terminator, and the server knows whether to expect further non-preview 
chunks by the presence or absence of the ieof chunk extension.

-- 
  - Steve Hill
    Technical Director | Cyfarwyddwr Technegol
    Opendium    Online Safety & Web Filtering     http://www.opendium.com
                Diogelwch Ar-Lein a Hidlo Gwefan

    Enquiries | Ymholiadau:   sales at opendium.com     +44-1792-824568
    Support   | Cefnogi:      support at opendium.com   +44-1792-825748

------------------------------------------------------------------------
Opendium Limited is a company registered in England and Wales.
Mae Opendium Limited yn gwmni sydd wedi'i gofrestru yn Lloegr a Chymru.

Company No. | Rhif Cwmni:   5465437
Highfield House, 1 Brue Close, Bruton, Somerset, BA10 0HY, England.