[squid-users] What is the proper way to close an ICAP transaction?

[Alex Rousskov]

On 11/27/19 11:01 AM, Felipe Arturo Polanco wrote:

> How can we then terminate an ICAP 200 OK transaction to squid without
> sending the complete body back to it? We don't want squid to mark an
> ICAP failure on us if we just close the TCP connection.

To properly answer your question, I have to come back to the question I
asked earlier: What do you want Squid to do when your ICAP service finds
a virus after trickling a few virgin HTTP body bytes to the HTTP client?

The "we want Squid to send an HTTP 307 redirect to the client" desire
you responded with earlier was ruled impossible to satisfy in your
trickling environment. A few realistic options remain though, including:

1. Terminate Squid-to-client transmission as if the whole virgin HTTP
response body was sent to the client.

2. Terminate Squid-to-client transmission while indicating (to that same
client) that the HTTP response body was cut short (i.e. that the
delivery of the response was aborted).

3. #2 plus annotate the transaction specially in Squid access.log and/or
detect this special outcome using Squid ACLs.

Alex.


> On Wed, Nov 27, 2019 at 10:54 AM Alex Rousskov wrote:
> 
>     On 11/26/19 4:12 PM, Felipe Arturo Polanco wrote:
> 
>     > The flow is the following:
>     > ICAP transaction is sent to ICAP server with a PREVIEW header
>     > ICAP server sends ICAP header 100 Continue
>     > ICAP server sends ICAP header 200 OK to start data transfer
> 
>     (*) Your notes below imply that the ICAP server also sends the embedded
>     HTTP message header back to Squid (and not just the ICAP 200 header).
> 
> 
>     > <data transfer begins>
>     > ICAP server receives a chunk, checks if its the last chunk, if not
>     then
>     > append to temp file and send it back to Squid;
> 
>     The "send it back to Squid" part implies that not only your ICAP server
>     sent an ICAP 200 response header, but it sent the HTTP message header as
>     well. See (*) above. Once the ICAP server sent the HTTP message header,
>     it is committed to finish sending that HTTP message (and nothing else).
> 
> 
>     > if it is the last chunk then analyze the temp file for virus.
>     > <repeat for next data transfer>
>     > If virus found then send encapsulated HTTP header 307 redirect.
> 
>     This is an ICAP service bug: One cannot send a second encapsulated HTTP
>     message in one ICAP response. A single ICAP response cannot contain
>     multiple HTTP messages. The ICAP protocol does not allow for that, and
>     there would be no way to actually support something like that in an ICAP
>     client because the HTTP message cannot be interrupted and replaced with
>     another mid-flight. You may assume that the HTTP client is already
>     seeing the beginning of the first HTTP response. The train has left the
>     station.
> 
>     If the above is accurate, and you are using the ICAP service correctly,
>     then the ICAP service that allowed you to do the above is badly broken,
>     probably written by somebody who thought that ICAP is "easy". You may be
>     better off reusing an existing higher-quality ICAP service instead.
> 
> 
>     > If virus not found, send the last chunk to squid.
>     >
>     > The part where we send 307 is the part that Squid doesn't like, I
>     > believe is because we are not sending the last chunk since the
>     file is a
>     > virus.
> 
>     Not exactly: Even if you send the last HTTP chunk, you would not be able
>     to follow up with an HTTP 307 message. The HTTP fate of this transaction
>     was sealed when you started trickling virgin HTTP message pieces back to
>     Squid (and, hence, back to the HTTP client talking to Squid).
> 
> 
>     HTH,
> 
>     Alex.
> 
> 
>     > On Tue, Nov 26, 2019 at 4:52 PM Alex Rousskov wrote:
>     >
>     >     On 11/26/19 2:52 PM, Felipe Arturo Polanco wrote:
>     >
>     >      > We are sending an encapsulated HTTP 307 redirect webpage header
>     >     whenever
>     >      > a Virus is found and stop sending any other data after that
>     >
>     >     You must use ICAP status code 200 then. Make sure your
>     encapsulated HTTP
>     >     307 body (if any) is properly sent to Squid.
>     >
>     >
>     >      > but squid
>     >      > complains about ICAP failure when we do that:
>     >      > Adaptation::Icap::Xaction::noteCommRead threw exception:
>     >     corrupted chunk
>     >      > size
>     >
>     >     What chunk size did Squid not like? You should be able to tell by
>     >     looking at the packet capture of the failed transaction (or
>     low-level
>     >     Squid debugging).
>     >
>     >
>     >      > We are not sending an ICAP header at this point because we
>     >     already told
>     >      > Squid ICAP 200 OK header and begun a body transaction, we
>     send some
>     >      > chunks back to the client for progress and hold the last
>     part for
>     >     scanning.
>     >
>     >     Are you sending HTTP 307 body chunks to Squid? How do you
>     indicate that
>     >     no more chunks will be coming?
>     >
>     >     It sounds like you are trying to cram two HTTP messages (one
>     with the
>     >     original HTTP response body prefix and one with a generated 307
>     >     redirect) into one ICAP response, which is impossible, but
>     perhaps I
>     >     misunderstood your description. It would help if you post a
>     sample (but
>     >     complete) ICAP response that Squid does not like.
>     >
>     >
>     >      > Ideally, we would like to just send our 307 to Squid and not
>     >     having it
>     >      > count as a failure.
>     >
>     >     Yes, a 200 ICAP response with an embedded HTTP 307 response
>     should work
>     >     just fine, but all its pieces should be properly formed (and there
>     >     should be no extras).
>     >
>     >     Alex.
>     >
>     >
>     >      > On Tue, Nov 26, 2019 at 3:44 PM Alex Rousskov wrote:
>     >      >
>     >      >     On 11/26/19 10:15 AM, Felipe Arturo Polanco wrote:
>     >      >
>     >      >     > While we can successfully scan our files and do content
>     >     adaptation, we
>     >      >     > have been struggling to find a way to close the ICAP
>     >     transaction
>     >      >     before
>     >      >     > passing the whole body back to squid and at the same time
>     >     avoid squid
>     >      >     > marking one icap failure.
>     >      >
>     >      >     Squid needs a valid ICAP response. The right ICAP response
>     >     status code
>     >      >     depends on what you want Squid to do after receiving that
>     >     response. You
>     >      >     have mentioned what you do _not_ want Squid to do (i.e.
>     >     increase the
>     >      >     failure count), but that still leaves a lot of options.
>     >      >
>     >      >
>     >      >     > This is for an ICAP server that does Virus scanning
>     and if
>     >     virus
>     >      >     found,
>     >      >     > the body is not sent back.
>     >      >
>     >      >     What do you want Squid to do when the ICAP service finds a
>     >     virus? For
>     >      >     example, what message do you want Squid to send to the next
>     >     HTTP hop?
>     >      >
>     >      >     Alex.
>     >      >
>     >
>