[squid-users] What is the proper way to close an ICAP transaction?

[Alex Rousskov]

On 11/26/19 4:12 PM, Felipe Arturo Polanco wrote:

> The flow is the following:
> ICAP transaction is sent to ICAP server with a PREVIEW header
> ICAP server sends ICAP header 100 Continue
> ICAP server sends ICAP header 200 OK to start data transfer

(*) Your notes below imply that the ICAP server also sends the embedded
HTTP message header back to Squid (and not just the ICAP 200 header).


> <data transfer begins>
> ICAP server receives a chunk, checks if its the last chunk, if not then 
> append to temp file and send it back to Squid;

The "send it back to Squid" part implies that not only your ICAP server
sent an ICAP 200 response header, but it sent the HTTP message header as
well. See (*) above. Once the ICAP server sent the HTTP message header,
it is committed to finish sending that HTTP message (and nothing else).


> if it is the last chunk then analyze the temp file for virus.
> <repeat for next data transfer>
> If virus found then send encapsulated HTTP header 307 redirect.

This is an ICAP service bug: One cannot send a second encapsulated HTTP
message in one ICAP response. A single ICAP response cannot contain
multiple HTTP messages. The ICAP protocol does not allow for that, and
there would be no way to actually support something like that in an ICAP
client because the HTTP message cannot be interrupted and replaced with
another mid-flight. You may assume that the HTTP client is already
seeing the beginning of the first HTTP response. The train has left the
station.

If the above is accurate, and you are using the ICAP service correctly,
then the ICAP service that allowed you to do the above is badly broken,
probably written by somebody who thought that ICAP is "easy". You may be
better off reusing an existing higher-quality ICAP service instead.


> If virus not found, send the last chunk to squid.
> 
> The part where we send 307 is the part that Squid doesn't like, I 
> believe is because we are not sending the last chunk since the file is a 
> virus.

Not exactly: Even if you send the last HTTP chunk, you would not be able
to follow up with an HTTP 307 message. The HTTP fate of this transaction
was sealed when you started trickling virgin HTTP message pieces back to
Squid (and, hence, back to the HTTP client talking to Squid).


HTH,

Alex.


> On Tue, Nov 26, 2019 at 4:52 PM Alex Rousskov wrote:
> 
>     On 11/26/19 2:52 PM, Felipe Arturo Polanco wrote:
> 
>      > We are sending an encapsulated HTTP 307 redirect webpage header
>     whenever
>      > a Virus is found and stop sending any other data after that
> 
>     You must use ICAP status code 200 then. Make sure your encapsulated HTTP
>     307 body (if any) is properly sent to Squid.
> 
> 
>      > but squid
>      > complains about ICAP failure when we do that:
>      > Adaptation::Icap::Xaction::noteCommRead threw exception:
>     corrupted chunk
>      > size
> 
>     What chunk size did Squid not like? You should be able to tell by
>     looking at the packet capture of the failed transaction (or low-level
>     Squid debugging).
> 
> 
>      > We are not sending an ICAP header at this point because we
>     already told
>      > Squid ICAP 200 OK header and begun a body transaction, we send some
>      > chunks back to the client for progress and hold the last part for
>     scanning.
> 
>     Are you sending HTTP 307 body chunks to Squid? How do you indicate that
>     no more chunks will be coming?
> 
>     It sounds like you are trying to cram two HTTP messages (one with the
>     original HTTP response body prefix and one with a generated 307
>     redirect) into one ICAP response, which is impossible, but perhaps I
>     misunderstood your description. It would help if you post a sample (but
>     complete) ICAP response that Squid does not like.
> 
> 
>      > Ideally, we would like to just send our 307 to Squid and not
>     having it
>      > count as a failure.
> 
>     Yes, a 200 ICAP response with an embedded HTTP 307 response should work
>     just fine, but all its pieces should be properly formed (and there
>     should be no extras).
> 
>     Alex.
> 
> 
>      > On Tue, Nov 26, 2019 at 3:44 PM Alex Rousskov wrote:
>      >
>      >     On 11/26/19 10:15 AM, Felipe Arturo Polanco wrote:
>      >
>      >     > While we can successfully scan our files and do content
>     adaptation, we
>      >     > have been struggling to find a way to close the ICAP
>     transaction
>      >     before
>      >     > passing the whole body back to squid and at the same time
>     avoid squid
>      >     > marking one icap failure.
>      >
>      >     Squid needs a valid ICAP response. The right ICAP response
>     status code
>      >     depends on what you want Squid to do after receiving that
>     response. You
>      >     have mentioned what you do _not_ want Squid to do (i.e.
>     increase the
>      >     failure count), but that still leaves a lot of options.
>      >
>      >
>      >     > This is for an ICAP server that does Virus scanning and if
>     virus
>      >     found,
>      >     > the body is not sent back.
>      >
>      >     What do you want Squid to do when the ICAP service finds a
>     virus? For
>      >     example, what message do you want Squid to send to the next
>     HTTP hop?
>      >
>      >     Alex.
>      >
>