[squid-users] How to extend Squid ICAP preview size

[Alex Rousskov]

On 9/16/19 2:58 PM, Felipe Arturo Polanco wrote:

> In our case we don't need to modify the initial 10MB, just scan it for
> virus and if found, send a reset back to squid to not transmit the file.

Yes, my original response was based on that assumption.

In summary, you can ask Squid to own the 10MB prefix (i.e. do a huge
ICAP Preview) or you can ask the ICAP server to own the 10MB prefix
(i.e. extend ICAP 206). The choice balances various trade-offs,
including different overheads and different risks related to Squid
development associated with each option. Pick your poison :-).

Alex.


> On Mon, Sep 16, 2019 at 11:54 AM Alex Rousskov wrote:
> 
>     On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote:
> 
>     > We would like to add some logic to our custom made ICAP server, one of
>     > these logics is to analyze up to 10MB of data of a given file and
>     if the
>     > file is larger than that, squid should not keep sending it to icap,
>     > basically, a 204 message should be returned.
> 
>     > squid has a cap of 64K of preview size that limit us in the aspect.
> 
>     True.
> 
> 
>     > Is there a way to extend this limit? Modifying the source code
>     perhaps?
> 
>     Yes, of course.
> 
> 
>     > If so, what are the disadvantages of doing so?
> 
>     Simply increasing BodyPipe::MaxCapacity might open you up to
>     difficult-to-find vulnerabilities where some old Squid code still
>     assumes that the buffers are limited by 64KB and crashes/asserts when
>     that assumption becomes false. FWIW, I have not audited all
>     BodyPipe-related code and do not know whether such bad code exists
>     today.
> 
>     Increasing BodyPipe::MaxCapacity will also increase Squid RAM
>     requirements if your Squid receives requests (and/or ICAP
>     REQMOD-generated HTTP responses) that exceed 64KB body limit. I am not
>     sure about regular responses -- their accumulation is limited by
>     other/independent restrictions.
> 
> 
>     > Would it require more RAM per response
> 
>     [You should worry about "per message" accumulation -- requests will use
>     the same limit, even if you do not have REQMOD services.]
> 
>     Yes, but only for messages that exceed the limit.
> 
> 
>     > or is RAM dynamically allocated as the file is being received?
> 
>     Yes, the RAM in question is allocated dynamically on "as needed" basis.
> 
>     YMMV, but Squid may slow down a lot while dealing with the associated
>     reallocations because of the too-small 64KB increment used for each such
>     reallocation.
> 
> 
>     An alternative solution to consider here is extending ICAP features to
>     allow for splicing of the 10MB prefix (analyzed _and_ returned back by
>     the ICAP server) and the remaining virgin response suffix (paused
>     without excessive accumulation on the Squid side). This would be similar
>     to the existing ICAP 206 use-original-body="10MB" feature[1], but would
>     require additional negotiation and changes to disable storage of the
>     10MB prefix on the Squid side.
> 
>     This alternative can be implemented without worrying about hidden 64KB
>     body buffer assumptions, but the implementation will not be trivial.
> 
>     [1]
>     http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt
> 
> 
>     HTH,
> 
>     Alex.
>