[squid-users] How to extend Squid ICAP preview size

[Felipe Arturo Polanco]

Thanks for the detailed response Alex.

In our case we don't need to modify the initial 10MB, just scan it for
virus and if found, send a reset back to squid to not transmit the file.

Do you have any additional recommendation for such case?

Thanks,



On Mon, Sep 16, 2019 at 11:54 AM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote:
>
> > We would like to add some logic to our custom made ICAP server, one of
> > these logics is to analyze up to 10MB of data of a given file and if the
> > file is larger than that, squid should not keep sending it to icap,
> > basically, a 204 message should be returned.
>
> > squid has a cap of 64K of preview size that limit us in the aspect.
>
> True.
>
>
> > Is there a way to extend this limit? Modifying the source code perhaps?
>
> Yes, of course.
>
>
> > If so, what are the disadvantages of doing so?
>
> Simply increasing BodyPipe::MaxCapacity might open you up to
> difficult-to-find vulnerabilities where some old Squid code still
> assumes that the buffers are limited by 64KB and crashes/asserts when
> that assumption becomes false. FWIW, I have not audited all
> BodyPipe-related code and do not know whether such bad code exists today.
>
> Increasing BodyPipe::MaxCapacity will also increase Squid RAM
> requirements if your Squid receives requests (and/or ICAP
> REQMOD-generated HTTP responses) that exceed 64KB body limit. I am not
> sure about regular responses -- their accumulation is limited by
> other/independent restrictions.
>
>
> > Would it require more RAM per response
>
> [You should worry about "per message" accumulation -- requests will use
> the same limit, even if you do not have REQMOD services.]
>
> Yes, but only for messages that exceed the limit.
>
>
> > or is RAM dynamically allocated as the file is being received?
>
> Yes, the RAM in question is allocated dynamically on "as needed" basis.
>
> YMMV, but Squid may slow down a lot while dealing with the associated
> reallocations because of the too-small 64KB increment used for each such
> reallocation.
>
>
> An alternative solution to consider here is extending ICAP features to
> allow for splicing of the 10MB prefix (analyzed _and_ returned back by
> the ICAP server) and the remaining virgin response suffix (paused
> without excessive accumulation on the Squid side). This would be similar
> to the existing ICAP 206 use-original-body="10MB" feature[1], but would
> require additional negotiation and changes to disable storage of the
> 10MB prefix on the Squid side.
>
> This alternative can be implemented without worrying about hidden 64KB
> body buffer assumptions, but the implementation will not be trivial.
>
> [1]
>
> http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt
>
>
> HTH,
>
> Alex.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20190916/accd2b04/attachment.html>