[squid-users] How to extend Squid ICAP preview size

[Alex Rousskov]

On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote:

> We would like to add some logic to our custom made ICAP server, one of
> these logics is to analyze up to 10MB of data of a given file and if the
> file is larger than that, squid should not keep sending it to icap,
> basically, a 204 message should be returned.

> squid has a cap of 64K of preview size that limit us in the aspect.

True.


> Is there a way to extend this limit? Modifying the source code perhaps?

Yes, of course.


> If so, what are the disadvantages of doing so? 

Simply increasing BodyPipe::MaxCapacity might open you up to
difficult-to-find vulnerabilities where some old Squid code still
assumes that the buffers are limited by 64KB and crashes/asserts when
that assumption becomes false. FWIW, I have not audited all
BodyPipe-related code and do not know whether such bad code exists today.

Increasing BodyPipe::MaxCapacity will also increase Squid RAM
requirements if your Squid receives requests (and/or ICAP
REQMOD-generated HTTP responses) that exceed 64KB body limit. I am not
sure about regular responses -- their accumulation is limited by
other/independent restrictions.


> Would it require more RAM per response 

[You should worry about "per message" accumulation -- requests will use
the same limit, even if you do not have REQMOD services.]

Yes, but only for messages that exceed the limit.


> or is RAM dynamically allocated as the file is being received?

Yes, the RAM in question is allocated dynamically on "as needed" basis.

YMMV, but Squid may slow down a lot while dealing with the associated
reallocations because of the too-small 64KB increment used for each such
reallocation.


An alternative solution to consider here is extending ICAP features to
allow for splicing of the 10MB prefix (analyzed _and_ returned back by
the ICAP server) and the remaining virgin response suffix (paused
without excessive accumulation on the Squid side). This would be similar
to the existing ICAP 206 use-original-body="10MB" feature[1], but would
require additional negotiation and changes to disable storage of the
10MB prefix on the Squid side.

This alternative can be implemented without worrying about hidden 64KB
body buffer assumptions, but the implementation will not be trivial.

[1]
http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt


HTH,

Alex.