[squid-users] How to extend Squid ICAP preview size

Mon Sep 16 21:26:21 UTC 2019

Thanks for that Alex.

I do have another related question regarding Squid 206 handling.

I have a RESPMOD icap server that supports Allow:206 in the OPTIONS
response.

I can see that for html webpages, squid sends Allow: 204, 206, trailers in
RESPMOD requests, and my server can handle that fine.
But for files, specifically for zip files, squid sends Allow: trailers ,
showing no support for 206. Even if I reply with a 206, I get TCP reset and
connection is closed.

Do you know why squid doesn't Allow 206 for files?
============================
Here is my configured OPTIONS response.
ICAP/1.0 200 OK
Methods: RESPMOD
Service: PyICAP Server 1.0
Preview: 0
Transfer-Complete: *
Max-Connections: 100
Options-TTL: 3600
Allow: 206
ISTag: "CmLYfciETxyLwZW3XYsPwJdCXvVehd"
Date: b'Mon', 16 b'Sep' 2019 20:50:52 GMT
Server: BaseICAP/1.0 Python/3.7.3

==================================================

And here is the RESPMOD request.
RESPMOD icap://127.0.0.1:13440/example ICAP/1.0
Host: 127.0.0.1:13440
Date: Mon, 16 Sep 2019 20:51:00 GMT
Encapsulated: req-hdr=0, res-hdr=337, res-body=579
Allow: trailers
X-Client-IP: 192.168.0.6

GET http://203.0.113.1/file.zip HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0)
Gecko/20100101 Firefox/69.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Upgrade-Insecure-Requests: 1
Host: 203.0.113.1

HTTP/1.1 200 OK
Date: Mon, 16 Sep 2019 20:51:00 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Thu, 29 Aug 2019 13:54:58 GMT
ETag: "989680-59141d9f0557a"
Accept-Ranges: bytes
Content-Length: 10000000
Content-Type: application/zip

On Mon, Sep 16, 2019 at 3:54 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 9/16/19 2:58 PM, Felipe Arturo Polanco wrote:
>
> > In our case we don't need to modify the initial 10MB, just scan it for
> > virus and if found, send a reset back to squid to not transmit the file.
>
> Yes, my original response was based on that assumption.
>
> In summary, you can ask Squid to own the 10MB prefix (i.e. do a huge
> ICAP Preview) or you can ask the ICAP server to own the 10MB prefix
> (i.e. extend ICAP 206). The choice balances various trade-offs,
> including different overheads and different risks related to Squid
> development associated with each option. Pick your poison :-).
>
> Alex.
>
>
> > On Mon, Sep 16, 2019 at 11:54 AM Alex Rousskov wrote:
> >
> >     On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote:
> >
> >     > We would like to add some logic to our custom made ICAP server,
> one of
> >     > these logics is to analyze up to 10MB of data of a given file and
> >     if the
> >     > file is larger than that, squid should not keep sending it to icap,
> >     > basically, a 204 message should be returned.
> >
> >     > squid has a cap of 64K of preview size that limit us in the aspect.
> >
> >     True.
> >
> >
> >     > Is there a way to extend this limit? Modifying the source code
> >     perhaps?
> >
> >     Yes, of course.
> >
> >
> >     > If so, what are the disadvantages of doing so?
> >
> >     Simply increasing BodyPipe::MaxCapacity might open you up to
> >     difficult-to-find vulnerabilities where some old Squid code still
> >     assumes that the buffers are limited by 64KB and crashes/asserts when
> >     that assumption becomes false. FWIW, I have not audited all
> >     BodyPipe-related code and do not know whether such bad code exists
> >     today.
> >
> >     Increasing BodyPipe::MaxCapacity will also increase Squid RAM
> >     requirements if your Squid receives requests (and/or ICAP
> >     REQMOD-generated HTTP responses) that exceed 64KB body limit. I am
> not
> >     sure about regular responses -- their accumulation is limited by
> >     other/independent restrictions.
> >
> >
> >     > Would it require more RAM per response
> >
> >     [You should worry about "per message" accumulation -- requests will
> use
> >     the same limit, even if you do not have REQMOD services.]
> >
> >     Yes, but only for messages that exceed the limit.
> >
> >
> >     > or is RAM dynamically allocated as the file is being received?
> >
> >     Yes, the RAM in question is allocated dynamically on "as needed"
> basis.
> >
> >     YMMV, but Squid may slow down a lot while dealing with the associated
> >     reallocations because of the too-small 64KB increment used for each
> such
> >     reallocation.
> >
> >
> >     An alternative solution to consider here is extending ICAP features
> to
> >     allow for splicing of the 10MB prefix (analyzed _and_ returned back
> by
> >     the ICAP server) and the remaining virgin response suffix (paused
> >     without excessive accumulation on the Squid side). This would be
> similar
> >     to the existing ICAP 206 use-original-body="10MB" feature[1], but
> would
> >     require additional negotiation and changes to disable storage of the
> >     10MB prefix on the Squid side.
> >
> >     This alternative can be implemented without worrying about hidden
> 64KB
> >     body buffer assumptions, but the implementation will not be trivial.
> >
> >     [1]
> >
> http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt
> >
> >
> >     HTH,
> >
> >     Alex.
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20190916/ec7d4154/attachment-0001.html>