[squid-users] How to extend Squid ICAP preview size
Felipe Arturo Polanco
felipeapolanco at gmail.com
Mon Sep 16 21:26:21 UTC 2019
Thanks for that Alex.
I do have another related question regarding Squid 206 handling.
I have a RESPMOD icap server that supports Allow:206 in the OPTIONS
I can see that for html webpages, squid sends Allow: 204, 206, trailers in
RESPMOD requests, and my server can handle that fine.
But for files, specifically for zip files, squid sends Allow: trailers ,
showing no support for 206. Even if I reply with a 206, I get TCP reset and
connection is closed.
Do you know why squid doesn't Allow 206 for files?
Here is my configured OPTIONS response.
ICAP/1.0 200 OK
Service: PyICAP Server 1.0
Date: b'Mon', 16 b'Sep' 2019 20:50:52 GMT
Server: BaseICAP/1.0 Python/3.7.3
And here is the RESPMOD request.
RESPMOD icap://127.0.0.1:13440/example ICAP/1.0
Date: Mon, 16 Sep 2019 20:51:00 GMT
Encapsulated: req-hdr=0, res-hdr=337, res-body=579
GET http://203.0.113.1/file.zip HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0)
Accept-Encoding: gzip, deflate
HTTP/1.1 200 OK
Date: Mon, 16 Sep 2019 20:51:00 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Thu, 29 Aug 2019 13:54:58 GMT
On Mon, Sep 16, 2019 at 3:54 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:
> On 9/16/19 2:58 PM, Felipe Arturo Polanco wrote:
> > In our case we don't need to modify the initial 10MB, just scan it for
> > virus and if found, send a reset back to squid to not transmit the file.
> Yes, my original response was based on that assumption.
> In summary, you can ask Squid to own the 10MB prefix (i.e. do a huge
> ICAP Preview) or you can ask the ICAP server to own the 10MB prefix
> (i.e. extend ICAP 206). The choice balances various trade-offs,
> including different overheads and different risks related to Squid
> development associated with each option. Pick your poison :-).
> > On Mon, Sep 16, 2019 at 11:54 AM Alex Rousskov wrote:
> > On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote:
> > > We would like to add some logic to our custom made ICAP server,
> one of
> > > these logics is to analyze up to 10MB of data of a given file and
> > if the
> > > file is larger than that, squid should not keep sending it to icap,
> > > basically, a 204 message should be returned.
> > > squid has a cap of 64K of preview size that limit us in the aspect.
> > True.
> > > Is there a way to extend this limit? Modifying the source code
> > perhaps?
> > Yes, of course.
> > > If so, what are the disadvantages of doing so?
> > Simply increasing BodyPipe::MaxCapacity might open you up to
> > difficult-to-find vulnerabilities where some old Squid code still
> > assumes that the buffers are limited by 64KB and crashes/asserts when
> > that assumption becomes false. FWIW, I have not audited all
> > BodyPipe-related code and do not know whether such bad code exists
> > today.
> > Increasing BodyPipe::MaxCapacity will also increase Squid RAM
> > requirements if your Squid receives requests (and/or ICAP
> > REQMOD-generated HTTP responses) that exceed 64KB body limit. I am
> > sure about regular responses -- their accumulation is limited by
> > other/independent restrictions.
> > > Would it require more RAM per response
> > [You should worry about "per message" accumulation -- requests will
> > the same limit, even if you do not have REQMOD services.]
> > Yes, but only for messages that exceed the limit.
> > > or is RAM dynamically allocated as the file is being received?
> > Yes, the RAM in question is allocated dynamically on "as needed"
> > YMMV, but Squid may slow down a lot while dealing with the associated
> > reallocations because of the too-small 64KB increment used for each
> > reallocation.
> > An alternative solution to consider here is extending ICAP features
> > allow for splicing of the 10MB prefix (analyzed _and_ returned back
> > the ICAP server) and the remaining virgin response suffix (paused
> > without excessive accumulation on the Squid side). This would be
> > to the existing ICAP 206 use-original-body="10MB" feature, but
> > require additional negotiation and changes to disable storage of the
> > 10MB prefix on the Squid side.
> > This alternative can be implemented without worrying about hidden
> > body buffer assumptions, but the implementation will not be trivial.
> > 
> > HTH,
> > Alex.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the squid-users