[squid-users] Need help about ICAP scan timeout/max file size for big files

[Amos Jeffries]

On 4/01/19 11:38 pm, info at schroeffu.ch wrote:
> 
> Hi all,
> 
> i am trying to solve the problem, that SQUID is caching all the big
> files (for example 1GB) before sending them to the client, but the
> connected ICAP virus scanner is configured with max_file_size 2MB and
> scan_timeout 5 seconds. So all bigger files, or longer scanning times,
> should result in "clean" state from the icap virus scanner.
> 
> I am running antivirus FSIGK (F-Secure Internet GateKeeper) as an ICAP
> daemon connected to Squid with this configuration:
> 
> #ICAP
> icap_enable on
> acl domains_dont_icapscan url_regex -i
> "/etc/squid/ka/domains_dont_icapscan.acl"
> acl audio rep_mime_type -i
> ^(audio\/x-mpegurl|audio\/mpeg|audio\/ogg|audio\/aac|audio/mp3)$
> 
> icap_service service_req reqmod_precache bypass=1
> icap://127.0.0.1:1344/request
> adaptation_access service_req allow !domains_dont_icapscan

> icap_service service_resp respmod_precache bypass=1
> icap://127.0.0.1:1344/response

> adaptation_access service_resp allow !domains_dont_icapscan !audio

The above line says that everything which is not *both* an audio file
and on your dont-scan list does get scanned.

In other words, you can only whitelist audio responses.


> 
> Detecting viruses is working, but downloading large files is a huge
> problem. Squid is downloading them completely first into the servers
> memory and caching them,> before sending them to the client. Its not stop
> scanning & caching after 2MB/5Seconds.

Squid is not scanning. The whole point of ICAP is that something *else*
is doing the content manipulation/scanning.

Squid is just a relaying the content octets blindly between the various
agents using it.


> When downloading big files (f.e.
> 1gb) the browser just does nothing but waiting a long time, because
> squid is downloading and caching 1gb before forward to client.

The amount of memory used will depend on other config settings which you
have not shown. Please provide all your config so we can analyze the
problem in full context of what is going on around these ICAP services.


> 
> I tried change respmod_precache to respmod_postcache but it seems not to
> be implemented yet, with respmod_postcache fsigk icap log is empty , no
> virus detection works anymore.

Correct. Post-cache ICAP hooks are not supported/implemented by Squid.

If scanning it once (pre-cache) is slow then scanning it per-fetch / N
times (aka post-cache) would be at least N times slower.



> I have a test-virus-file with 100MB
> (https://schroeffu.ch/100mbrandomvirus_begin.txt eicar+randomcontent)
> and the virus is detected by fsigk with settings max_scan_size=104400136
> / scan_timeout=9000 , change them to max_scan_size=2147483 (2mb) and
> scan_timeout=5 (5Seconds) the virus is correctly not detected anymore,

These sound like config setting for the scanning operation. None of that
has any relevance to Squid.

The file is a TXT file not an audio file, so as far as Squid can tell it
is always to be delivered to the scanner.


> but, squid still does cache the 100mb before sending to the client.
> 
> How can I configure the ICAP Service to truly let bigger files/longer
> scan times through the icap service marked as "clean"?


What you describe sounds like problems identified with ClamAV early on.
It turned out clam was storing the object to disk and waiting for it to
complete before scanning and providing any output to Squid.

Please check whether the scanner you are using does that type of behaviour.


Amos