[squid-users] multiple log_file_daemon settings in squid.conf

[Amos Jeffries]

On 20/03/18 23:38, Rafael Akchurin wrote:
> Greetings all,
> 
> I am trying to find the best (easiest, least interfering) solution for
> the following problem.
> 
> Our custom ICAP server writes various information about ICAP transaction
> (user name, policy ip, detection module, timings, words triggered
> detection, etc) into the record database. This happens at the time when
> ICAP transaction ends. Based on that information we build extensive
> reports of web filtering activity.
> 
>  
> 
> As known, end of the ICAP transaction does not mean the end of the
> original transaction in Squid – e.g. after scanning CONNECT request and
> allowing it to proceed, the actual data transfer from the Squid’s
> perspective may end much much later. The piece of information that would
> be very interesting for reporting module is how many bytes were pumped
> through that connection. This information is only available after the
> original Squid transaction ends.
> 
>  
> 
> So somehow in the record database we must correlate the ICAP
> transaction(s) with original Squid transaction.
> 
>  
> 
> Question 1:
> 
> Is there any unique transaction ID in the Squid’s inner workings that I
> can see in the ICAP server, by passing it as additional X-* ICAP header?
> 
> I see some references to so called “master transaction” in the docs but
> could not find any log format like identifier that can be used for ICAP
> header value.

see <http://www.squid-cache.org/Doc/config/adaptation_meta/>

There is also the note directive. Although that is only added right at
the point of logging - so it is useful for Squid adding notes if the
ICAP service does not add any itself. All Squid helpers used for a
transaction add notes about their results to the transaction (except
security tokens for NTLM, Kerberos and Digest).
 <http://www.squid-cache.org/Doc/config/note/>


> 
> Question 2:
> 
> If there is no such transaction ID, I can use ICAP header to pass the
> ICAP specific transaction ID back to Squid **and** I can get that ID
> written to Squid’s access log as
> "X-WebSafety-IID=%{X-WebSafety-IID}adapt::<last_h". Is it a valid approach?
> 

Yes.


>  
> 
> Question 3:
> 
> If q1 or q2 is answered positively, I still need to somehow get the data
> from squid’s access log the ICAP record database. Currently the idea is
> to have the custom logfile_daemon setting that would fork original
> log_file_daemon to have log entries written to access_log **and** parse
> out the ICAP ID **and** update the corresponding ICAP record in the
> database with transferred bytes information. But this seems complex and
> fragile.
> 

There is only one daemon at a time supported.

That said the daemon can be instructed via access_log parameter
"daemon:PLACE" to send its data to a specific output PLACE - where PLACE
is a string passed to the helper command line and interpreted in any way
the helper author desires.
 The bundled helpers treat is respectively as the name+path of a file,
or a database DSN to drop the relevant access_log lines to.


Alternatively the "tcp:" logging module can send the log lines as a raw
TCP stream to some place that handles input with the logformat defined
syntax as a series of lines.
 I've seen people defining custom logformat with variously SQL, XML and
JSON markup for delivery to custom daemon or tcp modules.


There is also the option of an external ACL helper that gets passed the
%note details along with anything else you wish it to use. That helper
can do any updates you want prior to the logformat using the same
details. The only restriction here is that it must run in a slow-group
access check. The last one that seems reasonable is http_reply_access
which preceeds ICAP RESPMOD alterations.


Amos