[squid-users] Squid full request logging

[Alex Rousskov]

On 3/4/21 4:33 PM, Niels Hofmans wrote:

> Although I am not proficient in C for writing
> an ecap service, is there some binding available online for Go?

eCAP is C++ and, unfortunately, the first version of its API evidently
requires elevated programming skills. I am not aware of Go bindings, and
I would not recommend eCAP for those who are not fluent in C++ (or
cannot hire somebody fluent in C++).


> This was the reason I originally opted for an ICAP service since I can
> abstract Go behind the HTTP ICAP layer.

I do not know what "abstract Go behind the HTTP ICAP layer" means, but I
hope that you are not writing an ICAP service from scratch. If you do,
be prepared to battle many problems, both short- and long-term. ICAP
looks simple but it is difficult to get right.


> Now I understand this has its limitations, but AFAIK a preview cap at
> 100kb would be sufficient per request.

I am not sure Squid supports 100KB previews -- the internal buffers may
not be big enough for that. 64KB might be the limit, but do not quote me
on that.

BTW, please note that, AFAICT, Squid ignores icap_preview_size in
squid.conf (a bug) -- your ICAP service has to request the desired
maximum preview size via an ICAP OPTIONS response.


> Would you think that ... using two ICAP services.
> One would validate the headers and return OK or NOT (bypass=0), while
> the other only pushes the 1kb request/response to a queue.
> Ideally those two would be contacted simultaneously while only the first
> one is blocking.
> ..just thinking aloud tough.

Sorry, I cannot evaluate this design because I do not know what you want
to optimize and what your logging requirements/limitations are.


Good luck,

Alex.


> On 4 Mar 2021, at 22:23, Alex Rousskov wrote:
> 
> On 3/4/21 2:52 PM, Niels Hofmans wrote:
> 
>> is it possible to do full request/response logging?
> 
> Squid can log HTTP headers with %>h and %<h logformat codes.
> 
> Squid cannot log HTTP message bodies.
> 
> 
>> I do not see the appropriate log_format directive in the docs.
>> I was hoping not having to do this in my ICAP service since this slows
>> down approval of the HTTP request. (Empty preview v.s. a request capped
>> at 1MB that needs to be sent over every time)
> 
> FWIW, an ICAP or eCAP service can start responding to the request
> _before_ the service receives the entire HTTP message body. To get
> things going, all the service needs is HTTP headers (and even that is,
> technically, optional in some cases).
> 
> Using an adaptation service is still an overhead, of course, but, very
> few legitimate Squid use cases involve logging message bodies, so there
> is no built-in mechanism optimized for that specific rare purpose
> (yet?). The fastest option available today is probably a dedicated eCAP
> service that refuses to adapt the message bit continues to receive (and
> log) the message body.
> 
> 
> HTH,
> 
> Alex.
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org <mailto:squid-users at lists.squid-cache.org>
> http://lists.squid-cache.org/listinfo/squid-users
>