[squid-dev] Fake CONNECT requests during SSL Bump

[Eliezer Croitoru]

* Now I see that the question was sent to your private email instead of 
squid-dev.

So after you have confirmed my assumption I can understand why actually 
the external_acl helpers interface do not slow down squid.
So for example I wrote a helper in golang 
[https://github.com/elico/squidblocker-client] which can actually handle 
a huge amount of requests per second utilizing the native concurrency of 
golang.
I have tried in the past to compare perl\python\ruby and other 
interpreted languages and they seem to lack the simplicity to handle 
concurrent request.
And I must say that when I am considering now in what language to use 
for a helper then just because of the concurrency simplicity compared to 
other languages I will first try golang.

I have replaced squidguard with my client and DB services and using the 
external_acl helper interface I managed to handle what more then 40 
squidguard helpers was required for with barely 5 helpers in golang.
I am still struggling with a way to handle ICAP requests properly in 
golang but now that you have mentioned threading in a C++ program it 
just gave me a rejection feeling.
I do not want to say bad things about C++ because I see how it is such a 
great language but for now C++ threading kind of frighten me and I 
assume that it's a blocker for me to try and do something in it.

And back to the acl eCAP module\service.
I have seen something in the adapter_async about threading but haven't 
had the chance to actually read it not talking about understanding it.
I also cannot think about an advanced acl adaptation service that will 
not use some DB or other socket based code(which by nature blocks).
For example basic authorization such as mysql, LDAP, session, RADIUS.
(As a side note I want to list eCAP modules that are open source at 
github that I know about:
- https://github.com/maxpmaxp/ecap-exif
- https://github.com/creamy/ecap-mongo
- https://github.com/c-rack/squid-ecap-gzip
- https://github.com/billowkiller/ecap
- https://github.com/Securepoint/squid-ecap-av
- https://github.com/hynnet/ecap_clamav_adapter
)

So even converting any existing squid helpers will be turned into a non 
simple task since most of the current helpers by default do not 
implement concurrency at all.

So to sum all the above:
If I would have been asked to choose what to use in order to implement 
some acl helper I would prefer(based on my current knowledge in 
programming and CS in general)
1 - external_acl helper (done already)
2 - ICAP service (done for GET request and struggling with other methods 
and bodies handling)
3 - eCAP service (It is actually writing a boxed C++ application aka 
module inside another service and just mentioning threading rejects me)

Eliezer

On 27/09/2015 02:36, Alex Rousskov wrote:
> On 09/26/2015 04:52 PM, Eliezer Croitoru wrote:
>
>> if I will run some DB action inside an eCAP module it
>> should be done in a very specific way to not block the whole squid
>> process or to use an ICAP service instead.
>>
>> I wanted to verify that I am right on the assumption.
>
>
> You are correct:
>
> * The ICAP client implementation in Squid does not block the whole Squid
> process because communication is done using non-blocking socket I/O.
> What happens on the other side of that ICAP connection depends on the
> ICAP server implementation. Some ICAP servers never create threads, some
> create a thread per transaction, some support a pool of threads, and
> some let ICAP adapters decide how to handle each transaction.
>
> * The eCAP client (a.k.a. "host") implementation has to block the whole
> host process because it has to use regular C++ function calls. What
> happens on the other side of that function call is the eCAP adapter
> responsibility. If your adapter can return from all eCAP function calls
> "fast", there is no need to worry (and you are saving a lot by not
> having to go over a TCP connection). If you cannot answer some of the
> host questions "immediately", then you have to implement an async eCAP
> adapter that uses threads or other non-blocking mechanisms to shield the
> host application from "slow" or "blocking" adapter code.
>
>
> Cheers,
>
> Alex.
>