<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#464646;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:#464646;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;
color:#464646;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='color:windowtext'>Hey David,<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>First, PHP is a good language however it doesn’t handle well STDIN/STDOUT helpers and crashes more then once without any warnings.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>It’s documented in the PHP docs (Don’t remember exactly where).<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Regarding PHP compared to PHP being faster, it’s pretty simple to test and validate if the cost of the crash is better than the speed.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>What python and PHP code have used in your tests? (I would be happy to try and test it..).<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>You can see this session helper written in python:<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><a href="https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper/Python">https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper/Python</a><o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>And about the cache of each helpers, the cost of a cache on a single helper is not much in terms of memory comparing to some network access.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Again it’s possible to test and verify this on a loaded system to get results. The delay itself can be seen from squid side in the cache manager statistics.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>You can also try to compare the next ruby helper:<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><a href="https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper">https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper</a><o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>About a shared “base” which allows helpers to avoid computation of the query…. It’s a good argument, however it depends what is the cost of<br>pulling from the cache compared to calculating the answer.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>A very simple string comparison or regex matching would probably be faster than reaching a shared storage in many cases.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Also take into account the “concurrency” support from the helper side.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>A helper that supports parallel processing of requests/lines can do better then many single helpers in more than once use case.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>In any case I would suggest to enable requests concurrency from squid side since the STDIN buffer will emulate some level of concurrency<br>by itself and will allow squid to keep going forward faster.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Just to mention that SquidGuard have used a single helper cache for a very long time, ie every single SquidGuard helper has it’s own copy of the whole<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>configuration and database files in memory.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>And again, if you do have any option to implement a server service model and that the helpers will contact this main service you will be able to implement<br>much faster internal in-memory cache compared to a redis/memcahe/other external daemon(need to be tested).<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>A good example for this is ufdbguard which has helpers that are clients of the main service which does the whole heavy lifting and also holds <br>one copy of the DB.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>I have implemented SquidBlocker this way and have seen that it out-performs any other service I have tried until now.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Eliezer<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='color:windowtext'>----<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Eliezer Croitoru<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>NgTech, Tech Support<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Mobile: +972-5-28704261<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Email: <a href="mailto:ngtech1ltd@gmail.com">ngtech1ltd@gmail.com</a><o:p></o:p></span></p></div><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='color:windowtext'>From:</span></b><span style='color:windowtext'> squid-users <squid-users-bounces@lists.squid-cache.org> <b>On Behalf Of </b>David Touzeau<br><b>Sent:</b> Thursday, February 3, 2022 14:24<br><b>To:</b> squid-users@lists.squid-cache.org<br><b>Subject:</b> Re: [squid-users] external helper development<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Arial",sans-serif'>Hi Elizer<br><br>You are right in a way but when squid loads multiple helpers, each helper will use its own cache.<br>Using a shared "base" allows helpers to avoid having to compute a query already found by another helper who already has the answer.<br><br>Concerning PHP what we find strange is that with our tests, a simple loop and an "echo OK", php goes faster: 1.5x than python.</span><o:p></o:p></p><div><p class=MsoNormal>Le 03/02/2022 à 07:09, Eliezer Croitoru a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre>Hey Andre,<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Every language has a "cost" for it's qualities.<o:p></o:p></pre><pre>For example, Golang is a very nice language that offers a relatively simple way for concurrency support and cross hardware compilation/compatibility.<o:p></o:p></pre><pre>One cost in Golang is that the binary is in the size of an OS/Kernel.<o:p></o:p></pre><pre>In python you must write everything in a specific position and indentation and threading is not simple to implement for a novice.<o:p></o:p></pre><pre>However when you see what was written in Python you can see that most of OpenStack api's and systems are written in.. python and it means something.<o:p></o:p></pre><pre>I like very much ruby but it doesn't support threading by nature but supports "concurrency".<o:p></o:p></pre><pre>Squid doesn't implement threading but implements "concurrency".<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Don't touch PHP as a helper!!! (+1 to Alex)<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Also take into account that Redis or Memcached is less preferred in many cases if the library doesn't re-use the existing connection for multiple queries.<o:p></o:p></pre><pre>Squid also implements caching for helpers answers so it's possible to implement the helper and ACL's in such a way that squid caching will<o:p></o:p></pre><pre>help you to lower the access to the external API and or redis/memcahced/DB.<o:p></o:p></pre><pre>I also have good experience with some libraries which implements cache that I have used inside a helper with a limited size for "level 1" cache.<o:p></o:p></pre><pre>It's possible that if you will implement both the helper and server side of the solution like ufdbguard you would be able to optimize the system<o:p></o:p></pre><pre>to take very high load.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>I hope the above will help you.<o:p></o:p></pre><pre>Eliezer<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>----<o:p></o:p></pre><pre>Eliezer Croitoru<o:p></o:p></pre><pre>NgTech, Tech Support<o:p></o:p></pre><pre>Mobile: +972-5-28704261<o:p></o:p></pre><pre>Email: <a href="mailto:ngtech1ltd@gmail.com">ngtech1ltd@gmail.com</a><o:p></o:p></pre><pre><o:p> </o:p></pre><pre>-----Original Message-----<o:p></o:p></pre><pre>From: squid-users <a href="mailto:squid-users-bounces@lists.squid-cache.org"><squid-users-bounces@lists.squid-cache.org></a> On Behalf Of André Bolinhas<o:p></o:p></pre><pre>Sent: Wednesday, February 2, 2022 00:09<o:p></o:p></pre><pre>To: 'Alex Rousskov' <a href="mailto:rousskov@measurement-factory.com"><rousskov@measurement-factory.com></a>; <a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><o:p></o:p></pre><pre>Subject: Re: [squid-users] external helper development<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Hi<o:p></o:p></pre><pre>Thanks for the reply.<o:p></o:p></pre><pre>I will take a look on Rust as you recommend.<o:p></o:p></pre><pre>Also, between Python and Go and is the best for multithreading and concurrency?<o:p></o:p></pre><pre>Rust supports multithreading and concurrency?<o:p></o:p></pre><pre>Best regards<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>-----Mensagem original-----<o:p></o:p></pre><pre>De: squid-users <a href="mailto:squid-users-bounces@lists.squid-cache.org"><squid-users-bounces@lists.squid-cache.org></a> Em Nome De Alex Rousskov<o:p></o:p></pre><pre>Enviada: 1 de fevereiro de 2022 22:01<o:p></o:p></pre><pre>Para: <a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><o:p></o:p></pre><pre>Assunto: Re: [squid-users] external helper development<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>On 2/1/22 16:47, André Bolinhas wrote:<o:p></o:p></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre>Hi<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>I’m building an external helper to get the categorization of an <o:p></o:p></pre><pre>website, I know how to build it, but I need you option about the best <o:p></o:p></pre><pre>language for the job in terms of performance, bottlenecks, I/O blocking..<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>The helper will work like this.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>1º will check the hot memory for faster response (memcache or redis)<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>2º If the result not exist in hot memory then will check an external <o:p></o:p></pre><pre>api to fetch the categorie and saved it in hot memory.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>In what language do you recommend develop such helper? PHP, Python, Go..<o:p></o:p></pre></blockquote><pre><o:p> </o:p></pre><pre>If this helper is for long-term production use, and you are willing to learn new things, then use Rust[1]. Otherwise, use whatever language you are the most comfortable with already (except PHP), especially if that language has good libraries/wrappers for the external APIs you will need to use.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Alex.<o:p></o:p></pre><pre>[1] <a href="https://www.rust-lang.org/">https://www.rust-lang.org/</a><o:p></o:p></pre><pre>_______________________________________________<o:p></o:p></pre><pre>squid-users mailing list<o:p></o:p></pre><pre><a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><o:p></o:p></pre><pre><a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><o:p></o:p></pre><pre><o:p> </o:p></pre><pre>_______________________________________________<o:p></o:p></pre><pre>squid-users mailing list<o:p></o:p></pre><pre><a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><o:p></o:p></pre><pre><a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><o:p></o:p></pre><pre><o:p> </o:p></pre><pre>_______________________________________________<o:p></o:p></pre><pre>squid-users mailing list<o:p></o:p></pre><pre><a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><o:p></o:p></pre><pre><a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><o:p></o:p></pre></blockquote><p class=MsoNormal><o:p> </o:p></p></div></body></html>