<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#464646" bgcolor="#FFFFFF">
<font face="Arial">You are the best, <br>
We will launch a benchmark to see the diff<br>
</font><br>
<div class="moz-cite-prefix">Le 07/02/2022 à 16:14, Eliezer Croitoru
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:001701d81c35$77894980$669bdc80$@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:"Courier New \;color\:windowtext";
panose-1:0 0 0 0 0 0 0 0 0 0;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#464646;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Hey David,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Since the
handle_stdout runs in it’s own thread it’s sole purpose is
to send results to stdout.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">If I will
run the next code in a simple software without the 0.5 sleep
time:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> while RUNNING:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if quit > 0:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> return</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> while len(queue)
> 0:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> item =
queue.pop(0)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">
sys.stdout.write(item)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">
sys.stdout.flush()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> time.sleep(0.5)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">what will
happen is that the software will run with 100% CPU looping
over and over on the size of the queue<br>
while sometimes it will spit some data to stdout.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Adding a
small delay with 0.5 secs will allow some “idle” time for
the cpu in the loop preventing it from consuming<br>
all the CPU time.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">It’s a very
old technique and there are others which are more efficient
but it’s enough to demonstrate that a simple<br>
threaded helper is much better then any PHP code that was
not meant to be running as a STDIN/OUT daemon/helper
software.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">All The
Bests,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Eliezer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:windowtext">----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Eliezer
Croitoru<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">NgTech,
Tech Support<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Mobile:
+972-5-28704261<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Email: <a
href="mailto:ngtech1ltd@gmail.com"
moz-do-not-send="true" class="moz-txt-link-freetext">ngtech1ltd@gmail.com</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> David Touzeau
<a class="moz-txt-link-rfc2396E" href="mailto:david@articatech.com"><david@articatech.com></a> <br>
<b>Sent:</b> Monday, February 7, 2022 02:42<br>
<b>To:</b> Eliezer Croitoru
<a class="moz-txt-link-rfc2396E" href="mailto:ngtech1ltd@gmail.com"><ngtech1ltd@gmail.com></a>;
<a class="moz-txt-link-abbreviated" href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><br>
<b>Subject:</b> Re: [squid-users] external helper
development<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-family:"Arial",sans-serif">Sorry
Elizer<br>
<br>
It was a mistake... No, your code is clean..<br>
Impressive for the first shot<br>
Many thanks for your example, we will run our stress tool to
see the difference...<br>
<br>
Just a question<br>
<br>
Why did you send 500 milliseconds of sleep in the
handle_stdoud ? Is it for let squid closing the pipe ?<br>
<br>
<br>
</span><o:p></o:p></p>
<div>
<p class="MsoNormal">Le 06/02/2022 à 11:46, Eliezer Croitoru a
écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:windowtext">Hey David,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Not a
fully completed helper but it seems to works pretty nice
and might be better then what exist already:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"><a
href="https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/21ff1bbc0cf3d91719db27d9d027652e8bd3de4e/threaded-helper-example.py"
moz-do-not-send="true" class="moz-txt-link-freetext">https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/21ff1bbc0cf3d91719db27d9d027652e8bd3de4e/threaded-helper-example.py</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">#!/usr/bin/env python</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">import sys</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">import time</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">import urllib.request</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">import signal</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">import threading</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">#set debug mode for True or
False</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">debug = False</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">#debug = True</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">queue = []</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">threads = []</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">RUNNING = True</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">quit = 0</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">rand_api_url = <a
href="https://cloud1.ngtech.co.il/api/test.php"
moz-do-not-send="true">"https://cloud1.ngtech.co.il/api/test.php"</a></span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">def sig_handler(signum,
frame):</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">
sys.stderr.write("Signal is received:" + str(signum) +
"\n")</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> global quit</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> quit = 1</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> global RUNNING</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> RUNNING=False</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">def handle_line(line):</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if not RUNNING:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> return</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if not line:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> return</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if quit > 0:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> return</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> arr = line.split()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> response =
urllib.request.urlopen( rand_api_url )</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> response_text =
response.read()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> queue.append(arr[0] +
" " + response_text.decode("utf-8"))</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">def handle_stdout(n):</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> while RUNNING:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if quit > 0:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> return</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> while len(queue)
> 0:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> item =
queue.pop(0)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">
sys.stdout.write(item)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">
sys.stdout.flush()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> time.sleep(0.5)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">def handle_stdin(n):</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> while RUNNING:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> line =
sys.stdin.readline()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if not line:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> break</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> if quit > 0:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> break</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> line =
line.strip()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> thread =
threading.Thread(target=handle_line, args=(line,))</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> thread.start()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">
threads.append(thread)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">signal.signal(signal.SIGUSR1,
sig_handler)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">signal.signal(signal.SIGUSR2,
sig_handler)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">signal.signal(signal.SIGALRM,
sig_handler)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">signal.signal(signal.SIGINT,
sig_handler)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">signal.signal(signal.SIGQUIT,
sig_handler)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">signal.signal(signal.SIGTERM,
sig_handler)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">stdout_thread =
threading.Thread(target=handle_stdout, args=(1,))</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">stdout_thread.start()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">threads.append(stdout_thread)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">stdin_thread =
threading.Thread(target=handle_stdin, args=(2,))</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">stdin_thread.start()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">threads.append(stdin_thread)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">while(RUNNING):</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> time.sleep(3)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">print("Not RUNNING")</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">for thread in threads:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif"> thread.join()</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
;color:windowtext",serif">print("All threads
stopped.")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">## END</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Eliezer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:windowtext">----</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Eliezer
Croitoru</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">NgTech,
Tech Support</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Mobile:
+972-5-28704261</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Email: <a
href="mailto:ngtech1ltd@gmail.com"
moz-do-not-send="true" class="moz-txt-link-freetext">ngtech1ltd@gmail.com</a></span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> squid-users <a
href="mailto:squid-users-bounces@lists.squid-cache.org"
moz-do-not-send="true"><squid-users-bounces@lists.squid-cache.org></a>
<b>On Behalf Of </b>David Touzeau<br>
<b>Sent:</b> Friday, February 4, 2022 16:29<br>
<b>To:</b> <a
href="mailto:squid-users@lists.squid-cache.org"
moz-do-not-send="true" class="moz-txt-link-freetext">squid-users@lists.squid-cache.org</a><br>
<b>Subject:</b> Re: [squid-users] external helper
development</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Elizer,<br>
<br>
Thanks for all this advice and indeed your arguments are
valid between opening a socket, sending data, receiving data
and closing the socket unlike direct access to a regex or a
memory entry even if the calculation has already been done.<br>
<br>
But what surprises me the most is that we have produced a
python plugin in thread which I provide you a code below. <br>
The php code is like your mentioned example ( No thread,
just a loop and output OK ) <br>
<br>
Results are after 6k requests, squid freeze and no surf can
be made as with PHP code we can up to 10K requests and squid
is happy<br>
really, we did not understand why python is so low.<br>
<br>
<span style="font-family:"Arial",sans-serif">Here
a python code using threads<br>
<br>
#!/usr/bin/env python<br>
import os<br>
import sys<br>
import time<br>
import signal<br>
import locale<br>
import traceback<br>
import threading<br>
import select<br>
import traceback as tb<br>
<br>
class ClienThread():<br>
<br>
def __init__(self):<br>
self._exiting = False<br>
self._cache = {}<br>
<br>
def exit(self):<br>
self._exiting = True<br>
<br>
def stdout(self, lineToSend):<br>
try:<br>
sys.stdout.write(lineToSend)<br>
sys.stdout.flush()<br>
<br>
except IOError as e:<br>
if e.errno==32:<br>
# Error Broken PIPE!"<br>
pass<br>
except:<br>
# other execpt<br>
pass<br>
<br>
def run(self):<br>
while not self._exiting:<br>
if sys.stdin in select.select([sys.stdin], [],
[], 0.5)[0]:<br>
line = sys.stdin.readline()<br>
LenOfline=len(line)<br>
<br>
if LenOfline==0:<br>
self._exiting=True<br>
break<br>
<br>
if line[-1] == '\n':line = line[:-1]<br>
channel = None<br>
options = line.split()<br>
<br>
try:<br>
if options[0].isdigit(): channel =
options.pop(0)<br>
except IndexError:<br>
self.stdout("0 OK first=ERROR\n")<br>
continue<br>
<br>
# Processing here<br>
<br>
try:<br>
self.stdout("%s OK\n" % channel)<br>
except:<br>
self.stdout("%s ERROR first=ERROR\n" %
channel)<br>
<br>
<br>
<br>
<br>
class Main(object):<br>
def __init__(self):<br>
self._threads = []<br>
self._exiting = False<br>
self._reload = False<br>
self._config = ""<br>
<br>
for sig, action in (<br>
(signal.SIGINT, self.shutdown),<br>
(signal.SIGQUIT, self.shutdown),<br>
(signal.SIGTERM, self.shutdown),<br>
(signal.SIGHUP, lambda s, f: setattr(self,
'_reload', True)),<br>
(signal.SIGPIPE, signal.SIG_IGN),<br>
):<br>
try:<br>
signal.signal(sig, action)<br>
except AttributeError:<br>
pass<br>
<br>
<br>
<br>
def shutdown(self, sig = None, frame = None):<br>
self._exiting = True<br>
self.stop_threads()<br>
<br>
def start_threads(self):<br>
<br>
sThread = ClienThread()<br>
t = threading.Thread(target = sThread.run)<br>
t.start()<br>
self._threads.append((sThread, t))<br>
<br>
<br>
<br>
def stop_threads(self):<br>
for p, t in self._threads:<br>
p.exit()<br>
for p, t in self._threads:<br>
t.join(timeout = 1.0)<br>
self._threads = []<br>
<br>
def run(self):<br>
""" main loop """<br>
ret = 0<br>
self.start_threads()<br>
return ret<br>
<br>
<br>
if __name__ == '__main__':<br>
# set C locale<br>
locale.setlocale(locale.LC_ALL, 'C')<br>
os.environ['LANG'] = 'C'<br>
ret = 0<br>
try:<br>
main = Main()<br>
ret = main.run()<br>
except SystemExit:<br>
pass<br>
except KeyboardInterrupt:<br>
ret = 4<br>
except:<br>
sys.exit(ret)</span><o:p></o:p></p>
<div>
<p class="MsoNormal">Le 04/02/2022 à 07:06, Eliezer Croitoru
a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">And about the cache of each
helpers, the cost of a cache on a single helper is not
much in terms of memory comparing to some network
access.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">Again it’s possible to test and
verify this on a loaded system to get results. The delay
itself can be seen from squid side in the cache manager
statistics.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">You can also try to compare the
next ruby helper:</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"><a
href="https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper"
moz-do-not-send="true" class="moz-txt-link-freetext">https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper</a></span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">About a shared “base” which
allows helpers to avoid computation of the query…. It’s
a good argument, however it depends what is the cost of<br>
pulling from the cache compared to calculating the
answer.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">A very simple string comparison
or regex matching would probably be faster than reaching
a shared storage in many cases.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">Also take into account the
“concurrency” support from the helper side.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">A helper that supports parallel
processing of requests/lines can do better then many
single helpers in more than once use case.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">In any case I would suggest to
enable requests concurrency from squid side since the
STDIN buffer will emulate some level of concurrency<br>
by itself and will allow squid to keep going forward
faster.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">Just to mention that SquidGuard
have used a single helper cache for a very long time, ie
every single SquidGuard helper has it’s own copy of the
whole</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">configuration and database
files in memory.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">And again, if you do have any
option to implement a server service model and that the
helpers will contact this main service you will be able
to implement<br>
much faster internal in-memory cache compared to a
redis/memcahe/other external daemon(need to be tested).</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">A good example for this is
ufdbguard which has helpers that are clients of the main
service which does the whole heavy lifting and also
holds <br>
one copy of the DB.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:windowtext">I have implemented SquidBlocker
this way and have seen that it out-performs any other
service I have tried until now.</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"> <o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
squid-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a>
<a class="moz-txt-link-freetext" href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a>
</pre>
</blockquote>
<br>
</body>
</html>