<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \;color\:windowtext";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#464646;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='color:windowtext'>Hey David,<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Since the handle_stdout runs in it’s own thread it’s sole purpose is to send results to stdout.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>If I will run the next code in a simple software without the 0.5 sleep time:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> while RUNNING:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if quit > 0:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> return</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> while len(queue) > 0:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> item = queue.pop(0)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> sys.stdout.write(item)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> sys.stdout.flush()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> time.sleep(0.5)</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>what will happen is that the software will run with 100% CPU looping over and over on the size of the queue<br>while sometimes it will spit some data to stdout.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Adding a small delay with 0.5 secs will allow some “idle” time for the cpu in the loop preventing it from consuming<br>all the CPU time.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>It’s a very old technique and there are others which are more efficient but it’s enough to demonstrate that a simple<br>threaded helper is much better then any PHP code that was not meant to be running as a STDIN/OUT daemon/helper software.<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:windowtext'>All The Bests,<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Eliezer<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='color:windowtext'>----<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Eliezer Croitoru<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>NgTech, Tech Support<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Mobile: +972-5-28704261<o:p></o:p></span></p><p class=MsoNormal><span style='color:windowtext'>Email: <a href="mailto:ngtech1ltd@gmail.com">ngtech1ltd@gmail.com</a><o:p></o:p></span></p></div><p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='color:windowtext'>From:</span></b><span style='color:windowtext'> David Touzeau <david@articatech.com> <br><b>Sent:</b> Monday, February 7, 2022 02:42<br><b>To:</b> Eliezer Croitoru <ngtech1ltd@gmail.com>; squid-users@lists.squid-cache.org<br><b>Subject:</b> Re: [squid-users] external helper development<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Arial",sans-serif'>Sorry Elizer<br><br>It was a mistake... No, your code is clean..<br>Impressive for the first shot<br>Many thanks for your example, we will run our stress tool to see the difference...<br><br>Just a question<br><br>Why did you send 500 milliseconds of sleep in the handle_stdoud ? Is it for let squid closing the pipe ?<br><br><br></span><o:p></o:p></p><div><p class=MsoNormal>Le 06/02/2022 à 11:46, Eliezer Croitoru a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='color:windowtext'>Hey David,</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>Not a fully completed helper but it seems to works pretty nice and might be better then what exist already:</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'><a href="https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/21ff1bbc0cf3d91719db27d9d027652e8bd3de4e/threaded-helper-example.py">https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/21ff1bbc0cf3d91719db27d9d027652e8bd3de4e/threaded-helper-example.py</a></span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>#!/usr/bin/env python</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>import sys</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>import time</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>import urllib.request</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>import signal</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>import threading</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>#set debug mode for True or False</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>debug = False</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>#debug = True</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>queue = []</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>threads = []</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>RUNNING = True</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>quit = 0</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>rand_api_url = <a href="https://cloud1.ngtech.co.il/api/test.php">"https://cloud1.ngtech.co.il/api/test.php"</a></span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>def sig_handler(signum, frame):</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> sys.stderr.write("Signal is received:" + str(signum) + "\n")</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> global quit</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> quit = 1</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> global RUNNING</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> RUNNING=False</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>def handle_line(line):</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if not RUNNING:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> return</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if not line:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> return</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if quit > 0:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> return</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> arr = line.split()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> response = urllib.request.urlopen( rand_api_url )</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> response_text = response.read()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> queue.append(arr[0] + " " + response_text.decode("utf-8"))</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>def handle_stdout(n):</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> while RUNNING:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if quit > 0:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> return</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> while len(queue) > 0:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> item = queue.pop(0)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> sys.stdout.write(item)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> sys.stdout.flush()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> time.sleep(0.5)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>def handle_stdin(n):</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> while RUNNING:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> line = sys.stdin.readline()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if not line:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> break</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> if quit > 0:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> break</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> line = line.strip()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> thread = threading.Thread(target=handle_line, args=(line,))</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> thread.start()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> threads.append(thread)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>signal.signal(signal.SIGUSR1, sig_handler)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>signal.signal(signal.SIGUSR2, sig_handler)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>signal.signal(signal.SIGALRM, sig_handler)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>signal.signal(signal.SIGINT, sig_handler)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>signal.signal(signal.SIGQUIT, sig_handler)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>signal.signal(signal.SIGTERM, sig_handler)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>stdout_thread = threading.Thread(target=handle_stdout, args=(1,))</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>stdout_thread.start()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>threads.append(stdout_thread)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>stdin_thread = threading.Thread(target=handle_stdin, args=(2,))</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>stdin_thread.start()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>threads.append(stdin_thread)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>while(RUNNING):</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> time.sleep(3)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>print("Not RUNNING")</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>for thread in threads:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'> thread.join()</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New ;color:windowtext",serif'>print("All threads stopped.")</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>## END</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>Eliezer</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'> </span><o:p></o:p></p><div><p class=MsoNormal><span style='color:windowtext'>----</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>Eliezer Croitoru</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>NgTech, Tech Support</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>Mobile: +972-5-28704261</span><o:p></o:p></p><p class=MsoNormal><span style='color:windowtext'>Email: <a href="mailto:ngtech1ltd@gmail.com">ngtech1ltd@gmail.com</a></span><o:p></o:p></p></div><p class=MsoNormal><span style='color:windowtext'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='color:windowtext'>From:</span></b><span style='color:windowtext'> squid-users <a href="mailto:squid-users-bounces@lists.squid-cache.org"><squid-users-bounces@lists.squid-cache.org></a> <b>On Behalf Of </b>David Touzeau<br><b>Sent:</b> Friday, February 4, 2022 16:29<br><b>To:</b> <a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><br><b>Subject:</b> Re: [squid-users] external helper development</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>Elizer,<br><br>Thanks for all this advice and indeed your arguments are valid between opening a socket, sending data, receiving data and closing the socket unlike direct access to a regex or a memory entry even if the calculation has already been done.<br><br>But what surprises me the most is that we have produced a python plugin in thread which I provide you a code below. <br>The php code is like your mentioned example ( No thread, just a loop and output OK ) <br><br>Results are after 6k requests, squid freeze and no surf can be made as with PHP code we can up to 10K requests and squid is happy<br>really, we did not understand why python is so low.<br><br><span style='font-family:"Arial",sans-serif'>Here a python code using threads<br><br>#!/usr/bin/env python<br>import os<br>import sys<br>import time<br>import signal<br>import locale<br>import traceback<br>import threading<br>import select<br>import traceback as tb<br><br>class ClienThread():<br><br> def __init__(self):<br> self._exiting = False<br> self._cache = {}<br><br> def exit(self):<br> self._exiting = True<br><br> def stdout(self, lineToSend):<br> try:<br> sys.stdout.write(lineToSend)<br> sys.stdout.flush()<br><br> except IOError as e:<br> if e.errno==32:<br> # Error Broken PIPE!"<br> pass<br> except:<br> # other execpt<br> pass<br><br> def run(self):<br> while not self._exiting:<br> if sys.stdin in select.select([sys.stdin], [], [], 0.5)[0]:<br> line = sys.stdin.readline()<br> LenOfline=len(line)<br><br> if LenOfline==0:<br> self._exiting=True<br> break<br><br> if line[-1] == '\n':line = line[:-1]<br> channel = None<br> options = line.split()<br><br> try:<br> if options[0].isdigit(): channel = options.pop(0)<br> except IndexError:<br> self.stdout("0 OK first=ERROR\n")<br> continue<br><br> # Processing here<br><br> try:<br> self.stdout("%s OK\n" % channel)<br> except:<br> self.stdout("%s ERROR first=ERROR\n" % channel)<br><br><br><br><br>class Main(object):<br> def __init__(self):<br> self._threads = []<br> self._exiting = False<br> self._reload = False<br> self._config = ""<br><br> for sig, action in (<br> (signal.SIGINT, self.shutdown),<br> (signal.SIGQUIT, self.shutdown),<br> (signal.SIGTERM, self.shutdown),<br> (signal.SIGHUP, lambda s, f: setattr(self, '_reload', True)),<br> (signal.SIGPIPE, signal.SIG_IGN),<br> ):<br> try:<br> signal.signal(sig, action)<br> except AttributeError:<br> pass<br><br><br><br> def shutdown(self, sig = None, frame = None):<br> self._exiting = True<br> self.stop_threads()<br><br> def start_threads(self):<br><br> sThread = ClienThread()<br> t = threading.Thread(target = sThread.run)<br> t.start()<br> self._threads.append((sThread, t))<br><br><br><br> def stop_threads(self):<br> for p, t in self._threads:<br> p.exit()<br> for p, t in self._threads:<br> t.join(timeout = 1.0)<br> self._threads = []<br><br> def run(self):<br> """ main loop """<br> ret = 0<br> self.start_threads()<br> return ret<br><br><br>if __name__ == '__main__':<br> # set C locale<br> locale.setlocale(locale.LC_ALL, 'C')<br> os.environ['LANG'] = 'C'<br> ret = 0<br> try:<br> main = Main()<br> ret = main.run()<br> except SystemExit:<br> pass<br> except KeyboardInterrupt:<br> ret = 4<br> except:<br> sys.exit(ret)</span><o:p></o:p></p><div><p class=MsoNormal>Le 04/02/2022 à 07:06, Eliezer Croitoru a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>And about the cache of each helpers, the cost of a cache on a single helper is not much in terms of memory comparing to some network access.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>Again it’s possible to test and verify this on a loaded system to get results. The delay itself can be seen from squid side in the cache manager statistics.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>You can also try to compare the next ruby helper:</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'><a href="https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper">https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>About a shared “base” which allows helpers to avoid computation of the query…. It’s a good argument, however it depends what is the cost of<br>pulling from the cache compared to calculating the answer.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>A very simple string comparison or regex matching would probably be faster than reaching a shared storage in many cases.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>Also take into account the “concurrency” support from the helper side.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>A helper that supports parallel processing of requests/lines can do better then many single helpers in more than once use case.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>In any case I would suggest to enable requests concurrency from squid side since the STDIN buffer will emulate some level of concurrency<br>by itself and will allow squid to keep going forward faster.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>Just to mention that SquidGuard have used a single helper cache for a very long time, ie every single SquidGuard helper has it’s own copy of the whole</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>configuration and database files in memory.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>And again, if you do have any option to implement a server service model and that the helpers will contact this main service you will be able to implement<br>much faster internal in-memory cache compared to a redis/memcahe/other external daemon(need to be tested).</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>A good example for this is ufdbguard which has helpers that are clients of the main service which does the whole heavy lifting and also holds <br>one copy of the DB.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:windowtext'>I have implemented SquidBlocker this way and have seen that it out-performs any other service I have tried until now.</span><o:p></o:p></p></blockquote><p class=MsoNormal> <o:p></o:p></p></blockquote><p class=MsoNormal><o:p> </o:p></p></div></body></html>