<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#464646" bgcolor="#FFFFFF">
Elizer,<br>
<br>
Thanks for all this advice and indeed your arguments are valid
between opening a socket, sending data, receiving data and closing
the socket unlike direct access to a regex or a memory entry even if
the calculation has already been done.<br>
<br>
But what surprises me the most is that we have produced a python
plugin in thread which I provide you a code below. <br>
The php code is like your mentioned example ( No thread, just a loop
and output OK ) <br>
<br>
Results are after 6k requests, squid freeze and no surf can be made
as with PHP code we can up to 10K requests and squid is happy<br>
really, we did not understand why python is so low.<br>
<br>
<font face="Arial">Here a python code using threads<br>
<br>
#!/usr/bin/env python<br>
import os<br>
import sys<br>
import time<br>
import signal<br>
import locale<br>
import traceback<br>
import threading<br>
import select<br>
import traceback as tb<br>
<br>
class ClienThread():<br>
<br>
def __init__(self):<br>
self._exiting = False<br>
self._cache = {}<br>
<br>
def exit(self):<br>
self._exiting = True<br>
<br>
def stdout(self, lineToSend):<br>
try:<br>
sys.stdout.write(lineToSend)<br>
sys.stdout.flush()<br>
<br>
except IOError as e:<br>
if e.errno==32:<br>
# Error Broken PIPE!"<br>
pass<br>
except:<br>
# other execpt<br>
pass<br>
<br>
def run(self):<br>
while not self._exiting:<br>
if sys.stdin in select.select([sys.stdin], [], [],
0.5)[0]:<br>
line = sys.stdin.readline()<br>
LenOfline=len(line)<br>
<br>
if LenOfline==0:<br>
self._exiting=True<br>
break<br>
<br>
if line[-1] == '\n':line = line[:-1]<br>
channel = None<br>
options = line.split()<br>
<br>
try:<br>
if options[0].isdigit(): channel =
options.pop(0)<br>
except IndexError:<br>
self.stdout("0 OK first=ERROR\n")<br>
continue<br>
<br>
# Processing here<br>
<br>
try:<br>
self.stdout("%s OK\n" % channel)<br>
except:<br>
self.stdout("%s ERROR first=ERROR\n" %
channel)<br>
<br>
<br>
<br>
<br>
class Main(object):<br>
def __init__(self):<br>
self._threads = []<br>
self._exiting = False<br>
self._reload = False<br>
self._config = ""<br>
<br>
for sig, action in (<br>
(signal.SIGINT, self.shutdown),<br>
(signal.SIGQUIT, self.shutdown),<br>
(signal.SIGTERM, self.shutdown),<br>
(signal.SIGHUP, lambda s, f: setattr(self, '_reload',
True)),<br>
(signal.SIGPIPE, signal.SIG_IGN),<br>
):<br>
try:<br>
signal.signal(sig, action)<br>
except AttributeError:<br>
pass<br>
<br>
<br>
<br>
def shutdown(self, sig = None, frame = None):<br>
self._exiting = True<br>
self.stop_threads()<br>
<br>
def start_threads(self):<br>
<br>
sThread = ClienThread()<br>
t = threading.Thread(target = sThread.run)<br>
t.start()<br>
self._threads.append((sThread, t))<br>
<br>
<br>
<br>
def stop_threads(self):<br>
for p, t in self._threads:<br>
p.exit()<br>
for p, t in self._threads:<br>
t.join(timeout = </font><font face="Arial"><font
face="Arial"> 1.0</font>)<br>
self._threads = []<br>
<br>
def run(self):<br>
""" main loop """<br>
ret = 0<br>
self.start_threads()<br>
return ret<br>
<br>
<br>
if __name__ == '__main__':<br>
# set C locale<br>
locale.setlocale(locale.LC_ALL, 'C')<br>
os.environ['LANG'] = 'C'<br>
ret = 0<br>
try:<br>
main = Main()<br>
ret = main.run()<br>
except SystemExit:<br>
pass<br>
except KeyboardInterrupt:<br>
ret = 4<br>
except:<br>
sys.exit(ret)<br>
</font><br>
<div class="moz-cite-prefix">Le 04/02/2022 à 07:06, Eliezer Croitoru
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:000401d8198d$5c30e120$1492a360$@gmail.com">
<p class="MsoNormal"><span style="color:windowtext">And about the
cache of each helpers, the cost of a cache on a single helper
is not much in terms of memory comparing to some network
access.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Again it’s
possible to test and verify this on a loaded system to get
results. The delay itself can be seen from squid side in the
cache manager statistics.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">You can also
try to compare the next ruby helper:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><a
href="https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper"
moz-do-not-send="true" class="moz-txt-link-freetext">https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">About a shared
“base” which allows helpers to avoid computation of the
query…. It’s a good argument, however it depends what is the
cost of<br>
pulling from the cache compared to calculating the answer.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">A very simple
string comparison or regex matching would probably be faster
than reaching a shared storage in many cases.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Also take into
account the “concurrency” support from the helper side.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">A helper that
supports parallel processing of requests/lines can do better
then many single helpers in more than once use case.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">In any case I
would suggest to enable requests concurrency from squid side
since the STDIN buffer will emulate some level of concurrency<br>
by itself and will allow squid to keep going forward faster.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">Just to
mention that SquidGuard have used a single helper cache for a
very long time, ie every single SquidGuard helper has it’s own
copy of the whole<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">configuration
and database files in memory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">And again, if
you do have any option to implement a server service model and
that the helpers will contact this main service you will be
able to implement<br>
much faster internal in-memory cache compared to a
redis/memcahe/other external daemon(need to be tested).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">A good example
for this is ufdbguard which has helpers that are clients of
the main service which does the whole heavy lifting and also
holds <br>
one copy of the DB.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">I have
implemented SquidBlocker this way and have seen that it
out-performs any other service I have tried until now.</span></p>
</blockquote>
<br>
</body>
</html>