[squid-users] Regex optimization

Thu Jun 16 19:11:50 UTC 2016

Well.. I tried.
I need to ban 8613 URLs. Because a law.

If I put one per line in a file and set the filename for an url_regex acl
it works. But when the traffic goes up the cpu load goes 100% (even using
workers) and the proxy turns unusable.

I tested and saw my squid can't parse regexes with more than 8192
characters.
I managed to combine the 8000 uris in 34 regexes using a ruby gem, and the
cpu load stays almost at the same level it is without any acl (same
traffic).

The problem is that of the 34 regexes 33 works fine and 1 matches
everything.

"http://something" matches.

the regex is:

^http:\/\/(m(i(n(i(\.free\-teen\-pussy\.com\/scj\/thumbs\/(0\/(253_Mini_Mini\.jpg|306_Mini_Mini\.jpg|468_Pree_Pree\.jpg|519_Models_Models\.jpg|688_Models_nn\.jpg|7(17_Models_Models\.jpg|22_Models_Mini\.jpg)|880_nn_Models\.jpg)|1\/(170_little_models\.jpg|271_Young_Young\.jpg|330_models_little\.jpg|412_little_models\.jpg|6(28_little_models\.jpg|43_little_little\.jpg)|742_Young_Young\.jpg|877_littlemodels_littlemodels\.jpg|903_littlemodels_littlemodels\.jpg)|2\/(0(47_models_little\.jpg|92_models_models\.jpg)|1(44_little_little\.jpg|75_Young_Young\.jpg)|2(04_littlemodels_littlemodels\.jpg|29_russian_russian\.jpg|76_russian_little\.jpg)))|girls\.biz)|t(ladies\.com\/|teens\.com\/))|ragetube\.com\/content\/(35\/907_un_pequena\.jpg|75\/939_sex_Simpsons\.jpg|95\/473_Babe_039\.jpg))|mmgay\.com\/|o(dels(\.world\-collections\.com\/index\.html\?(4[69]|7(34|7)|88)|\-(hot\.net\/|me\.com\/index\.html\?(23|838)))|e(img(1\.moesearch\.net\/imgs1\/01058000\/1058747\.jpg|2\.moesearch\.net\/imgs2\/(0(1058000\/1058604\.jpg|2095000\/2095432\.jpg|4027000\/4027262\.jpg)|12509000\/12509478\.jpg))|search\.net\/img\.php\?mode=view&id=1058747)|ist\.lolajunior\.com\/images\/cache\/240x180\/(1(00\.125\.1231425553\.jpg|11\.85\.1921663855\.jpg|40\.76\.1735890287\.jpg|50\.115\.1167057586\.jpg|6(1\.133\.2028798891\.jpg|6\.130\.165837020\.jpg|7\.111\.1548016707\.jpg)|72\.107\.88564122\.jpg)|2(42\.142\.777367129\.jpg|76\.170\.1873449959\.jpg)|95\.80\.959860653\.jpg)|n(dayporn\.com\/f\/teen\/|stercams\.in\/)|ppetdollz\.com\/|r(asmovie\.com\/|e\-porn\.net\/)|ther\.taboo\.cc\/images\/cache\/300x250\/191\.170\.32154737\.jpg|v(\.(18\-21\-teens\.com\/|ftvcash\.com\/Free\-Movies\/galleries\/FTV\/1\/165\/)|ie(1820\.com\/trifuns\/sabine\.61sec\.shtml|galls(1(\.t(eensexreality\.com\/665\/\?nats=MjgwOjI6NA|inseks\.com\/video2\/070\/\?nats=MjI6Mjo2,0,0,0,5506)|2\.teenburg\.com\/262\-265\/265\/070\/\?nats=Mjc6Mjox)|3\.teen(burg\.com\/videos15\/093\/\?nats=MTc6Mjox,0,0,0,4396|sexreality\.com\/videos3\/042\/\?nats=MjU6Mjo0))|pornshop\.com\/?|sroom\.net\/index2\.shtml)))|pxgirls\.com\/stream\/thumbs\/(196\/196614\.jpg|96\/96855\.jpg)|rpornsite\.com\/tube\/teen\/|snwhores\.com\/|u(lti(\.xnxx\.com\/gallery\/132750\/ad59\/|grab\.olimptraffic\.com\/thumbs\/(003a\/8e8675a8f2fb40dfedbde11abedb4685\.jpg|new(\/00(02\/1fe44498d20b8ca368b23b517f03af47\.jpg|14\/07df98c7a77ab7069dc236273948fe7d\.jpg|37\/a209f10ecc66fa95c4e5d0c56c6eb6b5\.jpg|52\/0a61ea933a3b7a5febcfc94f3e225c98\.jpg)|2\/0(0(07\/0bae6ad1630ca38eafe01383eeb0d860\.jpg|89\/(86365a52f500fd038170116e4026fe94\.jpg|a9bb7df30ede34bb568978495968ed7b\.jpg))|1(36\/e00a6e199a1655fb757448e857b63607\.jpg|59\/3434c873b5544bfef18c58518b89eb81\.jpg)))))|n\-da\.com\/\?ref=nuvilon\.com|s(etoons\.com\/|icnet\.sil\.at\/phpbb\/viewtopic\.php\?p=35246\#35246)|ychicas\.com\.ar\/imgoct2013\/17\.jpeg)|y(18teens\.com\/(movies\/(amazing\-teenage\-whore\-kacey\-gives\-head\/\?nats=NTM6Mjox,0,0,0,10461|brenda\-blowjob\/\?nats=NjY1OjI6MQ|gorgeous\-teen\-girl\-katya\-pornmovies\/\?nats=NTM6Mjox,0,0,0,10507|teenage\-model\-gina\-porn\-clips\/\?nats=NTM6Mjox,0,0,0,10492)|pictures\/(naked\-teen\-hoe\-sabrina\/\?nats=NTM6Mjox,0,0,0,10440|sweet\-teen\-babe\-sabrina\/\?nats=NTM6Mjox,0,0,0,10431|teen\-fuck\-movie\/\?nats=MTI6Mjox,0,0,0,1189))|3(dsex\.com\/|xxx\.com\/(dtr\/galls\/(03104b\/|6(0298a\/|64bda\/)))?)|boyssite\.com\/\?id=10gay\.com|crazyvids\.com\/mov\/best\/petite\-1\.html|e(mogirl\.com\/|xbaby\.com\/)|firstteens\.com\/|\-fruits\.info\/topics\/(HOT%20NEW%203D%20SITE%20!!%20Photos%20of%20young%20nonude%20models%20in%203D%20!%20%28Anaglyph%20and%20MPO%29\/|More%20young%20talents%20Vinka%20Model\/|NEW%20star%20Tiana%20Model\/|Preteen%20art%20from%20Talent%20Young%20%28NEW%29\/|S(ummer%20days%20\-%20best%20nonude%20preteen%20photos%20and%20videos%20from%20summer%20beaches\/|veta%20with%20piggy%20tails\/)|Talent%20Young%20very%20HOT%20preteens%20collection%20%28NEW%29\/)|idealgirl\.com\/archives\/1|juniorsister\.com\/videos\/(0(29\.jpg|61\.jpg)|1(3(7\.jpg|8\.jpg)|54\.jpg|88\.jpg)|2(4(4\.jpg|6\.jpg)|89\.jpg|98\.jpg)|375\.jpg)|s(ilverteens\.com\/index\.php|luts\.net\/(\?x=7106\.8911\.3297)?)|teen(ie\.com\/|pics\.co\.tv\/tgp\/|tv\.com\/|video\.com\/pictures\/(amateur\-teenage\-girl\-nesti\-sex\/\?nats=bogk84q9;827:notrials50:myteenvideo,0,0,0,19752|sweet\-teen\-babe\-aletta\/\?nats=NTM6Mjoz,0,0,0,10608))|youngsex\.com\/))|n(a(dia(31\.firstmo\.com\/|kristina\.firstmo\.com\/thumbs\/(0(5\.jpg|6\.jpg|8\.jpg)|1(2\.jpg|6\.jpg|7\.jpg|8\.jpg)|2(0\.jpg|1\.jpg)))|ked(\-(girls\-vids\.com\/|nude\.com\/|photos\.org\/)|teenspictures\.com\/str\/thumbs\/7\/7(168\.jpg|485\.jpg))|na\.legalmodolls\.com\/promo\/240x460_02\.jpg|onlinedee\.qipim\.ru\/twocj\.html|rutoepisodeporno\.com\/|sty(\.preteennonude\.com\/(\?ref=preteennonude\.com&x=9622\.8898\.|pic\/pic\.php\?285)|teensdesire\.com\/)|turistteenphotos\.com\/th\/thumb034\.jpg|ughtyteenvids\.com\/\?id=kind\-girls\.net)|c\-mcadd\.org\/model\-girls\-preteen\/tn\/(20952\.jpg|4086\.jpg|7905\.jpg)|e(mo\-glamour\.com\/|odet\.com\/\?id=(nonude\-modelcom|topnonude\-girlsinfo|virginstubenet)|w(\.(18onlygirls\.com\/4234c2e9\/MTE3NjU6NToyNA\/|beataporn\.com\/a31412b5\/Njg1MDo1OjMx\/|nonudeplace\.com\/(aida\.holymodels\.jpg|bella\.holymodels\.jpg|evie\.holymodels\.jpg|g(irls(1\.holymodels\.jpg|3\.holymodels\.jpg)|lenda\.holymodels\.jpg)|kamilla\.holymodels\.jpg|lia\.holymodels\.jpg|scj\/tmp\/0\/(2(29_preteen_model\.jpg|62_nonude_glenda\.jpg)|351_lolita_nude\.jpg|4(02_non_preteen\.jpg|50_prelolitas_prelolitas\.jpg)|511_nonude_pics\.jpg|6(03_lolita_lolita\.jpg|85_prelolitas_prelolitas\.jpg)|7(27_nn_model\.jpg|63_nude_preteen\.jpg)))|younglegalporn\.com\/(3f68aa4a\/Njg1MDo1OjI1\/|ee0598ed\/MzcyODo1OjI1\/))|\-(art\-nude\.net\/streamrotator\/thumbs\/(0\/(234\.jpg|32\.jpg|493\.jpg)|59\/59594\.jpg|65\/65515\.jpg)|content\.info\/cj\/g\/index\.php\?ft=gaybase\.biz|models\.in\/scj\/thumbs\/0\/(576_0053_0053\.jpg|733_From_From\.jpg)|teensex\.com\/(\?facename=indexpdl&updatestat=off)?)|estpornlinks\.com\/|n(nmodels\.com\/scj\/thumbs\/0\/300_Star_Preten\.jpg|ubiles\.net\/)|pornsite(\.org\/(\?x=0058\.4384\.|vids\/x09\/05\/26\/A\-nice\-Blowjob\/index\.html)?|s\.net\/)|yes\.moy\.su\/sixcj\.html)|zemnoykaif\.com\/(\?d=p1)?)|fsx\.com\/(\?x=(0693\.6976\.0182\.9567\.9074\.4120\.5480\.1010\.9410\.5849|2504\.9354\.5283\.5928\.5849\.|4113\.5849\.|5849\.?|7119\.5849nfsx\.com\/\?x=7119\.5849\.))?|i(c(exvideostube\.com\/dtr\/thumbs\/(0(82339\.jpg|92429\.jpg)|3c2af8\.jpg|4c9c39\.jpg|6(2ba29\.jpg|61782\.jpg)|76803f\.jpg|97c126\.jpg|b(46397\.jpg|65702\.jpg)|c(0ab58\.jpg|3d805\.jpg|9065b\.jpg)|d(799f4\.jpg|8b70c\.jpg))|hegalz\.com\/moviesgeneral\.php\?d=etvtube\.com&acc=arkan&in2=)|ghttubes\.com\/|neclips\.com\/(watch\.php\?tag=0glreYaIS9)?|vonog\.zonalibre\.org\/archives\/015324\.html)|n(\-(1\.com\/(ban_468\.jpg)?|art\.info\/(image\/8\.jpg|portal\/main\/index_files\/tn_(1\.jpg|reyna25\.jpg))|b(bs\.info\/(100150\.gif|ban\.gif)|eauties\.com\/)|girls\.biz\/sitesx\/(2\/stella\.jpg|4\/(bella\.jpg|evie\.jpg)|5\/mashamodel\.jpg)|l(ist\.in\/scj\/thumbs\/0\/394_and_5\.jpg|ola\.net\/index\.html\?44)|mods\.com\/index\.html\?36|sites\.com\/(nnmodels(3\.html|4\.html))?|t(een\.thumblogger\.com\/|op\.com\/100x150\.jpg))|100\.in\/(\?id=simply\-models\.net|s(cj\/thumbs\/1\/(3(09_Elba_Pretty\.jpg|31_young_models\.jpg)|68(3_Beautiful_Mara\.jpg|9_Hot_Young\.jpg)|7(03_Model_0071\.jpg|29_Other_Girls\.jpg|57_Model_Girls\.jpg|91_From_Models\.jpg)|8(28_From_Vilma\.jpg|43_Models_Models\.jpg|51_0068_5\.jpg|76_Model_Mara\.jpg))|ites\/(lila\/Lila\-ind\-400x600\.jpg|natasha\/Natasha\-ind\-400x600\.jpg)))|desire\.com\/bannersmall\.gif|elis\.com\/|lopics\.com\/(PC\-tn(alina6b\.jpg|daphne3b\.jpg|roxana2\-2b\.jpg))?|neversleeps\.info\/|preteen\.com\/index\.html\?18|t(eens\.com\/tgp\/my18teens\-arielfuck\/bunny\.htm|op\.org\/(100150\.gif|miniban\/inna\.jpg))|ville\.com\/(\?(ref=n(onudemodel\.net|uvilon\.com)|x=5608\.9251\.9012\.6426\.)|GALL\/art\/trixie\/)?)|o(chesexo\.com\/|n(\-nude\.tv\/non\-nude_room\/151\-3_gb_private_colection_jb_bonus\.html|nudesitescatalog\.com\/banner\.gif|stop\-nn\.info\/(\?ref=newnnmodels\.net|100150\.jpg|galls\/mnrv\/More%20preteen%20nonude%20models%20Dany%20and%20Camy\/)))))

Testing the regex in regex101 it should not match.

My squid was compiles with gnu regex

Squid Cache: Version 3.5.17
Service Name: squid
configure options:  '--prefix=/opt/sepia/distro/squid'
'--sysconfdir=/var/lib/sepia/' '--disable-auth' '--disable-auto-locale'
'--disable-cache-digests' '--disable-cpu-profiling'
'--disable-debug-cbdata' '--disable-delay-pools' '--disable-devpoll'
'--disable-ecap' '--disable-esi' '--disable-eui'
'--disable-external-acl-helpers' '--disable-follow-x-forwarded-for'
'--disable-forw-via-db' '--enable-gnuregex' '--disable-htcp'
'--enable-icap-client' '--disable-ident-lookups' '--enable-internal-dns'
'--disable-ipf-transparent' '--disable-ipfw-transparent' '--disable-ipv6'
'--disable-leakfinder' '--disable-pf-transparent' '--disable-poll'
'--disable-select' '--disable-snmp' '--with-openssl'
'--disable-stacktraces' '--disable-translation'
'--disable-url-rewrite-helpers' '--disable-wccp' '--disable-wccpv2'
'--disable-win32-service' '--disable-x-accelerator-vary' '--disable-icmp'
'--disable-storeid-rewrite-helpers' '--enable-async-io' '--enable-disk-io'
'--enable-epoll' '--enable-http-violations' '--enable-inline'
'--enable-kill-parent-hack' '--enable-linux-netfilter'
'--enable-log-daemon-helpers' '--enable-removal-policies'
'--enable-storeio' '--enable-unlinkd' '--enable-x-accelerator-vary'
'--enable-zph-qos' '--with-default-user=nobody'
'--with-logdir=/var/log/sepia' '--with-pthreads' '--with-included-ltdl'
'--with-pidfile=/var/lib/sepia/squid.pid' '--with-netfilter-conntrack'
'--disable-arch-native' --enable-ltdl-convenience

Testing with more lines of smaller regexes sometimes leads to the same
problem. One regex matching everything

2016-04-27 8:32 GMT-03:00 Alfredo Rezinovsky <alfrenovsky at gmail.com>:

> I saw in debug log that when an ACL has many regexes each one is compared
> sequentially.
>
> If I have
>
> www.facebook.com
> facebook.com
> www.google.com
> google.com
>
> If will be faster to check just ONE optimized regex like
> (www\.)?(facebook|google).com than the previous three?
>
> I'm really talking about optimizing about 3000 url regexes in one huge
> regex because comparing each and every url to 3000 regexes is too slow.
>
> I know using
> (www\.facebook\.com)|(facebook\.com)|(www\.google\.com)|(google\.com) with
> PCRE will produce the same optimized result as
> (www\.)?(facebook|google)\.com. Squid uses GnuRegex. Does GNURegex lib
> optimizes this as well ?
>
>
> --
> Alfrenovsky
>

-- 
Alfrenovsky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160616/37882326/attachment-0001.html>