[squid-users] assertion failed: comm.cc:178: "fd_table[conn->fd].halfClosedReader != NULL"
Amos Jeffries
squid3 at treenet.co.nz
Fri May 1 13:05:22 UTC 2015
On 1/05/2015 10:09 p.m., HackXBack wrote:
> squid.conf you can see it all , and the answer on your question is no i dont
> have .
>
Okay, a bit of an audit...
>
> # should be allowed
> acl localnet src 10.11.20.0/24
> acl localnet src 10.150.15.0/24
>
> # ACL for rewriter
> acl fakespeed url_regex -i
> \.*(speedtest|espeed).*\/((latency|random.*|upload)\.(jpg|txt|php)).*
The trailing ".*" at the end of this is not useful.
> acl rewriter-link url_regex -i
> ^http.*(google|googlesyndication)\.com\/(pagead|js)\/(bg|js)\/.*\.js
>
> # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
> #
> -----------------------------------------------------------------------------
> cache_peer 10.11.20.100 parent 80 0
> cache_peer_access 10.11.20.100 allow fakespeed
> cache_peer_access 10.11.20.100 deny all
>
> # OPTIONS INFLUENCING REQUEST FORWARDING
> #
> -----------------------------------------------------------------------------
> never_direct allow fakespeed
> never_direct deny all
> always_direct deny fakespeed
>
>
> # add on squid.conf to remove ads
> ########################
> acl ads-block url_regex -i "/etc/squid/ads.block"
> http_access deny ads-block
> http_reply_access deny ads-block
Since you are denying requests being made to these URL the only possible
reply will be the "Access Denied" error page.
You are then denying that error page being delivered, and replacing it
with ... the "Access Denied" error page.
Sigh.
Hint #1: Your Squid would operate a bit faster by removing that
"http_reply_access deny ads-block" line.
Hint #2: it is a deny using slow regex pattern. Your Squid would work
even faster by moving the "http_access deny ads-block" down below the
"deny manager" config line.
>
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
>
> # Deny requests to certain unsafe ports
> http_access deny !Safe_ports
>
> # Deny CONNECT to other than secure SSL ports
> http_access deny CONNECT !SSL_ports
>
> # Only allow cachemgr access from localhost
> http_access allow localhost manager
> http_access deny manager
>
> ##Redirect some sites to storeid
> ################################################################################################
> ################################################################################################
> ################################################################################################
> # Windows update acls
> acl windowsupdate dstdomain windowsupdate.microsoft.com
> acl windowsupdate dstdomain .update.microsoft.com
> acl windowsupdate dstdomain download.windowsupdate.com
> acl windowsupdate dstdomain redir.metaservices.microsoft.com
> acl windowsupdate dstdomain images.metaservices.microsoft.com
> acl windowsupdate dstdomain c.microsoft.com
> acl windowsupdate dstdomain www.download.windowsupdate.com
> acl windowsupdate dstdomain wustat.windows.com
> acl windowsupdate dstdomain crl.microsoft.com
> acl windowsupdate dstdomain sls.microsoft.com
> acl windowsupdate dstdomain productactivation.one.microsoft.com
> acl windowsupdate dstdomain ntservicepack.microsoft.com
>
> # Windows update methods
> acl wuCONNECT dstdomain www.update.microsoft.com
> acl wuCONNECT dstdomain sls.microsoft.com
>
> # Windows updates rules
> http_access allow CONNECT wuCONNECT localnet
> http_access allow CONNECT wuCONNECT localhost
> http_access allow windowsupdate localnet
> http_access allow windowsupdate localhost
>
The below is one massively huge regex pattern for Squid to try and
compile then process. Lets see what can be removed...
Firstly, remove all the ".*" which are on the end of patterns.
> acl store_rewrite_list url_regex -i fbcdn\/.*(jpg|gif|png|swf)
> acl store_rewrite_list url_regex -i (akamaihd|fbcdn|facebook)\.(net|com)\/.*
> acl store_rewrite_list url_regex -i attachment\.fbsbx\.com
> acl store_rewrite_list url_regex -i fbcdn-dragon-a\.akamaihd\.net
> acl store_rewrite_list url_regex -i socialpointgames\.com
> acl store_rewrite_list url_regex -i miniclipcdn\.com
> acl store_rewrite_list url_regex -i
> syntasia\.hs\.llnwd\.net\/[a-z][0-9]+\/baseballheroes\/.*
> acl store_rewrite_list url_regex -i \.google\-analytics\.com
> acl store_rewrite_list url_regex -i google\-analytics\.com
The two above patterns overlap. The seconds one will match everything
the first one does, and more.
... Remove the "-i \.google\-analytics\.com" line.
> acl store_rewrite_list url_regex -i video\.google\.com\/ThumbnailServer
> acl store_rewrite_list url_regex -i
> (youtube|google).*(videoplayback|liveplay)
> acl store_rewrite_list url_regex -i
> youtube.*(ptracking|stream_204|player_204|gen_204).*
> acl store_rewrite_list url_regex -i
> (youtube|google|googlevideo).*videoplayback.*
The above line overlaps completely with an earlier pttern.
Remove the "(youtube|google|googlevideo).*videoplayback.*" line.
> acl store_rewrite_list url_regex -i c\.android\.clients\.google\.com
> acl store_rewrite_list url_regex -i phobos\.apple\.com
> acl store_rewrite_list url_regex -i \.apple\.com
These apple.com patterns overlap completely.
Remove the "-i phobos\.apple\.com" line.
> acl store_rewrite_list url_regex -i \/speedtest\/.*(jpg|txt|png|swf)
> acl store_rewrite_list url_regex -i speedtest.*\/.*(jpg|txt|png|swf)
These speedtest patterns also overlap completely.
Remove the "-i \/speedtest\/.*(jpg|txt|png|swf)" line.
> acl store_rewrite_list url_regex -i \.youjizz\.com\/.*(3gp|mpg|flv|mp4)
> acl store_rewrite_list url_regex -i \.phncdn\.com\/.*(mp4|flv|3gp|mpg|wmv)
> acl store_rewrite_list url_regex -i \.cdn13\.com\/.*(flv|mp3|mp4|3gp|wmv)
> acl store_rewrite_list url_regex -i \.filehippo\.com\/.*
> acl store_rewrite_list url_regex -i filehippo\.com\/.*
The above two lines are overlap.
Remove the "-i \.filehippo\.com\/.*" line.
Remove the ".*" from the end of the "-i filehippo\.com\/.*" line.
> acl store_rewrite_list url_regex -i dl\.sourceforge\.net\/project\/.*
> acl store_rewrite_list url_regex -i googlevideo\.com
> acl store_rewrite_list url_regex -i reverbnation\.com
> acl store_rewrite_list url_regex -i
> c2lo\.reverbnation\.com\/audio_player\/ec_stream_song\/.*
> acl store_rewrite_list url_regex -i (4shared|4shared\-china)\.com
> acl store_rewrite_list url_regex -i 4shared\.com
> acl store_rewrite_list url_regex -i bp\.blogspot\.com\/.*
> acl store_rewrite_list url_regex -i ytimg\.com
> acl store_rewrite_list url_regex -i (ggpht|googleusercontent)\.com
> acl store_rewrite_list url_regex -i (s|i[0-9]*)\.ytimg\.com\/.*
The above line is a more complicated pattern matching only things that
are matched by a simpler pattern elsewhere.
Remove the "-i (s|i[0-9]*)\.ytimg\.com\/.*" line.
> acl store_rewrite_list url_regex -i video\.google\.com\/ThumbnailServer
The above line is an exact duplicate of an earlier entry. Remove it.
> acl store_rewrite_list url_regex -i (google\.co(m|\.uk|\.id))\/.*
> acl store_rewrite_list url_regex -i (\.gstatic\.com.*|\.wikimapia\.org)
> acl store_rewrite_list url_regex -i gstatic.com\/images.*
> acl store_rewrite_list url_regex -i gstatic.com\/.*
You have three patterns there doing the same match.
Remove the "-i gstatic.com\/.*" and "-i gstatic.com\/images.*" lines
Replace the "-i (\.gstatic\.com.*|\.wikimapia\.org)" line with:
-i (gstatic\.com|\.wikimapia\.org)
> acl store_rewrite_list url_regex -i bing\.(com|net)\/.*
> acl store_rewrite_list url_regex -i (dmcdn\.net|dailymotion\.com).*
> acl store_rewrite_list url_regex -i avast\.com
> acl store_rewrite_list url_regex -i geo\.kaspersky\.com
> acl store_rewrite_list url_regex -i update\.avg\.com
> acl store_rewrite_list url_regex -i
> (cbk|mt|khm|mlt|tbn|mw)[0-9]?.google\.co(m|\.uk|\.id)
> acl store_rewrite_list url_regex -i
> (\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)
> acl store_rewrite_list url_regex -i sdlc\-esd\.sun\.com
> acl store_rewrite_list url_regex -i cloudfront\.net
> acl store_rewrite_list url_regex -i sendspace\.com
> acl store_rewrite_list url_regex -i rapidshare\.com
> acl store_rewrite_list url_regex -i 185\.27\.237\.[\d]*
> acl store_rewrite_list url_regex -i syntasia\.hs\.llnwd\.net
> acl store_rewrite_list url_regex -i playspace\.r\.worldssl\.net
> acl store_rewrite_list url_regex -i playit\.pk
> acl store_rewrite_list url_regex -i attachment\.fbsbx\.com
> acl store_rewrite_list url_regex -i firedrive\.com
> acl store_rewrite_list url_regex -i cache\.pack\.google\.com
> acl store_rewrite_list url_regex -i pack\.google\.com
You have other patterns that match on just "google.com".
Remove the "-i cache\.pack\.google\.com" and "-i pack\.google\.com" lines.
> acl store_rewrite_list url_regex -i dropboxusercontent\.com
> acl store_rewrite_list url_regex -i aclst\.com
> acl store_rewrite_list url_regex -i blackberry\.com
> acl store_rewrite_list url_regex -i (bitgravity|opera)\.com
> acl store_rewrite_list url_regex -i ggpht\.co(m|\.(id|uk))
> acl store_rewrite_list url_regex -i instagram\.com
> acl store_rewrite_list url_regex -i virtualearth\.net
> acl store_rewrite_list url_regex -i cnet\.com
> acl store_rewrite_list url_regex -i xvideos\.com
> acl store_rewrite_list url_regex -i .*xhcdn.*
> acl store_rewrite_list url_regex -i steampowered\.com
> acl store_rewrite_list url_regex -i starhub\.com
> acl store_rewrite_list url_regex -i (wargaming|hwcdn)\.net
> acl store_rewrite_list url_regex -i indowebster\.com
> acl store_rewrite_list url_regex -i filetrip\.net
> acl store_rewrite_list url_regex -i get4mobile\.net
> acl store_rewrite_list url_regex -i tube8\.com
> acl store_rewrite_list url_regex -i (redtube|redtubefiles)\.com
Replace the above pattern with:
-i redtube(files)?\.com
> acl store_rewrite_list url_regex -i .*nsimg.*
> acl store_rewrite_list url_regex -i .*mystreamservice.*
> acl store_rewrite_list url_regex -i youjizz\.com
> acl store_rewrite_list url_regex -i .*phncdn.*
> acl store_rewrite_list url_regex -i .*keezmovies.*
> acl store_rewrite_list url_regex -i .*youporn.*
> acl store_rewrite_list url_regex -i .*rncdn.*
> acl store_rewrite_list url_regex -i .*spankwire.*
> acl store_rewrite_list url_regex -i .*pornhub.*
> acl store_rewrite_list url_regex -i .*playvid.*
> acl store_rewrite_list url_regex -i .*maxporn.*
> acl store_rewrite_list url_regex -i .*fucktube.*
> acl store_rewrite_list url_regex -i .*slutload-media.*
> acl store_rewrite_list url_regex -i .*hardsextube.*
All of these lines ...
> acl store_rewrite_list url_regex -i public\.extremetube\.phncdn\.com
> acl store_rewrite_list url_regex -i video\.pornhub\.phncdn\.com
> acl store_rewrite_list url_regex -i public\.keezmovies\.phncdn\.com
> acl store_rewrite_list url_regex -i public\.youporn\.phncdn\.com
> acl store_rewrite_list url_regex -i public\.spankwire\.phncdn\.com
> acl store_rewrite_list url_regex -i public\.keezmovies\.com
> acl store_rewrite_list url_regex -i public\.spankwire\.com
> acl store_rewrite_list url_regex -i pornhub\.com
> acl store_rewrite_list url_regex -i slutload-media\.com
> acl store_rewrite_list url_regex -i hardsextube\.com
.. to here are duplicates. Remove them all.
>
> acl store_rewrite_list_domain url_regex
> ^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
So, this matches almost every URL that exists with a domain containing
at least two '.' characters.
Go through what remains of the store_rewrite_list ACL lines and remove
the patterns that are searching for domians with two dots in them.
For example "attachment\.fbsbx\.com".
Also simplify the patterns that are search for domains with two dots OR
some alternative. For Example; "-i ggpht\.co(m|\.(id|uk))" can be
replaced with "-i ggpht\.com" since the .co.id and .co.uk versions will
be matched here.
> acl store_rewrite_list_domain url_regex
> (([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
For the non- http:// URLs the above pattern did not match, this repeats
the search in a little more detail and matches all those 2-dot domains
and along with a number of others in the http:// URL space.
> acl store_rewrite_list_path urlpath_regex
> \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|avc|zip|mp3|3gp|rar|on2|mar|exe)$
> acl store_rewrite_list_domain_CDN url_regex (khm|mt)[0-9]?.google.com
Your "store_rewrite_list" list contains a pattern that matches the
sub-string "google.com" without any qualifications.
You can replace the "(khm|mt)[0-9]?.google.com" with "google\.com"
Then remove all the store_rewrite_list pattens for google.com
> streamate.doublepimp.com.*\.js\? photos-[a-z].ak.fbcdn.net
> \.rapidshare\.com.*\/[0-9]*\/.*\/[^\/]*
> ^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*) \.doubleclick\.net.*
> yieldmanager cpxinteractive
> ^http:\/\/[.a-z0-9]*\.photobucket\.com.*\.[a-z]{3}$ quantserve\.com
>
> store_id_access allow store_rewrite_list
> store_id_access allow store_rewrite_list_domain
> store_id_access allow store_rewrite_list_path
> store_id_access allow store_rewrite_list_domain_CDN
>
Since these are all doing "allow" action the order does not matter.
Your Squid will work much faster if you switch the order of these lines
so the smaller / simpler patterns are tested first:
store_id_access allow store_rewrite_list_domain
store_id_access allow store_rewrite_list_path
store_id_access allow store_rewrite_list_domain_CDN
store_id_access allow store_rewrite_list
>
> ####for looping 302 on youtube
> acl text-html rep_mime_type text/html
> acl http302 http_status 302
>
> store_miss deny text-html
Seriously? After all the patterns forcing caching and refresh_patterns
trying to force things to be stored longer than their designers coded
for ... you never want any HTML content to be cached?
> store_miss deny http302
> send_hit deny text-html
> send_hit deny http302
>
> acl norewrite url_regex -i redirector\.c\.android\.clients\.google\.com
> store_id_access deny norewrite
You have a pattern in the store_rewrite_list* ACLs that already allows
anything matching "google.com" to be re-written. By the time this config
line takes effect nothing can ever match it.
>
> ##this for send to storeid
> acl youtube_to_storeid url_regex -i
> ^https?:\/\/.*(youtube|google).*(set_awesome|stream_204|playback|ptracking|watchtime|atr|player_204|videogoodput|get_video|get_video_info|s\?|delayplay|ads|qoe|gen_204).*(video_id|docid|\&v|content_v)\=([^\&\s]*).*
> acl youtube_to_storeid url_regex -i
> ^https?:\/\/.*(youtube|google).*videoplayback.*
>
> acl gvt1_to_storeid url_regex -i ^https?:\/\/.*\.gvt1\.com\/market\/.*
> acl mgccw_to_storeid url_regex -i ^https?:\/\/.*\.mgccw\.com\/.*
>
Both of the above ACLs are looking for domains with 2 dots in them.
Your store_rewrite_list_domain ACL has already matched and passed the
request to the Store-ID helper. Neither of these ACLs will ever be used.
> store_id_access allow youtube_to_storeid
> store_id_access allow gvt1_to_storeid
> store_id_access allow mgccw_to_storeid
>
> ## this for 206
> acl partial dstdomain .googlevideo.com
> acl partial dstdomain .youtube.com
> acl partial dstdomain .mgccw.com
> range_offset_limit none partial
> store_id_access allow partial
>
>
> acl queryreg url_regex -i gemscool\.com\/registration\/.*
> acl queryreg url_regex -i gemscool\.com\/isiGcash\/.*
> acl queryreg url_regex -i ^http.*live\.mytrans\.com.*
> acl queryreg url_regex -i ^http.*socialpointgames\.com\/dragoncity.*USERID.*
> acl queryreg url_regex -i ^http.*fb_source=bookmark_apps.*
> acl queryreg url_regex -i ^http.*gvoucher.*
> acl queryreg url_regex -i ^http.*\.(asp|aspx|php|xml)(\?.*|)$
>
> cache deny queryreg
>
> acl playstoreandroid url_regex -i
> c.android.clients.google.com.market.GetBinary.GetBinary.*
> store_id_access allow playstoreandroid
Here is another pattern that has no effect behind the 2-dot domain name
ones. In particular the 2-dot pattern matching URLs without requiring
http:// initial prefix.
Thats all I've got time for today. The config has plenty of other little
gotchas though.
Amos
More information about the squid-users
mailing list