[squid-users] block user agent
Vieri
rentorbuy at yahoo.com
Fri Nov 17 15:27:51 UTC 2017
________________________________
From: Alex Rousskov <rousskov at measurement-factory.com>
> 1. Your "works" and "does not work" setups currently differ in at least
> three variables: user agent name, slash after the user agent name, and
> acl negation in http_access. Find out which single variable is
> responsible for the breakage by eliminating all other differences.
>
> 2. Post two ALL,2 cache.logs, each containing a single transaction, one
> for the "works" case and one for the "does not work" case polished as
> discussed in #1.
I can't really do anything about #1 except maybe leave out the forward slash.
That's because my 2 examples are trying to achieve the opposite.
Let me just rephrase everything so it's crystal clear.
My goal is to deny all client traffic from browsers that DO NOT have a specific user-agent string. So this is a negated statement. One of the things I can't do in Squid is define an ACL with a negated lookahead such as (?!useragentname).
So I set up two examples.
Common to both:
acl allowed_useragent browser MyAllowedUAstring
acl denied_useragent browser MyDeniedUAstring
# example 1:
http_access deny denied_useragent
http_reply_access deny denied_useragent
deny_info http://proxy-server1/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_useragent denied_useragent
I then run this from my test client:
# curl --insecure --user-agent MyAllowedUAstring https://www.gentoo.org
-> works as expected (I see the web site). I guess you don't need to see cache.log here.
Now I run this:
# curl --insecure --user-agent MyDeniedUAstring https://www.gentoo.org
-> works as expected (I'm denied access and I see Squid's error page).
I guess there's no need for the full log here either. It boils down to this anyway:
2017/11/17 13:24:26.937 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyDeniedUAstring)' found in 'MyDeniedUAstring'
2017/11/17 13:24:26.937 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET https://www.gentoo.org/ is DENIED; last ACL checked: denied_useragent
I'm done with example 1. That's because I cannot make a consistent list of all user agents I want to actively block. Instead, I want to "deny everyone except one or two".
Also, since negative lookaheads are not supported in regular expressions, I change my example 1 to:
# example 2:
http_access deny !allowed_useragent
http_reply_access deny !allowed_useragent
deny_info http://proxy-server1/proxy-error/?a=%a&B=%B&e=%e&E=%E&H=%H&i=%i&M=%M&o=%o&R=%R&T=%T&U=%U&u=%u&w=%w&x=%x&acl=denied_useragent allowed_useragent
Then I run this from the client:
# curl --insecure --user-agent MyAllowedUAstring https://www.gentoo.org
-> I was expecting to be allowed access since Squid denies "everything that's not" MyAllowedUAstring. Well, at least I should have passed the "deny" line in example 2.
However, I'm being blocked right there. This is the full log:
2017/11/17 13:30:42.216 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 88
2017/11/17 13:30:42.216 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3229 remote=[::] FD 88 flags=25
2017/11/17 13:30:42.216 kid1| 33,2| client_side.cc(3943) httpsSslBumpAccessCheckDone: sslBump needed for local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 method 4
2017/11/17 13:30:42.216 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17
2017/11/17 13:30:42.216 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT 89.16.167.134:443 HTTP/1.1
Host: 89.16.167.134:443
----------
2017/11/17 13:30:42.216 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request CONNECT 89.16.167.134:443 is DENIED; last ACL checked: allowed_useragent
2017/11/17 13:30:42.216 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:42.216 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:42.216 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:42.226 kid1| 83,2| client_side.cc(3843) clientNegotiateSSL: clientNegotiateSSL: New session 0x125e030 on FD 8 (10.215.144.48:65262)
2017/11/17 13:30:42.226 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17
2017/11/17 13:30:42.226 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:
---------
GET / HTTP/1.1
Host: www.gentoo.org
User-Agent: MyAllowedUAstring
Accept: */*
----------
2017/11/17 13:30:42.227 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyAllowedUAstring)' found in 'MyAllowedUAstring'
2017/11/17 13:30:42.227 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET https://www.gentoo.org/ is ALLOWED, because it matched denied_mimetypes_rep
2017/11/17 13:30:42.227 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17
2017/11/17 13:30:42.227 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 307 Temporary Redirect
Server: squid
Mime-Version: 1.0
Date: Fri, 17 Nov 2017 12:30:42 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Location: http://proxy-server1/proxy-error/?a=-&B=&e=0&E=%5BNo%20Error%5D&H=89.16.167.134&i=10.215.144.48&M=CONNECT&o=&R=/&T=Fri,%2017%20Nov%202017%2012%3A30%3A42%20GMT&U=https%3A%2F%2F89.16.167.134%2F*&u=89.16.167.134%3A443&w=IT%40mydomain.org&x=&acl=denied_useragent
X-Squid-Error: 403 Access Denied
X-Cache: MISS from proxy-server1
X-Cache-Lookup: NONE from proxy-server1:3227
Connection: close
Note that I have these defaults in my squid conf file:
acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports
Let's try another one:
# curl --insecure --user-agent MyDeniedUAstring https://www.gentoo.org
-> This is as expected, I guess.
Full log:
2017/11/17 13:30:10.365 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 88
2017/11/17 13:30:10.365 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3229 remote=[::] FD 88 flags=25
2017/11/17 13:30:10.365 kid1| 33,2| client_side.cc(3943) httpsSslBumpAccessCheckDone: sslBump needed for local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17 method 4
2017/11/17 13:30:10.365 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17
2017/11/17 13:30:10.365 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT 89.16.167.134:443 HTTP/1.1
Host: 89.16.167.134:443
----------
2017/11/17 13:30:10.365 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request CONNECT 89.16.167.134:443 is DENIED; last ACL checked: allowed_useragent
2017/11/17 13:30:10.365 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:10.365 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:10.365 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:10.385 kid1| 83,2| client_side.cc(3843) clientNegotiateSSL: clientNegotiateSSL: New session 0xdbdc70 on FD 8 (10.215.144.48:65237)
2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17
2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:
---------
GET / HTTP/1.1
Host: www.gentoo.org
User-Agent: MyDeniedUAstring
Accept: */*
----------
2017/11/17 13:30:10.386 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET https://www.gentoo.org/ is DENIED, because it matched allowed_useragent
2017/11/17 13:30:10.386 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:10.386 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:10.386 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 13:30:10.386 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET https://www.gentoo.org/ is ALLOWED, because it matched allowed_useragent
2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=89.16.167.134:443 remote=10.215.144.48 FD 8 flags=17
2017/11/17 13:30:10.386 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 302 Found
Server: squid
Mime-Version: 1.0
Date: Fri, 17 Nov 2017 12:30:10 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Location: http://proxy-server1/proxy-error/?a=-&B=&e=0&E=%5BNo%20Error%5D&H=www.gentoo.org&i=10.215.144.48&M=GET&o=&R=/&T=Fri,%2017%20Nov%202017%2012%3A30%3A10%20GMT&U=https%3A%2F%2Fwww.gentoo.org%2F&u=https%3A%2F%2Fwww.gentoo.org%2F&w=IT%40mydomain.org&x=&acl=denied_useragent
X-Squid-Error: 403 Access Denied
X-Cache: MISS from proxy-server1
X-Cache-Lookup: NONE from proxy-server1:3227
Connection: close
Now for plain HTTP with example 2.
# curl --user-agent MyDeniedUAstring http://www.fltk.org/index.php
-> As expected. It blocks access.
Full log:
2017/11/17 15:56:52.648 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 85
2017/11/17 15:56:52.648 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3228 remote=[::] FD 85 flags=25
2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17
2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:
---------
GET /index.php HTTP/1.1
Host: www.fltk.org
User-Agent: MyDeniedUAstring
Accept: */*
----------
2017/11/17 15:56:52.648 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.fltk.org/index.php is DENIED; last ACL checked: allowed_useragent
2017/11/17 15:56:52.648 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 15:56:52.648 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 15:56:52.648 kid1| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable
2017/11/17 15:56:52.648 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET http://www.fltk.org/index.php is ALLOWED, because it matched allowed_useragent
2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17
2017/11/17 15:56:52.648 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 302 Found
Server: squid
Mime-Version: 1.0
Date: Fri, 17 Nov 2017 14:56:52 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 0
Location: http://proxy-server1/proxy-error/?a=-&B=&e=0&E=%5BNo%20Error%5D&H=www.fltk.org&i=10.215.144.48&M=GET&o=&R=/index.php&T=Fri,%2017%20Nov%202017%2014%3A56%3A52%20GMT&U=http%3A%2F%2Fwww.fltk.org%2Findex.php&u=http%3A%2F%2Fwww.fltk.org%2Findex.php&w=IT%40mydomain.org&x=&acl=denied_useragent
X-Squid-Error: 403 Access Denied
X-Cache: MISS from proxy-server1
X-Cache-Lookup: NONE from proxy-server1:3227
Connection: keep-alive
However, now comes the interesting part.
# curl --user-agent MyAllowedUAstring http://www.fltk.org/index.php
-> works as expected (I see the web site).
Full log:
2017/11/17 15:55:23.550 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 85
2017/11/17 15:55:23.550 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3228 remote=[::] FD 85 flags=25
2017/11/17 15:55:23.551 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17
2017/11/17 15:55:23.551 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:
---------
GET /index.php HTTP/1.1
Host: www.fltk.org
User-Agent: MyAllowedUAstring
Accept: */*
----------
2017/11/17 15:55:23.551 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyAllowedUAstring)' found in 'MyAllowedUAstring'
2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(805) aclMatchExternal: bllookup("http www.fltk.org 80 /index.php") = lookup needed
2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(808) aclMatchExternal: "http www.fltk.org 80 /index.php": queueing a call.
2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(1444) Start: fg lookup in 'bllookup' for 'http www.fltk.org 80 /index.php'
2017/11/17 15:55:23.551 kid1| 82,2| external_acl.cc(811) aclMatchExternal: "http www.fltk.org 80 /index.php": return -1.
2017/11/17 15:55:23.553 kid1| 82,2| external_acl.cc(1372) externalAclHandleReply: reply={result=OK, notes={message: www.fltk.org site not found in blacklist; }}
2017/11/17 15:55:23.553 kid1| 82,2| external_acl.cc(1288) external_acl_cache_add: external_acl_cache_add: Adding 'http www.fltk.org 80 /index.php' = ALLOWED
2017/11/17 15:55:23.553 kid1| 82,2| external_acl.cc(841) aclMatchExternal: bllookup = ALLOWED
2017/11/17 15:55:23.553 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.fltk.org/index.php is ALLOWED; last ACL checked: bl_lookup
2017/11/17 15:55:23.553 kid1| 85,2| client_side_request.cc(721) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2017/11/17 15:55:23.553 kid1| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET http://www.fltk.org/index.php is ALLOWED; last ACL checked: bl_lookup
2017/11/17 15:55:23.554 kid1| 88,2| client_side_reply.cc(593) cacheHit: clientProcessHit: Vary detected!
2017/11/17 15:55:23.554 kid1| 17,2| FwdState.cc(133) FwdState: Forwarding client request local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17, url=http://www.fltk.org/index.php
2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://www.fltk.org/index.php'
2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED
2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED
2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=10.215.144.48 remote=66.39.46.122:80 flags=25
2017/11/17 15:55:23.554 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths: timedout = 0
2017/11/17 15:55:23.708 kid1| 11,2| http.cc(2229) sendRequest: HTTP Server local=10.215.144.48:35373 remote=66.39.46.122:80 FD 13 flags=25
2017/11/17 15:55:23.708 kid1| 11,2| http.cc(2230) sendRequest: HTTP Server REQUEST:
---------
GET /index.php HTTP/1.1
User-Agent: MyAllowedUAstring
Accept: */*
Host: www.fltk.org
Cache-Control: max-age=259200
Connection: keep-alive
----------
2017/11/17 15:55:23.884 kid1| ctx: enter level 0: 'http://www.fltk.org/index.php'
2017/11/17 15:55:23.884 kid1| 11,2| http.cc(719) processReplyHeader: HTTP Server local=10.215.144.48:35373 remote=66.39.46.122:80 FD 13 flags=25
2017/11/17 15:55:23.884 kid1| 11,2| http.cc(720) processReplyHeader: HTTP Server REPLY:
---------
HTTP/1.1 200 OK
Date: Fri, 17 Nov 2017 14:55:23 GMT
Server: Apache/2.4.29
Cache-Control: no-cache
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
3a02
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Fast Light Toolkit - Fast Light Toolkit (FLTK)</title>
<meta http-equiv='Pragma' content='no-cache'>
<meta http-equiv='Content-Type' content='text/html; charset=utf-8'>
<link rel='stylesheet' type='text/css' href='fltk.css'>
<link rel='alternate' title='FLTK RSS' type='application/rss+xml' href='index.rss'>
<link rel='shortcut icon' href='favicon.ico' type='image/x-icon'>
<meta name='keywords' content='gui toolkit,c++,linux,unix,macos x,x11,windows'>
</head>
<body>
<table width='100%' border='0' cellspacing='0' cellpadding='0' summary='Page'>
<tr class='header'><td valign='top' width='15' rowspan='2'><a href='index.php'><img src='images/top-left.gif' width='15' height='70' border='0' alt=''></a></td><td valign='top' width='224' rowspan='2'><a href='index.php'><img src='images/top-middle.gif' width='224' height='70' border='0' alt=''></a></td><td width='100%' height='40'><h1>Fast Light Toolkit</h1> </td><td align='right' nowrap>
<table cellpadding=0 cellspacing=0 border=0><tr><td valign=top nowrap>
<a href=fltk-rss.xml><img src=images/rss-fee
----------
2017/11/17 15:55:23.885 kid1| ctx: exit level 0
2017/11/17 15:55:23.885 kid1| 23,2| url.cc(407) urlParse: urlParse: URI has whitespace: {icap://127.0.0.1:1344/clamav ICAP/1.0
}
2017/11/17 15:55:24.038 kid1| 28,2| RegexData.cc(73) match: aclRegexData::match: match '(MyAllowedUAstring)' found in 'MyAllowedUAstring'
2017/11/17 15:55:24.038 kid1| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET http://www.fltk.org/index.php is ALLOWED, because it matched denied_mimetypes_rep
2017/11/17 15:55:24.038 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=66.39.46.122:80 remote=10.215.144.48 FD 8 flags=17
2017/11/17 15:55:24.038 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 200 OK
Date: Fri, 17 Nov 2017 14:55:23 GMT
Server: Apache/2.4.29
Cache-Control: no-cache
Vary: Accept-Encoding
Content-Type: text/html
Via: ICAP/1.0 proxy-server1.hospitalmanacor.org (C-ICAP/0.5.2 SquidClamav/Antivirus service )
X-Cache: MISS from proxy-server1
X-Cache-Lookup: MISS from proxy-server1:3227
Transfer-Encoding: chunked
Connection: keep-alive
How can I modify my example 2 settings so this access control works the same way with both http and https in an ssl-bumped environment.
Thanks,
Vieri
More information about the squid-users
mailing list