[squid-users] make URL bypass squid proxy
ngtech1ltd at gmail.com
ngtech1ltd at gmail.com
Thu Jun 29 10:40:20 UTC 2023
Hey Rob,
it’s a great question.
I will assume you are using squid 5.x since it’s the stable one.
There is a configuration reference documentation at:
http://www.squid-cache.org/Versions/v5/cfgman/
And the relevant one is ‘acl’:
http://www.squid-cache.org/Versions/v5/cfgman/acl.html
In the config we the next options:
* dstdom_regex
* dstdomain
* ssl::server_name
* ssl::server_name_regex
Each one of them is different in two things.
The dstdomain is happening when the http request verification is done, usually in the http_access part.
So for a CONNECT request in your case it should work but will only work for http_access rules ie allow or deny the connection to being proxied by the proxy.
All this happens before the bump stage.
The dstdom_regex is doing the same but with another "lookup" mechanism.
Just to illustrate we will use the domain www.example.com.
With dstdomain .example.com which is a wildcard domain the www.example .com will match since what it does it..
Take apart the .example.com into .com and .example and example (logically, not the code actually does..)
Then when a request for www.example-2.com arrives the dstdomain acl checks the strings one by one ie:
com
example-2
www
So the "com" would match and since it has a dot "." as a prefix it continues to the next part of the domain name ie:
example-2
and it will try to match it to "example" from the ".example.com'.
then it will fail and there for it will declare the specific definition in the acl as "do not match.
But for www.example .com it would be different:
Squid will find that the .com is in the "com" part of www.example.com and will continue to the next part ie
"example" and it will match.
Then it will see the dot "." which means that all subdomains are a part of this rule ie both example.com and any other domain
under .example.com will match so... www.example.com and www2.example.com and w.example.com will match and this is compared to
example.com
with dstdomain example.com it will only match a full match of example.com but not www.example.com and it's pointless to define both:
- example.com
- .example.com
In the same acl file.
Squid will compare it to the basic proxy url request and the Host: header inside of it.
(Amos might remember which one of these wins or which one is the main one)
So that's how a dstdomain test works.
The dstdom_regex is applied on the same "part" of the proxy request ie request url and Host header but...
Tries to match it with a regex that is a much more "CPU" intensive but can do magic if you know how to use it right.
The pros are that you can use one regex to match about 100 domains however dstdomain is much faster in many cases.
To test regex in general I love:
https://rubular.com/
and it's very simple to put a line with the url or the domain itself and write the pattern and then get instant result.
There are many sites which does this exact same thing.
You need: activate.redshift3d.com for the test.
For the server_name dstdom and regex the difference is on what part of the "known" information on the request done.
Ie it's not on the known plain text proxy request but a "peek" into the client or server TLS part of the connection bumping.
It's possible on TLS 1.2 with ease but 1.3 with encryption it's a whole other story.
(Alex and his team is more involved in this part then me)
It's a bit more complex in the server_name part since from what I remember the server_name can be the SNI itself or parts of the
certificate of the server.
In the certificate of the server in many cases there is no real domain but a catch all ie multidomain which is for example:
*.example.com
The ssl::server_name is like dstdomain so .example.com should match both SNI with www.example.com and the *.example.com
If it doesn't then it’s a bug..
About the ssl::server_name_regex you will need to test and match the regex against couple possibilities that are in the certificate ie:
*.example.com
activation.example.com
etc...
The current certificate I see is:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
c0:71:02:fc:1f:e5:64:5b:0d:d6:ae:c8:1a:17:e6:80
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
Validity
Not Before: Dec 14 00:00:00 2022 GMT
Not After : Jan 12 23:59:59 2024 GMT
Subject: CN = activate.redshift3d.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:cd:f9:bd:58:a4:7a:16:3f:23:12:9d:e0:1e:39:
a0:ef:66:ae:b3:ae:32:5c:69:9e:cc:89:83:fb:a2:
e0:c7:9d:86:aa:e6:9b:b3:d9:0c:bb:35:36:2f:cf:
be:ec:1e:62:ca:d8:ab:16:66:6e:00:8d:f7:42:3d:
5b:d6:bf:a0:30:a1:c2:6f:5e:69:76:ef:0c:38:73:
d2:e8:42:b8:83:04:a3:2c:da:22:a4:c4:13:71:38:
36:00:1b:e7:b3:f0:fe:6e:59:17:11:e7:0a:81:3e:
04:b2:ad:e0:61:c1:15:f3:9a:36:33:24:a6:f1:0d:
b6:e6:32:91:34:54:7c:da:40:14:72:06:25:91:78:
75:07:84:62:47:7b:3f:05:60:e6:35:d3:27:55:1e:
ff:23:82:18:09:f5:8a:0f:a6:cb:02:bc:d6:09:98:
54:5b:10:e3:d2:09:4e:31:2a:75:42:29:00:86:9b:
f9:78:2f:fd:99:9e:2e:10:3e:bc:84:f5:9b:3a:3f:
fa:d9:b6:20:c0:2e:86:f3:de:14:be:3a:1a:05:30:
3b:00:e6:50:8d:4c:13:a3:97:dc:f4:03:9e:84:31:
0d:23:f2:02:50:6d:53:a5:30:03:b0:fd:f2:46:64:
49:22:a4:d1:5a:00:fb:78:37:fc:51:b1:f0:6e:41:
3b:5d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:8D:8C:5E:C4:54:AD:8A:E1:77:E9:9B:F9:9B:05:E1:B8:01:8D:61:E1
X509v3 Subject Key Identifier:
5D:BC:D9:F2:C4:99:83:5E:C0:3A:F1:BF:FE:F2:E3:92:92:F1:F0:29
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Certificate Policies:
Policy: 1.3.6.1.4.1.6449.1.2.2.7
CPS: https://sectigo.com/CPS
Policy: 2.23.140.1.2.1
Authority Information Access:
CA Issuers - URI:http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt
OCSP - URI:http://ocsp.sectigo.com
X509v3 Subject Alternative Name:
DNS:activate.redshift3d.com, DNS:www.activate.redshift3d.com
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 76:FF:88:3F:0A:B6:FB:95:51:C2:61:CC:F5:87:BA:34:
B4:A4:CD:BB:29:DC:68:42:0A:9F:E6:67:4C:5A:3A:74
Timestamp : Dec 14 15:34:39.227 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:44:02:20:6E:D0:96:68:F3:07:A0:5F:DD:C5:B2:61:
2F:51:6B:06:4A:3C:7F:9B:DA:08:6D:5A:31:0D:B8:0B:
83:7D:D6:37:02:20:64:66:3F:A2:40:87:78:9E:C8:90:
7C:EE:7C:77:3C:BB:43:C1:9F:52:54:F1:04:85:B6:AC:
6B:F2:36:8E:E3:03
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
Timestamp : Dec 14 15:34:39.153 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:44:02:20:69:BA:F2:DD:32:B3:48:0C:26:E3:FB:F0:
DD:4E:5E:10:95:1E:B4:59:5B:67:64:C4:F2:40:7C:A8:
62:35:71:FE:02:20:13:15:41:26:E5:9C:DB:34:DA:D7:
B9:CB:B3:1A:4E:33:C7:46:7F:D9:93:45:25:7F:DE:A3:
72:EB:07:03:DB:C5
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : EE:CD:D0:64:D5:DB:1A:CE:C5:5C:B7:9D:B4:CD:13:A2:
32:87:46:7C:BC:EC:DE:C3:51:48:59:46:71:1F:B5:9B
Timestamp : Dec 14 15:34:39.124 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:2C:E2:85:9C:A6:54:1B:1C:31:E5:F8:37:
E9:CD:09:8B:D8:26:29:E4:C7:65:94:9C:FF:32:D2:41:
CD:16:A3:51:02:21:00:A0:2F:C3:F7:A6:55:3B:21:EB:
9B:CA:6E:4E:07:A2:8C:40:4B:E2:27:D6:82:44:0F:09:
C9:F7:7D:1B:72:6F:13
Signature Algorithm: sha256WithRSAEncryption
25:bd:bb:de:57:c0:7f:07:5e:18:62:2e:0b:d3:03:54:a7:45:
ab:c6:1f:e2:f6:58:ff:6e:8e:6b:4f:09:9a:87:66:32:81:7f:
f4:35:4f:7e:65:e5:6a:04:d6:62:62:ff:d9:3a:f2:6f:19:ba:
fa:e6:35:0e:2a:44:5c:3b:ee:9d:97:72:05:86:0c:4c:01:c1:
f0:8c:21:c1:c4:84:54:d8:a8:05:25:18:72:db:f7:53:9b:f1:
13:d6:0b:bc:92:6e:01:e3:fd:de:a1:45:e9:29:37:e1:2e:64:
36:b4:4d:38:c1:60:02:6a:17:3d:87:a2:5f:33:3b:86:eb:0d:
cc:dd:fa:d4:43:58:50:43:e7:b7:ec:0a:4f:86:72:15:e5:30:
c9:bb:5f:0b:83:9c:26:6f:60:49:dd:1a:7c:92:45:45:4e:b5:
ce:cd:64:8c:12:83:e9:3d:5c:6b:65:97:75:99:4c:66:eb:d0:
3a:ca:18:62:8a:08:07:16:ab:09:66:bd:65:43:94:00:d9:79:
3e:84:b4:60:7d:7e:f9:09:3c:fe:2d:ad:98:94:17:0c:24:8f:
e1:a2:74:b6:3b:68:c0:01:f9:67:e8:b9:d2:6a:65:9e:99:a3:
4a:5f:39:31:ae:c1:59:02:7b:ef:db:b2:94:06:f8:1a:74:c1:
d7:5b:5b:6a
So the DNS names it will check are:
Subject: CN = activate.redshift3d.com
DNS:activate.redshift3d.com, DNS:www.activate.redshift3d.com
So to summarize the checks of ssl::server_name/ will be done on:
* activate.redshift3d.com
* activate.redshift3d.com
* www.activate.redshift3d.com
So . redshift3d.com ssl::server_name should match the certificate.
If for any reason it doesn't work you can try ssl::server_name_regex with something like:
(^|\.)activate\.redshift3d\.com$
Or just to verify if there is a bug in squid code try:
(^|\.)activate\.redshift3d\.com
Now, the splice should be able to take into account also dstdomain and dstdom_regex but it should match them only if they exist in a plain text
form like in any simple forward proxy CONNECT request.
If for any reason it doesn’t work we should investigate what might cause this issue.
I hope the scroll I wrote make sense to you and with hopes it will clear out the doubts about the wiki article you mentioned.
I believe this is considered a summary of the subject and if Alex and others might think so it can be converted into an example article in the wiki.
Let me know if this makes sense and resolve the issue.
Yours,
Eliezer
From: robert k Wild <robertkwild at gmail.com>
Sent: Thursday, June 29, 2023 12:18
To: ngtech1ltd at gmail.com
Cc: Squid Users <squid-users at lists.squid-cache.org>
Subject: Re: [squid-users] make URL bypass squid proxy
very clever, so you bunch all the acls up
acl NoSSLInterceptAnyOf any-of NoSSLInterceptDstDom NoSSLInterceptDstDomFile NoSSLInterceptRegEx NoSSLInterceptRegExFile
the key word is "any-of" ie if the url hits any one do that first
what about instead of making it
ssl::server_name_regex
i make it
dstdom_regex
On Thu, 29 Jun 2023 at 01:38, <mailto:ngtech1ltd at gmail.com> wrote:
Hey Rob,
The first thing is to allow the domain in the http_acces just to be sure and use a basic deny all bottom line.
Let me try to simplify your squid.conf
In a link:
https://gist.github.com/elico/b49f4a28d4b5db5ba882b10d40872d5e
In plain text:
## START OF FILE
# SSL Interception basic rules
acl DiscoverSNIHost at_step SslBump1
acl NoSSLInterceptRegEx ssl::server_name_regex (^|.*\.)redshift3d\.com$
acl NoSSLInterceptRegExFile ssl::server_name_regex "/usr/local/squid/etc/no-intercept-ssl-regex.txt"
acl NoSSLInterceptDstDom ssl::server_name .redshift3d.com
acl NoSSLInterceptDstDomFile ssl::server_name "/usr/local/squid/etc/no-intercept-ssl-dstdom.txt"
## Any of will test what ever rule match first in a first match/hit fasion
acl NoSSLInterceptAnyOf any-of NoSSLInterceptDstDom NoSSLInterceptDstDomFile NoSSLInterceptRegEx NoSSLInterceptRegExFile
ssl_bump peek DiscoverSNIHost
ssl_bump splice NoSSLInterceptAnyOf
ssl_bump bump all
#SSL Bump port
http_port 3128 ssl-bump cert=/usr/local/squid/etc/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s /var/lib/ssl_db -M 4MB
## http_access acls, will apply on incomming requests and not on responses
acl special_url_regex url_regex https?://(^|.*\.)redshift3d\.com\/
acl special_url_regex_file url_regex "/usr/local/squid/etc/special_url_regex.txt"
acl special_url_dst_dom dstdomain .redshift3d.com
acl special_url_dst_dom_file dstdomain "/usr/local/squid/etc/special_url_dstdom.txt"
acl special_url_any_of any-of special_url_dst_dom special_url_dst_dom_file special_url_regex special_url_regex_file
acl localnet src http://192.168.0.0/16
acl localnet src http://10.0.0.0/8
http_access allow localnet special_url_any_of
http_access deny all
## END OF FILE
Once the above will work try to add other http_access rule like reply access rules
Let me know what happens,
Eliezer
From: robert k Wild <mailto:robertkwild at gmail.com>
Sent: Tuesday, June 27, 2023 09:36
To: mailto:ngtech1ltd at gmail.com
Cc: Squid Users <mailto:squid-users at lists.squid-cache.org>
Subject: Re: [squid-users] make URL bypass squid proxy
Hi Eliezer,
this is a snippet of my whitelist and no intercept SSL config
#SSL Interception
acl DiscoverSNIHost at_step SslBump1
acl NoSSLIntercept ssl::server_name_regex "/usr/local/squid/etc/interceptssl.txt"
ssl_bump peek DiscoverSNIHost
ssl_bump splice NoSSLIntercept
ssl_bump bump all
#
#SSL Bump
http_port 3128 ssl-bump cert=/usr/local/squid/etc/ssl_cert/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s /var/lib/ssl_db -M 4MB
#
#deny up MIME types
acl upmime req_mime_type "/usr/local/squid/etc/mimedeny.txt"
#
#deny URL links
acl url_links url_regex "/usr/local/squid/etc/linksurl.txt"
#
#allow special URL paths
acl special_url url_regex "/usr/local/squid/etc/urlspecial.txt"
#
#deny down MIME types
acl downmime rep_mime_type "/usr/local/squid/etc/mimedeny.txt"
#
http_reply_access allow special_url
http_reply_access deny downmime
#http_access deny upmime
#http_access deny url_links
#
#HTTP_HTTPS whitelist websites
acl whitelist ssl::server_name_regex "/usr/local/squid/etc/urlwhite.txt"
#
http_access allow activation whitelist
http_access deny all
so basically no SSL interception
#SSL Interception
acl DiscoverSNIHost at_step SslBump1
acl NoSSLIntercept ssl::server_name_regex "/usr/local/squid/etc/interceptssl.txt"
ssl_bump peek DiscoverSNIHost
ssl_bump splice NoSSLIntercept
ssl_bump bump all
and whitelisting
#HTTP_HTTPS whitelist websites
acl whitelist ssl::server_name_regex "/usr/local/squid/etc/urlwhite.txt"
in both txt files ie
/usr/local/squid/etc/interceptssl.txt
/usr/local/squid/etc/urlwhite.txt
i have a URL that first i have to whitelist and then if i want squid not to inspect the url traffic i put it in the SSL interception (i do this as some websites dont like MITM )
but even putting the URL in question in both files im still having issues with this website ie its still being detected that its passing through a proxy
thanks,
rob
On Mon, 26 Jun 2023 at 23:35, <mailto:mailto:ngtech1ltd at gmail.com> wrote:
Hey Robert,
I am not sure what forward proxy setup you have there.
A simple forward proxy?
What tool are you using for whitelisting?
You can use an external acl helper to allow dynamic updates of the whitelists or
to periodic update your lists and reload.
It will depend on the size of your lists.
What OS are you using for your squid proxy?
More details will help us help you.
Eliezer
From: squid-users <mailto:mailto:squid-users-bounces at lists.squid-cache.org> On Behalf Of robert k Wild
Sent: Monday, June 26, 2023 22:25
To: Squid Users <mailto:mailto:squid-users at lists.squid-cache.org>
Subject: [squid-users] make URL bypass squid proxy
hi all,
i have set up squid for url whitelisting and no intercept SSL (see below)
https://wiki.squid-cache.org/ConfigExamples/Caching/AdobeProducts
but some websites i want the client to bypass the squid proxy and go straight to the website as i think this is why a url isnt working even when i add the url to both files ie urlwhite and no intercept SSL
thanks,
rob
--
Regards,
Robert K Wild.
--
Regards,
Robert K Wild.
--
Regards,
Robert K Wild.
More information about the squid-users
mailing list