[squid-users] make URL bypass squid proxy
robert k Wild
robertkwild at gmail.com
Thu Jul 6 12:29:51 UTC 2023
hi Eliezer,
unfortunately that did not work ie
acl NoSSLInterceptRegEx ssl::server_name_regex (^|.*\.)redshift3d\.com$
acl NoSSLInterceptDstDom ssl::server_name .redshift3d.com
acl NoSSLInterceptAnyOf any-of NoSSLInterceptDstDom NoSSLInterceptRegEx
ssl_bump peek DiscoverSNIHost
ssl_bump splice NoSSLInterceptAnyOf
ssl_bump bump all
but strange is on the firewall if i make a rule for that network, port 80,
443 to access redshift3d.com it works, so ive narrowed it down to the squid
thanks,
rob
On Thu, 29 Jun 2023 at 11:40, <ngtech1ltd at gmail.com> wrote:
> Hey Rob,
>
> it’s a great question.
> I will assume you are using squid 5.x since it’s the stable one.
> There is a configuration reference documentation at:
> http://www.squid-cache.org/Versions/v5/cfgman/
>
> And the relevant one is ‘acl’:
> http://www.squid-cache.org/Versions/v5/cfgman/acl.html
>
> In the config we the next options:
> * dstdom_regex
> * dstdomain
> * ssl::server_name
> * ssl::server_name_regex
>
> Each one of them is different in two things.
> The dstdomain is happening when the http request verification is done,
> usually in the http_access part.
> So for a CONNECT request in your case it should work but will only work
> for http_access rules ie allow or deny the connection to being proxied by
> the proxy.
> All this happens before the bump stage.
> The dstdom_regex is doing the same but with another "lookup" mechanism.
> Just to illustrate we will use the domain www.example.com.
>
> With dstdomain .example.com which is a wildcard domain the www.example
> .com will match since what it does it..
> Take apart the .example.com into .com and .example and example
> (logically, not the code actually does..)
> Then when a request for www.example-2.com arrives the dstdomain acl
> checks the strings one by one ie:
> com
> example-2
> www
>
> So the "com" would match and since it has a dot "." as a prefix it
> continues to the next part of the domain name ie:
> example-2
>
> and it will try to match it to "example" from the ".example.com'.
> then it will fail and there for it will declare the specific definition in
> the acl as "do not match.
> But for www.example .com it would be different:
>
> Squid will find that the .com is in the "com" part of www.example.com and
> will continue to the next part ie
> "example" and it will match.
> Then it will see the dot "." which means that all subdomains are a part of
> this rule ie both example.com and any other domain
> under .example.com will match so... www.example.com and www2.example.com
> and w.example.com will match and this is compared to
> example.com
>
> with dstdomain example.com it will only match a full match of example.com
> but not www.example.com and it's pointless to define both:
> - example.com
> - .example.com
>
> In the same acl file.
> Squid will compare it to the basic proxy url request and the Host: header
> inside of it.
> (Amos might remember which one of these wins or which one is the main one)
>
> So that's how a dstdomain test works.
> The dstdom_regex is applied on the same "part" of the proxy request ie
> request url and Host header but...
> Tries to match it with a regex that is a much more "CPU" intensive but can
> do magic if you know how to use it right.
> The pros are that you can use one regex to match about 100 domains however
> dstdomain is much faster in many cases.
> To test regex in general I love:
> https://rubular.com/
>
> and it's very simple to put a line with the url or the domain itself and
> write the pattern and then get instant result.
> There are many sites which does this exact same thing.
> You need: activate.redshift3d.com for the test.
>
> For the server_name dstdom and regex the difference is on what part of the
> "known" information on the request done.
> Ie it's not on the known plain text proxy request but a "peek" into the
> client or server TLS part of the connection bumping.
> It's possible on TLS 1.2 with ease but 1.3 with encryption it's a whole
> other story.
> (Alex and his team is more involved in this part then me)
>
> It's a bit more complex in the server_name part since from what I remember
> the server_name can be the SNI itself or parts of the
> certificate of the server.
> In the certificate of the server in many cases there is no real domain but
> a catch all ie multidomain which is for example:
> *.example.com
>
> The ssl::server_name is like dstdomain so .example.com should match both
> SNI with www.example.com and the *.example.com
> If it doesn't then it’s a bug..
>
> About the ssl::server_name_regex you will need to test and match the regex
> against couple possibilities that are in the certificate ie:
> *.example.com
> activation.example.com
>
> etc...
>
> The current certificate I see is:
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number:
> c0:71:02:fc:1f:e5:64:5b:0d:d6:ae:c8:1a:17:e6:80
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: C = GB, ST = Greater Manchester, L = Salford, O = Sectigo
> Limited, CN = Sectigo RSA Domain Validation Secure Server CA
> Validity
> Not Before: Dec 14 00:00:00 2022 GMT
> Not After : Jan 12 23:59:59 2024 GMT
> Subject: CN = activate.redshift3d.com
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> RSA Public-Key: (2048 bit)
> Modulus:
> 00:cd:f9:bd:58:a4:7a:16:3f:23:12:9d:e0:1e:39:
> a0:ef:66:ae:b3:ae:32:5c:69:9e:cc:89:83:fb:a2:
> e0:c7:9d:86:aa:e6:9b:b3:d9:0c:bb:35:36:2f:cf:
> be:ec:1e:62:ca:d8:ab:16:66:6e:00:8d:f7:42:3d:
> 5b:d6:bf:a0:30:a1:c2:6f:5e:69:76:ef:0c:38:73:
> d2:e8:42:b8:83:04:a3:2c:da:22:a4:c4:13:71:38:
> 36:00:1b:e7:b3:f0:fe:6e:59:17:11:e7:0a:81:3e:
> 04:b2:ad:e0:61:c1:15:f3:9a:36:33:24:a6:f1:0d:
> b6:e6:32:91:34:54:7c:da:40:14:72:06:25:91:78:
> 75:07:84:62:47:7b:3f:05:60:e6:35:d3:27:55:1e:
> ff:23:82:18:09:f5:8a:0f:a6:cb:02:bc:d6:09:98:
> 54:5b:10:e3:d2:09:4e:31:2a:75:42:29:00:86:9b:
> f9:78:2f:fd:99:9e:2e:10:3e:bc:84:f5:9b:3a:3f:
> fa:d9:b6:20:c0:2e:86:f3:de:14:be:3a:1a:05:30:
> 3b:00:e6:50:8d:4c:13:a3:97:dc:f4:03:9e:84:31:
> 0d:23:f2:02:50:6d:53:a5:30:03:b0:fd:f2:46:64:
> 49:22:a4:d1:5a:00:fb:78:37:fc:51:b1:f0:6e:41:
> 3b:5d
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Authority Key Identifier:
>
> keyid:8D:8C:5E:C4:54:AD:8A:E1:77:E9:9B:F9:9B:05:E1:B8:01:8D:61:E1
>
> X509v3 Subject Key Identifier:
> 5D:BC:D9:F2:C4:99:83:5E:C0:3A:F1:BF:FE:F2:E3:92:92:F1:F0:29
> X509v3 Key Usage: critical
> Digital Signature, Key Encipherment
> X509v3 Basic Constraints: critical
> CA:FALSE
> X509v3 Extended Key Usage:
> TLS Web Server Authentication, TLS Web Client
> Authentication
> X509v3 Certificate Policies:
> Policy: 1.3.6.1.4.1.6449.1.2.2.7
> CPS: https://sectigo.com/CPS
> Policy: 2.23.140.1.2.1
>
> Authority Information Access:
> CA Issuers - URI:
> http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt
> OCSP - URI:http://ocsp.sectigo.com
>
> X509v3 Subject Alternative Name:
> DNS:activate.redshift3d.com, DNS:
> www.activate.redshift3d.com
> CT Precertificate SCTs:
> Signed Certificate Timestamp:
> Version : v1 (0x0)
> Log ID :
> 76:FF:88:3F:0A:B6:FB:95:51:C2:61:CC:F5:87:BA:34:
>
> B4:A4:CD:BB:29:DC:68:42:0A:9F:E6:67:4C:5A:3A:74
> Timestamp : Dec 14 15:34:39.227 2022 GMT
> Extensions: none
> Signature : ecdsa-with-SHA256
>
> 30:44:02:20:6E:D0:96:68:F3:07:A0:5F:DD:C5:B2:61:
>
> 2F:51:6B:06:4A:3C:7F:9B:DA:08:6D:5A:31:0D:B8:0B:
>
> 83:7D:D6:37:02:20:64:66:3F:A2:40:87:78:9E:C8:90:
>
> 7C:EE:7C:77:3C:BB:43:C1:9F:52:54:F1:04:85:B6:AC:
> 6B:F2:36:8E:E3:03
> Signed Certificate Timestamp:
> Version : v1 (0x0)
> Log ID :
> 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
>
> 67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
> Timestamp : Dec 14 15:34:39.153 2022 GMT
> Extensions: none
> Signature : ecdsa-with-SHA256
>
> 30:44:02:20:69:BA:F2:DD:32:B3:48:0C:26:E3:FB:F0:
>
> DD:4E:5E:10:95:1E:B4:59:5B:67:64:C4:F2:40:7C:A8:
>
> 62:35:71:FE:02:20:13:15:41:26:E5:9C:DB:34:DA:D7:
>
> B9:CB:B3:1A:4E:33:C7:46:7F:D9:93:45:25:7F:DE:A3:
> 72:EB:07:03:DB:C5
> Signed Certificate Timestamp:
> Version : v1 (0x0)
> Log ID :
> EE:CD:D0:64:D5:DB:1A:CE:C5:5C:B7:9D:B4:CD:13:A2:
>
> 32:87:46:7C:BC:EC:DE:C3:51:48:59:46:71:1F:B5:9B
> Timestamp : Dec 14 15:34:39.124 2022 GMT
> Extensions: none
> Signature : ecdsa-with-SHA256
>
> 30:45:02:20:2C:E2:85:9C:A6:54:1B:1C:31:E5:F8:37:
>
> E9:CD:09:8B:D8:26:29:E4:C7:65:94:9C:FF:32:D2:41:
>
> CD:16:A3:51:02:21:00:A0:2F:C3:F7:A6:55:3B:21:EB:
>
> 9B:CA:6E:4E:07:A2:8C:40:4B:E2:27:D6:82:44:0F:09:
> C9:F7:7D:1B:72:6F:13
> Signature Algorithm: sha256WithRSAEncryption
> 25:bd:bb:de:57:c0:7f:07:5e:18:62:2e:0b:d3:03:54:a7:45:
> ab:c6:1f:e2:f6:58:ff:6e:8e:6b:4f:09:9a:87:66:32:81:7f:
> f4:35:4f:7e:65:e5:6a:04:d6:62:62:ff:d9:3a:f2:6f:19:ba:
> fa:e6:35:0e:2a:44:5c:3b:ee:9d:97:72:05:86:0c:4c:01:c1:
> f0:8c:21:c1:c4:84:54:d8:a8:05:25:18:72:db:f7:53:9b:f1:
> 13:d6:0b:bc:92:6e:01:e3:fd:de:a1:45:e9:29:37:e1:2e:64:
> 36:b4:4d:38:c1:60:02:6a:17:3d:87:a2:5f:33:3b:86:eb:0d:
> cc:dd:fa:d4:43:58:50:43:e7:b7:ec:0a:4f:86:72:15:e5:30:
> c9:bb:5f:0b:83:9c:26:6f:60:49:dd:1a:7c:92:45:45:4e:b5:
> ce:cd:64:8c:12:83:e9:3d:5c:6b:65:97:75:99:4c:66:eb:d0:
> 3a:ca:18:62:8a:08:07:16:ab:09:66:bd:65:43:94:00:d9:79:
> 3e:84:b4:60:7d:7e:f9:09:3c:fe:2d:ad:98:94:17:0c:24:8f:
> e1:a2:74:b6:3b:68:c0:01:f9:67:e8:b9:d2:6a:65:9e:99:a3:
> 4a:5f:39:31:ae:c1:59:02:7b:ef:db:b2:94:06:f8:1a:74:c1:
> d7:5b:5b:6a
>
> So the DNS names it will check are:
> Subject: CN = activate.redshift3d.com
> DNS:activate.redshift3d.com, DNS:
> www.activate.redshift3d.com
>
> So to summarize the checks of ssl::server_name/ will be done on:
> * activate.redshift3d.com
> * activate.redshift3d.com
> * www.activate.redshift3d.com
>
> So . redshift3d.com ssl::server_name should match the certificate.
> If for any reason it doesn't work you can try ssl::server_name_regex with
> something like:
> (^|\.)activate\.redshift3d\.com$
>
> Or just to verify if there is a bug in squid code try:
> (^|\.)activate\.redshift3d\.com
>
> Now, the splice should be able to take into account also dstdomain and
> dstdom_regex but it should match them only if they exist in a plain text
> form like in any simple forward proxy CONNECT request.
> If for any reason it doesn’t work we should investigate what might cause
> this issue.
>
> I hope the scroll I wrote make sense to you and with hopes it will clear
> out the doubts about the wiki article you mentioned.
> I believe this is considered a summary of the subject and if Alex and
> others might think so it can be converted into an example article in the
> wiki.
>
> Let me know if this makes sense and resolve the issue.
>
> Yours,
> Eliezer
>
> From: robert k Wild <robertkwild at gmail.com>
> Sent: Thursday, June 29, 2023 12:18
> To: ngtech1ltd at gmail.com
> Cc: Squid Users <squid-users at lists.squid-cache.org>
> Subject: Re: [squid-users] make URL bypass squid proxy
>
> very clever, so you bunch all the acls up
>
> acl NoSSLInterceptAnyOf any-of NoSSLInterceptDstDom
> NoSSLInterceptDstDomFile NoSSLInterceptRegEx NoSSLInterceptRegExFile
>
> the key word is "any-of" ie if the url hits any one do that first
>
> what about instead of making it
>
> ssl::server_name_regex
>
> i make it
>
> dstdom_regex
>
> On Thu, 29 Jun 2023 at 01:38, <mailto:ngtech1ltd at gmail.com> wrote:
> Hey Rob,
>
> The first thing is to allow the domain in the http_acces just to be sure
> and use a basic deny all bottom line.
> Let me try to simplify your squid.conf
> In a link:
> https://gist.github.com/elico/b49f4a28d4b5db5ba882b10d40872d5e
>
> In plain text:
> ## START OF FILE
> # SSL Interception basic rules
> acl DiscoverSNIHost at_step SslBump1
>
> acl NoSSLInterceptRegEx ssl::server_name_regex (^|.*\.)redshift3d\.com$
> acl NoSSLInterceptRegExFile ssl::server_name_regex
> "/usr/local/squid/etc/no-intercept-ssl-regex.txt"
>
> acl NoSSLInterceptDstDom ssl::server_name .redshift3d.com
> acl NoSSLInterceptDstDomFile ssl::server_name
> "/usr/local/squid/etc/no-intercept-ssl-dstdom.txt"
>
> ## Any of will test what ever rule match first in a first match/hit fasion
> acl NoSSLInterceptAnyOf any-of NoSSLInterceptDstDom
> NoSSLInterceptDstDomFile NoSSLInterceptRegEx NoSSLInterceptRegExFile
>
> ssl_bump peek DiscoverSNIHost
> ssl_bump splice NoSSLInterceptAnyOf
> ssl_bump bump all
>
> #SSL Bump port
> http_port 3128 ssl-bump cert=/usr/local/squid/etc/ssl_cert/myCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s
> /var/lib/ssl_db -M 4MB
>
> ## http_access acls, will apply on incomming requests and not on responses
> acl special_url_regex url_regex https?://(^|.*\.)redshift3d\.com\/
> acl special_url_regex_file url_regex
> "/usr/local/squid/etc/special_url_regex.txt"
>
> acl special_url_dst_dom dstdomain .redshift3d.com
> acl special_url_dst_dom_file dstdomain
> "/usr/local/squid/etc/special_url_dstdom.txt"
>
> acl special_url_any_of any-of special_url_dst_dom special_url_dst_dom_file
> special_url_regex special_url_regex_file
>
> acl localnet src http://192.168.0.0/16
> acl localnet src http://10.0.0.0/8
>
> http_access allow localnet special_url_any_of
> http_access deny all
> ## END OF FILE
>
> Once the above will work try to add other http_access rule like reply
> access rules
>
> Let me know what happens,
> Eliezer
>
> From: robert k Wild <mailto:robertkwild at gmail.com>
> Sent: Tuesday, June 27, 2023 09:36
> To: mailto:ngtech1ltd at gmail.com
> Cc: Squid Users <mailto:squid-users at lists.squid-cache.org>
> Subject: Re: [squid-users] make URL bypass squid proxy
>
> Hi Eliezer,
>
> this is a snippet of my whitelist and no intercept SSL config
>
> #SSL Interception
> acl DiscoverSNIHost at_step SslBump1
> acl NoSSLIntercept ssl::server_name_regex
> "/usr/local/squid/etc/interceptssl.txt"
> ssl_bump peek DiscoverSNIHost
> ssl_bump splice NoSSLIntercept
> ssl_bump bump all
> #
> #SSL Bump
> http_port 3128 ssl-bump cert=/usr/local/squid/etc/ssl_cert/myCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s
> /var/lib/ssl_db -M 4MB
> #
> #deny up MIME types
> acl upmime req_mime_type "/usr/local/squid/etc/mimedeny.txt"
> #
> #deny URL links
> acl url_links url_regex "/usr/local/squid/etc/linksurl.txt"
> #
> #allow special URL paths
> acl special_url url_regex "/usr/local/squid/etc/urlspecial.txt"
> #
> #deny down MIME types
> acl downmime rep_mime_type "/usr/local/squid/etc/mimedeny.txt"
> #
> http_reply_access allow special_url
> http_reply_access deny downmime
> #http_access deny upmime
> #http_access deny url_links
> #
> #HTTP_HTTPS whitelist websites
> acl whitelist ssl::server_name_regex "/usr/local/squid/etc/urlwhite.txt"
> #
> http_access allow activation whitelist
> http_access deny all
>
> so basically no SSL interception
>
> #SSL Interception
> acl DiscoverSNIHost at_step SslBump1
> acl NoSSLIntercept ssl::server_name_regex
> "/usr/local/squid/etc/interceptssl.txt"
> ssl_bump peek DiscoverSNIHost
> ssl_bump splice NoSSLIntercept
> ssl_bump bump all
>
> and whitelisting
>
> #HTTP_HTTPS whitelist websites
> acl whitelist ssl::server_name_regex "/usr/local/squid/etc/urlwhite.txt"
>
> in both txt files ie
>
> /usr/local/squid/etc/interceptssl.txt
> /usr/local/squid/etc/urlwhite.txt
>
> i have a URL that first i have to whitelist and then if i want squid not
> to inspect the url traffic i put it in the SSL interception (i do this as
> some websites dont like MITM )
>
> but even putting the URL in question in both files im still having issues
> with this website ie its still being detected that its passing through a
> proxy
>
> thanks,
> rob
>
> On Mon, 26 Jun 2023 at 23:35, <mailto:mailto:ngtech1ltd at gmail.com> wrote:
> Hey Robert,
>
> I am not sure what forward proxy setup you have there.
> A simple forward proxy?
> What tool are you using for whitelisting?
> You can use an external acl helper to allow dynamic updates of the
> whitelists or
> to periodic update your lists and reload.
> It will depend on the size of your lists.
> What OS are you using for your squid proxy?
>
> More details will help us help you.
>
> Eliezer
>
> From: squid-users <mailto:mailto:squid-users-bounces at lists.squid-cache.org>
> On Behalf Of robert k Wild
> Sent: Monday, June 26, 2023 22:25
> To: Squid Users <mailto:mailto:squid-users at lists.squid-cache.org>
> Subject: [squid-users] make URL bypass squid proxy
>
> hi all,
>
> i have set up squid for url whitelisting and no intercept SSL (see below)
>
> https://wiki.squid-cache.org/ConfigExamples/Caching/AdobeProducts
>
> but some websites i want the client to bypass the squid proxy and go
> straight to the website as i think this is why a url isnt working even when
> i add the url to both files ie urlwhite and no intercept SSL
>
>
>
> thanks,
> rob
>
> --
> Regards,
>
> Robert K Wild.
>
>
> --
> Regards,
>
> Robert K Wild.
>
>
> --
> Regards,
>
> Robert K Wild.
>
>
--
Regards,
Robert K Wild.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20230706/95040306/attachment-0001.htm>
More information about the squid-users
mailing list