[squid-users] HTTPS issues with squidguard after upgrading from squid 2.7 to 3.5

Wed Jun 15 18:12:15 UTC 2016

Hey Michael,

Well I have not tested FreeBSD dependencies and patches and I am not following them daily.

The issue itself with SquidGuard and the url_rewrite interface is more of an issue in most cases with CONNECT requests as you mentioned.

Since you are not using ssl_bump then you need to deny the traffic\requests in a way that will not leave squid or the clients and the session in an unknown or unexpected situation.

When the url_rewrite interface is used to "Deny" something it's not really denying but rather "rewriting" something due to it's nature.

On regular plain http requests some mangling to the request(affecting the response) is possible but when handling CONNECT requests it's a whole new story.

We don't know how the client will respond to a malformed response or squid to a malformed rewritten request, and it is possible that the client will expect a specific 50x\40x response code.

The external_acl interface was built as an ACL extension with the basic ability to overcome the limits of the url_rewrite interface.

Content or url filtering is an ACL level operation and not url rewriting\mangling and there for it is better(in squid) to do it in a way that the client can identify(30x).

The best possible way to deny such a connection(CONNECT) is using a 50x or 40x code.
I do not remember which one is accepted better by browsers and clients and you will be able to find it yourself easily adding couple lines or using the scripts I wrote before.

And more directly to the subject, I will say that if these network "issues" are might be because of any squid side of wrongly handling CONNECT requests that was mangled with a url_rewrite.
I would say that these specific cases is the proof of the wrongly used interface.

The url_rewrite might contain a bug when handling a CONNECT request but the bug is that it is handling these connections at all and not otherwise.

My suggestion is that you would try to see if you can use something like that:

## START

url_rewrite_program /usr/local/bin/squidGuard url_rewrite_children 8 startup=4 idle=4 concurrency=0

url_rewrite_access deny CONNECT

url_rewrite_access allow all

## END

And  add another external_acl helper with the wrapper I gave you without any special deny_info for the ACL.
Squid will be able to handle these probably fine and will deny the connections using some right access deny code(don't expect fancy warning pages without using some level of ssl_bump).

I do not know if you need a fast solution or you are planning carefully with the available options.

The options are to either use a "fast" solution to mitigate an annoying issue or to find the right path.

The solution to either of the options is in your hands..

If for your squidGuard setup offers you the right solution from all the right aspects despite to the fact that it's working slower then other solutions then it is the solution for you!!

The technology of black and white listing was enhanced in the last decade but not too drastically.

The basic concept is that there are query and analysis components which are not forced to be one software.

There will always be false positives to this or another way but some categorizing systems are much stricter and sensitive about the subject then others.

All this leaving aside the sources of the lists you and since only you have a definition of the wanted\desired results with the available options.

You also stated that you have some funding issues, but just as a side note: the wanted result is eventually forcing the final product of work and the expense(any if at all).

I mean with these words that if you do not care about false positives and using some product satisfy the desire or need then I think it's ok for your case.

I will add that from my tests there are different results to basic pages\objects loading\access speed when using one software or another, and\or one protocol\interface or another.

Here if you need more help\advice handling the situation.

Eliezer

----

Eliezer Croitoru

Linux System Administrator

Mobile: +972-5-28704261

Email: eliezer at ngtech.co.il

-----Original Message-----
From: michail.pappas at gmail.com [mailto:michail.pappas at gmail.com] On Behalf Of reqman
Sent: Wednesday, June 15, 2016 2:19 PM
To: Eliezer Croitoru
Cc: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] HTTPS issues with squidguard after upgrading from squid 2.7 to 3.5

Hello Eliezer,

2016-06-15 11:45 GMT+03:00 Eliezer Croitoru < <mailto:eliezer at ngtech.co.il> eliezer at ngtech.co.il>:

> Hey Michael,

> 

> I am missing couple details about the setup which might affect the way we would be able to understand what is causing the issue and how to resolve it.

> There are changes from squid 2.7 to 3.5 and to my opinion these are mandatory to resolve and to not go one step back.

Yes, I saw that 3.5 effectively disabled/obsoleted/deactivated various options. I believe I took care in following through those requirements.

> What version of SquidGuard 1.4 did you installed? The patched for squid 3.4+ compatibility?

I installed a ready to use package from FreeBSD. I presume that is is ok with squid 3.4, according to its dependencies:

# pkg query "%do%dv" squidGuard

www/squid3.5.19

databases/db55.3.28_3

Other information for squidGuard:

# pkg info squidGuard

squidGuard-1.4_15

Name           : squidGuard

Version        : 1.4_15

Installed on   : Mon May 30 08:24:17 2016 EEST

Origin         : www/squidguard

Architecture   : freebsd:10:x86:64

Prefix         : /usr/local

Categories     : www

Licenses       : GPLv2

Maintainer     :  <mailto:garga at FreeBSD.org> garga at FreeBSD.org

WWW            :  <http://www.squidguard.org/> http://www.squidguard.org/

Comment        : Fast redirector for squid

Options        :

       DNS_BL         : off

       DOCS           : on

       EXAMPLES       : on

       LDAP           : off

       QUOTE_STRING   : off

       STRIP_NTDOMAIN : off

Shared Libs required:

       libdb-5.3.so.0

Annotations    :

       repo_type      : binary

       repository     : FreeBSD

Flat size      : 2.24MiB

> More details about it here: 

>  <http://bugs.squid-cache.org/show_bug.cgi?id=3978> http://bugs.squid-cache.org/show_bug.cgi?id=3978

> Now if it is indeed patched and works as expected it from the 3.4+ computability level of things then lets move on.

Checking the bug and if I understand correctly, if my squidGuard was not patched then it wouldn't work. This is not the case. It works ok for http urls, ie blocks fine and the user is correctly redirected to the block page setup for this purpose. It's HTTPS I'm having issues

with: according to some talks, if something gets blocked by squidguard, something is miscommunicated (or not communicated at all) with squid. Instead of a block, the users waits endlessly for the page.

> Are you using Squid in intercept\transparent\trpoxy mode or is it defined in the browsers directly?

> If you are using intercept mode, what have you defined on the FreeBSD pf\ipfw?

Squid operates in normal mode. I've configured a 10year old proxy autodiscovery script, which is published through DHCP. All browsers are set to configure everything automatically.

> And about the quote from the mailing list:

> SquidGuard was written to operate under the url_rewrite interface\protocol and not external_acl.

I've lost you here: why do I need squidGuard to operate under external_acl? Can't I leave it running with url_rewrite?

> Due to this it has some disadvantages and the advised details are to modify the helper(SquidGuard or another) to operate in another way.

> It is possible to use the patched version of SquidGuard under the external_acl interface and use squid options to deny\redirect the request.

I saw your other email, but I again have to ask. My own squidguard "seems" to work. What are the merits of doing things with external_acl instead of the way I am doing things right now?

> It removes some things from the complexity of the issue.

> I have just written an example on how to use use my software SquidBlocker under external_acl and here the adapted example that can be used with a patched SquidGuard:

> ## START OF SETTINGS

> external_acl_type filter_url ipv4 concurrency=0 ttl=3 %URI %SRC/- 

> %LOGIN %METHOD /usr/local/bin/squidGuard url_rewrite_children acl 

> filter_url_acl external filter_url deny_info 

>  <http://ngtech.co.il/block_page/?url=%25u&domain=%25H> http://ngtech.co.il/block_page/?url=%u&domain=%H filter_url_acl #or # 

> deny_info 302:http://ngtech.co.il/block_page/?url=%u&domain=%H 

> filter_url_acl http_access deny !filter_url_acl http_access allow 

> localnet filter_url_acl ## END OF SETTINGS

> 

> I have not tested this request format but if it doesn't work this way then a little cosmetics will make it work.

> 

> When more information will be available we can try to see where the issue is from.

> 

> Eliezer

> 

> ----

> Eliezer Croitoru

> Linux System Administrator

> Mobile: +972-5-28704261

> Email:  <mailto:eliezer at ngtech.co.il> eliezer at ngtech.co.il

> 

> 

> -----Original Message-----

> From: squid-users [ <mailto:squid-users-bounces at lists.squid-cache.org> mailto:squid-users-bounces at lists.squid-cache.org] 

> On Behalf Of reqman

> Sent: Wednesday, June 15, 2016 10:22 AM

> To:  <mailto:squid-users at lists.squid-cache.org> squid-users at lists.squid-cache.org

> Subject: [squid-users] HTTPS issues with squidguard after upgrading 

> from squid 2.7 to 3.5

> 

> Hello all,

> 

> I have been running squid 2.7.X alongside squidguard 1.4 on a FreeBSD 8.x box for years.

> Started out some 10 years ago, with a much older squid/squidguard/FreeBSD combination.

> 

> Having to upgrade to FreeBSD 10.3, I examined my option regarding squid.

> 3.5.19 was available which I assumed would behave the same as 2.7, regarding compatibility.

> Squidguard 1.4 was also installed.

> 

> - Squid was configured to behave along the lines of what I had on 2.7.

> - For squidguard I used the exact same blocklists and configurations.

> Note that I do not employ an URL rewriting in squidguard, only redirection.

> - no SSL-bump or other SSL interception takes place

> - the squidguard-related lines on squid are the following:

> 

> url_rewrite_program /usr/local/bin/squidGuard url_rewrite_children 8 

> startup=4 idle=4 concurrency=0 url_rewrite_access allow all

> 

> - In squidGuard.conf, the typical redirect section is like:

> 

>  default {

>                 pass local-ok !block1 !block2 !blockN all

>                 redirect

> 301:http://localsite/block.htm?clientaddr=%a+clientname=%n+clientident=%i+srcclass=%s+targetclass=%t+url=%u

>         }

> 

> I am now experiencing problems that I did not have. Specifically, access to certain but *not* all HTTPS sites seems to timeout.

> Furthermore, I see entries similar to the following in cache.log:

> 

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3446 FD 591 flags=1

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3448 FD 592 flags=1

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3452 FD 594 flags=1

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3456 FD 596 flags=1

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3454 FD 595 flags=1

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3458 FD 597 flags=1

> 2016/06/15 09:27:59 kid1| abandoning local=192.168.0.1:3128

> remote=192.168.2.239:3462 FD 599 flags=1

> 

> Searching around, the closest I have come to an answer is the

> following: 

>  <http://www.squid-cache.org/mail-archive/squid-users/201211/0165.html> http://www.squid-cache.org/mail-archive/squid-users/201211/0165.html

> I am not sure though whether I am plagued by the same issue, considering that the thread refers to a squid version dated 4 years ago. And I definitely do not understand what the is meant by the poster's proposal:

> 

> "If you can't alter the re-writer to perform redirection you can work around that by using:

> 

>   acl foo ... some test to match the re-written URL ...

>   deny_info 302:%s foo

>   adapted_http_access deny foo "

> 

> Can someone help resolve this?

> Is the 2.7 series supported at all?

> As is if everything fails, I'll have to go back to it if there's some support.

> 

> BR,

> 

> 

> Michael.-

> _______________________________________________

> squid-users mailing list

>  <mailto:squid-users at lists.squid-cache.org> squid-users at lists.squid-cache.org

>  <http://lists.squid-cache.org/listinfo/squid-users> http://lists.squid-cache.org/listinfo/squid-users

> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160615/7f60c8ec/attachment-0001.html>