[squid-users] ssl bump and url_rewrite_program (like squidguard)

Thu Nov 12 09:50:55 UTC 2015

Hi again,

Just forget what I said about REDIRECT answers, there are the same with 
or without SSL (it was a side effect of "-C5" on my logs grep).

But, why are browsers handling that in a different way?

Without SSL, it's all right. With SSL it's getting to the conclusion it 
should try to connect directly...

Regards, EG

Le 12/11/2015 10:03, Edouard Gaulué a écrit :
> Hi Marcus, Amos and maybe others,
>
> Here were I am. I've looked in the log. Let me describe what I 
> observe. It's maybe linked with some other posts I've read.
>
> Imagine I try to connect to http://ad.doubleclick.net/ad.jpg. I 
> observe the request in wireshark. It goes to the squid process: there 
> is no SSL involved so no bump. Squidguard sends its redirect to squid 
> and Squid sends to the browser :
> HTTP/1.1 302 Found
> Server: squid/3.5.10
> Date: Wed, 11 Nov 2015 22:49:44 GMT
> Content-Length: 0
> Location: 
> https://proxyweb.xxx.xxx/cgi-bin/squidGuard-simple.cgi?clientaddr=xxx&clientgroup=low-ip&targetgroup=adv&url=http://ad.doubleclick.net/ad.jpg
> X-Cache: MISS from squid
> X-Cache-Lookup: MISS from squid:3128
> Via: 1.1 squid (squid/3.5.10)
> Connection: keep-alive
>
> The browser next sends a request to proxyweb: everything is fine.
>
>
> Now let's add some SSL stuff and connect to 
> https://ad.doubleclick.net/ad.jpg. I observe the request in wireshark. 
> It goes to the squid process with SSL. Squidguard sends its redirect 
> to squid and Squid sends to the browser :
> HTTP/1.1 302 Found
> Server: squid/3.5.10
> Date: Wed, 11 Nov 2015 22:49:44 GMT
> Content-Length: 0
> Location: 
> https://proxyweb.xxx.xxx/cgi-bin/squidGuard-simple.cgi?clientaddr=xxx&clientgroup=low-ip&targetgroup=adv&url=http://ad.doubleclick.net/ad.jpg
> X-Cache: MISS from squid
>
> The browser next tries to send a direct request to 
> https://ad.doubleclick.net that is never ending as this machine hasn't 
> got the right to go on the Internet.
>
> Why is the REDIRECT not the same with and without SSL? It looks it 
> disturbs the navigator (at least Mozilla and IE).
>
> I can also provide squid logs, but tell me what because I've got a lot...
>
> Regards, EG
>
>
> Le 05/11/2015 14:01, Marcus Kool a écrit :
>>
>>
>> On 11/04/2015 08:55 PM, Edouard Gaulué wrote:
>>> Hi Marcus,
>>>
>>> Well that just an URL rewriter program. You can just test it from 
>>> the command line :
>>> echo "URL" | /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
>>>
>>> Before I understood it was possible to precise the redirect code I 
>>> got that:
>>> #> echo
>>> "https://ad.doubleclick.net/N4061/adi/com.ythome/_default;sz=970x250;tile=1;ssl=1;dc_yt=1;kbsg=HPFR151103;kga=-1;kgg=-1;klg=fr;kmyd=ad_creative_1;ytexp=9406852,9408210,9408502,9417689,9419444,9419802,9420440,9420473,9421645,9421711,9422141,9422865,9423510,9423563,9423789;ord=968558538238386? 
>>>
>>> - - GET"|/usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf
>>> #> OK
>>> rewrite-url="https://proxyweb.XXXXX.XXXXX/cgi-bin/squidGuard-simple.cgi?clientaddr=-pipo&clientname=&clientuser=&clientgroup=default&targetgroup=unknown&url=https://ad.doubleclick.net/N4061/adi/com.ythome/_default;sz=970x250;tile=1;ssl=1;dc_yt=1;kbsg=HPFR151103;kga=-1;kgg=-1;klg=fr;kmyd=ad_creative_1;ytexp=9406852,9408210,9408502,9417689,9419444,9419802,9420440,9420473,9421645,9421711,9422141,9422865,9423510,9423563,9423789;ord=968558538238386?" 
>>>
>>>
>>>
>>> After a little change in the squidguard.conf, I get:
>>> #> OK status=302
>>> url="https://proxyweb.echoppe.lan/cgi-bin/squidGuard-simple.cgi?clientaddr=-pipo&clientname=&clientuser=&clientgroup=default&targetgroup=unknown&url=https://ad.doubleclick.net/N4061/adi/com.ythome/_default;sz=970x250;tile=1;ssl=1;dc_yt=1;kbsg=HPFR151103;kga=-1;kgg=-1;klg=fr;kmyd=ad_creative_1;ytexp=9406852,9408210,9408502,9417689,9419444,9419802,9420440,9420473,9421645,9421711,9422141,9422865,9423510,9423563,9423789;ord=968558538238386?" 
>>>
>>
>> This looks fine, so now you need to look at Squid and set the debug 
>> options to find out what it is doing.
>>
>> Note that squidGuard does not percent-escape the URL parameter as it 
>> should (see RFC 3986).
>> This is, however, most likely not the cause of the issue that you are 
>> seeing.
>>
>> Marcus
>>
>>>
>>> It's not so better handled by my browser saying "can't connect to 
>>> https://ad.doubleclick.net" message. But, I don't get the squid 
>>> message anymore regarding http/https.
>>>
>>> It may be that rewrite_rule_program come after peek and splice stuff 
>>> leading squid to an unpredictable situation. Is there a way to play 
>>> on order things happen in squid?
>>>
>>> Regards, EG
>>>
>>>
>>> Le 04/11/2015 14:10, Marcus Kool a écrit :
>>>> You need to know what squidGuard actually sends to Squid.
>>>> squidGuard does not have a debug option for this, so you have to set
>>>>    debug_options ALL,1 61,9
>>>> in squid.conf to see what Squid receives.
>>>> I bet that what Squid receives, is what it complains about:
>>>> the URL starts with 'https://http'
>>>>
>>>> Marcus
>>>>
>>>> On 11/04/2015 10:55 AM, Edouard Gaulué wrote:
>>>>> Le 04/11/2015 11:00, Amos Jeffries a écrit :
>>>>>> On 4/11/2015 12:48 p.m., Marcus Kool wrote:
>>>>>>> I suspect that the problem is that you redirect a HTTPS-based 
>>>>>>> URL to an
>>>>>>> HTTP URL and Squid does not like that.
>>>>>>>
>>>>>>> Marcus
>>>>> To give it a try in that direction I now redirect to an https 
>>>>> server. And I get :
>>>>>
>>>>> The following error was encountered while trying to retrieve the 
>>>>> URL: https://https/*
>>>>>
>>>>>     *Unable to determine IP address from host name "https"*
>>>>>
>>>>> The DNS server returned:
>>>>>
>>>>>     Name Error: The domain name does not exist.
>>>>>
>>>>>
>>>>> Moreover this would leads sometimes to HTTP-based URL to an HTTPS 
>>>>> URL and I don't know how much squid likes it either.
>>>>>
>>>>>> No it is apparently the fact that the domain name being 
>>>>>> redirected to is
>>>>>> "http".
>>>>>>
>>>>>> As in:"http://http/something"
>>>>>>
>>>>> I can assure my rewrite_url looks like 
>>>>> "https://proxyweb.xxxxx.xxxxx/var1=xxxx&...".
>>>>>
>>>>> And this confirm ssl_bump parse this result and get the left part 
>>>>> before the ":". To play with, I have also redirect to 
>>>>> "proxyweb.xxxxx.xxxxx:443/var1=xxxx&..." (ie. I removed the 
>>>>> "https://" and add a
>>>>> ":443") to force the parsing. Then I don't get this message 
>>>>> anymore, but Mozilla gets crazy waiting for the ad.doubleclick.net 
>>>>> certificate and getting the proxyweb.xxxxx.xxxxx one. And of 
>>>>> course it
>>>>> breaks my SG configuration and can't be production solution.
>>>>>> Which brings up the question of why you are using SG to block 
>>>>>> adverts?
>>>>>>
>>>>>> squid.conf:
>>>>>>   acl ads dstdomain .doubleclick.net
>>>>>>   http_access deny ads
>>>>>>
>>>>>> Amos
>>>>>>
>>>>>>
>>>>> I don't use SG to specificaly block adverts, I use it to block 90 
>>>>> % of the web. Here it's just an example with ads but it could be 
>>>>> with so much other things...
>>>>>
>>>>> I just want to try make SG and ssl_bump live together.
>>>>>
>>>>> Is this possible to have a rule like "if it has been rewrite then 
>>>>> don't try to ssl_bump"?
>>>>>
>>>>> Regards, EG
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> squid-users mailing list
>>>>> squid-users at lists.squid-cache.org
>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>>
>>>
>>>
>>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users