[squid-dev] [PATCH] Fix external_acl problems

Mon Feb 1 13:32:38 UTC 2016

Hi Christos,

Sorry my apologies - I had my build env a bit mixed up. Anyway I’ve cleared that down and re-applied the patch - it’s all working now which is excellent news (http_port / CONNECT and https_port / transparent/intercept).

For clarity - 

Reviewion: squid-4.0.4-20160111-r14487
Patch applied cleanly with
patch -p0 < ../cert_subject_gone-t4.patch 

It wouldn’t apply cleanly to the latest nightly.

May be related - the %>ru field in the ACL is now logged as host:port for http_port/CONNECT and ip:port for https_port/transparent - is the ‘:port’ expected? If so then I will modify our external ACL to strip it off before performing comparison.

When are you targeting 4.0 to be released?

On 01/02/2016 09:52, "Christos Tsantilas" <chtsanti at gmail.com on behalf of christos at chtsanti.net> wrote:

>Hi Dave,
>   Did you apply my latest t4 patch? The cert_subject_gone-t4.patch?
>Did you apply it over a clean tree?
>Did you get any error while applying the patch?
>Which trunk revision are you using?
>Can we have your squid configuration?
>
>
>
>On 02/01/2016 10:55 AM, Dave Lewthwaite wrote:
>> Hi Christos,
>>
>> Still no luck I’m afraid - is it worth creating a patch against a newer nightly or has not enough code changed in this area to make a difference?
>>
>> 2016/02/01 08:34:25 kid1| assertion failed: ../src/base/Lock.h:48: "count_ > 0"
>>
>> #0  0x00007ffff53da5d7 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>> #1  0x00007ffff53dbcc8 in __GI_abort () at abort.c:90
>> #2  0x00000000005858cf in xassert (msg=<optimized out>, file=<optimized out>, line=<optimized out>) at debug.cc:556
>> #3  0x000000000050d054 in unlock (this=0x2074c30) at ../src/base/Lock.h:48
>> #4  AccessLogEntry::~AccessLogEntry (this=0x2046d20, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at AccessLogEntry.cc:82
>> #5  0x000000000050d0f9 in AccessLogEntry::~AccessLogEntry (this=0x2046d20, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at AccessLogEntry.cc:87
>> #6  0x00000000006c1434 in dereference (newP=0x0, this=0x2053c88) at ../../src/base/RefCount.h:97
>> #7  ~RefCount (this=0x2053c88, __in_chrg=<optimized out>) at ../../src/base/RefCount.h:35
>> #8  ACLFilledChecklist::~ACLFilledChecklist (this=0x2053ad8, __in_chrg=<optimized out>) at FilledChecklist.cc:50
>> #9  0x00000000006c15a9 in ACLFilledChecklist::~ACLFilledChecklist (this=0x2053ad8, __in_chrg=<optimized out>) at FilledChecklist.cc:67
>> #10 0x00000000006f69ce in ACLChecklist::checkCallback (this=0x2053ad8, answer=...) at Checklist.cc:174
>> #11 0x00000000005a483a in fqdncacheCallback (f=f at entry=0x2053540, wait=wait at entry=20) at fqdncache.cc:316
>> #12 0x00000000005a4c4a in fqdncacheHandleReply (data=<optimized out>, answers=<optimized out>, na=<optimized out>, error_message=<optimized out>) at fqdncache.cc:407
>> #13 0x000000000058ad2d in idnsCallback (q=q at entry=0x204ee68, error=error at entry=0x0) at dns_internal.cc:1144
>> #14 0x000000000058fd66 in idnsGrokReply (buf=buf at entry=0xbc07e0 <idnsRead(int, void*)::rbuf> "\333\200", sz=sz at entry=267) at dns_internal.cc:1313
>> #15 0x0000000000590a4f in idnsRead (fd=20) at dns_internal.cc:1400
>> #16 0x0000000000811c70 in Comm::DoSelect (msec=<optimized out>) at ModEpoll.cc:273
>> #17 0x000000000075836e in CommSelectEngine::checkEvents (this=<optimized out>, timeout=<optimized out>) at comm.cc:1829
>> #18 0x0000000000599b39 in EventLoop::checkEngine (this=this at entry=0x7fffffffe3b0, engine=engine at entry=0x7fffffffe330, primary=primary at entry=true) at EventLoop.cc:36
>> #19 0x0000000000599d75 in EventLoop::runOnce (this=this at entry=0x7fffffffe3b0) at EventLoop.cc:115
>> #20 0x0000000000599f80 in EventLoop::run (this=this at entry=0x7fffffffe3b0) at EventLoop.cc:83
>> #21 0x0000000000606d71 in SquidMain (argc=<optimized out>, argv=<optimized out>) at main.cc:1638
>> #22 0x00000000004ff12d in SquidMainSafe (argv=<optimized out>, argc=<optimized out>) at main.cc:1363
>> #23 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1356
>>
>>
>> Thanks!
>>
>>
>>
>> On 29/01/2016 15:26, "squid-dev on behalf of Dave Lewthwaite" <squid-dev-bounces at lists.squid-cache.org on behalf of Dave.Lewthwaite at realitymine.com> wrote:
>>
>>> Hi Christos,
>>>
>>> I’ve tried your patch (I had to go back to nightly r14487 for the patch to apply cleanly so this may have something to do with my findings)
>>>
>>> http_port / CONNECT proxy works correctly with peek/splice decision making and all the correct information being available - of note is the %>ru field is now represented as FQDN:443 instead of just the FQDN (I don’t know if this is intentional or expected, but may be misleading).
>>>
>>> https_port/transparent on the other hand immediately crashes (see log/bt below) - although this may be related to running on the older nightly. I’ll try again if you have a clean patch for latest nightly and/or it’s being committed to trunk.
>>>
>>> 2016/01/29 15:21:48| assertion failed: ../src/base/Lock.h:48: "count_ > 0"
>>>
>>> #0  0x00007ffff53da5d7 in raise () from /lib64/libc.so.6
>>> #1  0x00007ffff53dbcc8 in abort () from /lib64/libc.so.6
>>> #2  0x00000000005858cf in xassert (msg=<optimized out>, file=<optimized out>, line=<optimized out>) at debug.cc:556
>>> #3  0x000000000050d054 in unlock (this=0x2059d40) at ../src/base/Lock.h:48
>>> #4  AccessLogEntry::~AccessLogEntry (this=0x205a2c0, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at AccessLogEntry.cc:82
>>> #5  0x000000000050d0f9 in AccessLogEntry::~AccessLogEntry (this=0x205a2c0, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at AccessLogEntry.cc:87
>>> #6  0x00000000006c1434 in dereference (newP=0x0, this=0x205a078) at ../../src/base/RefCount.h:97
>>> #7  ~RefCount (this=0x205a078, __in_chrg=<optimized out>) at ../../src/base/RefCount.h:35
>>> #8  ACLFilledChecklist::~ACLFilledChecklist (this=0x2059ec8, __in_chrg=<optimized out>) at FilledChecklist.cc:50
>>> #9  0x00000000006c15a9 in ACLFilledChecklist::~ACLFilledChecklist (this=0x2059ec8, __in_chrg=<optimized out>) at FilledChecklist.cc:67
>>> #10 0x00000000006f69ce in ACLChecklist::checkCallback (this=0x2059ec8, answer=...) at Checklist.cc:174
>>> #11 0x00000000005a483a in fqdncacheCallback (f=f at entry=0x205cb20, wait=wait at entry=10) at fqdncache.cc:316
>>> #12 0x00000000005a4c4a in fqdncacheHandleReply (data=<optimized out>, answers=<optimized out>, na=<optimized out>, error_message=<optimized out>) at fqdncache.cc:407
>>> #13 0x000000000058ad2d in idnsCallback (q=q at entry=0x205cbe8, error=error at entry=0x0) at dns_internal.cc:1144
>>> #14 0x000000000058fd66 in idnsGrokReply (buf=buf at entry=0xbc07e0 <idnsRead(int, void*)::rbuf> "\036<\201\200", sz=sz at entry=220) at dns_internal.cc:1313
>>> #15 0x0000000000590a4f in idnsRead (fd=20) at dns_internal.cc:1400
>>> #16 0x0000000000811c70 in Comm::DoSelect (msec=<optimized out>) at ModEpoll.cc:273
>>> #17 0x000000000075836e in CommSelectEngine::checkEvents (this=<optimized out>, timeout=<optimized out>) at comm.cc:1829
>>> #18 0x0000000000599b39 in EventLoop::checkEngine (this=this at entry=0x7fffffffe3b0, engine=engine at entry=0x7fffffffe330, primary=primary at entry=true) at EventLoop.cc:36
>>> #19 0x0000000000599d75 in EventLoop::runOnce (this=this at entry=0x7fffffffe3b0) at EventLoop.cc:115
>>> #20 0x0000000000599f80 in EventLoop::run (this=this at entry=0x7fffffffe3b0) at EventLoop.cc:83
>>> #21 0x0000000000606d71 in SquidMain (argc=<optimized out>, argv=<optimized out>) at main.cc:1638
>>> #22 0x00000000004ff12d in SquidMainSafe (argv=<optimized out>, argc=<optimized out>) at main.cc:1363
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 29/01/2016 12:46, "squid-dev on behalf of Amos Jeffries" <squid-dev-bounces at lists.squid-cache.org on behalf of squid3 at treenet.co.nz> wrote:
>>>
>>>> On 29/01/2016 8:10 a.m., Christos Tsantilas wrote:
>>>>> Hi all,
>>>>>
>>>>> After the patch r14351 created the following problems:
>>>>>   - external_acl requires AccessLogEntry but ALE is not available
>>>>>     in many cases such as ssl_bump ACLs.
>>>>>   - The %<cert_subject stopped working because it was supported by
>>>>>     external_acl code and not by logformat code.
>>>>>
>>>>> This patch:
>>>>>    - Passes AccessLogEntry in most cases.
>>>>>      For example, PeerConnector-related classes are now covered.
>>>>>    - Implements the %<cert_subject formating code for logformat.
>>>>>
>>>>>
>>>>> Still there are cases which are not handled correctly:
>>>>>    - In the case of transparent SSL bumping, the patch uses a local
>>>>> AccessLogEntry to allow external_acl work with the ssl_bump access list.
>>>>>
>>>>>   - The slow acls inside Ssl::PeerConnector can not support external_acl
>>>>> in the case of PeerPoolMgr
>>>>>
>>>>>    - Most of the fast acls does not support ALE based acls. I know that
>>>>> currently the only ALE based acl is the external_acl, which is slow acl,
>>>>> but my opinion is that it is not bad idea  to support cases the
>>>>> external_acl result is stored in cache.
>>>>>
>>>>>    - Also we need to check and review if the informations passed with the
>>>>> ALE is the same passed using the FilledChecklist object. This is not
>>>>> obvious.
>>>>>
>>>>>
>>>>> This is a Measurement Factory project.
>>>>>
>>>>
>>>> +1. Please apply.
>>>>
>>>> But note that PeerConnector child classes are now in different files
>>>> when merging to trunk.
>>>>
>>>> Amos