<html><body><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000"><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000"><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">I appreciate very much your looking at my question.<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">>Date: Thu, 30 May 2024 22:15:27 +1200<br>>From: Amos Jeffries <squid3@treenet.co.nz><br>>To: squid-users@lists.squid-cache.org<br>>Subject: Re: [squid-users] can't explain 403 denied for authenticated<br>> user<br>>Message-ID: <96746157-da51-47db-8d52-d65239f2717e@treenet.co.nz><br>>Content-Type: text/plain; charset=UTF-8; format=flowed<br>><br>>On 25/05/24 07:28, Kevin wrote:<br>>> Hi,<br>>><br>>> We have 2 external ACLs that take a request's data (IP, authenticated<br>>> username, URL, user-agent, etc) and uses that information to determine<br>>> whether a user or host should be permitted to access that URL.?? It<br>>> almost always works well, but we have a recurring occasional issue that<br>>> I can't figure out.<br>>><br>>> When it occurs, it's always around 4AM.?? This particular request occurs<br>>> often - averages about once a second throughout the day.<br>>><br>>> What we see is a "403 forbidden" for a (should be) permitted site from<br>>> an authenticated user from the same IP/user and to the same site that<br>>> gets a "202 connection established" every other time.<br>><br>>Is maybe 4am the time when your auth system refreshes all nonce?<br>> - thus making any currently in-use by the clients invalid until they<br>>re-auth. You might see a mix of 403/401/407 in a bunch at such times.<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">My auth system is just squid with one flat file each for basic and digest authentication (we allow basic only if an application doesn't support digest).<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>>Or maybe in a similar style one/some of the clients is broken and fails<br>>to update its nonce before it expires at 4am?<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">The client is a Java-based application. I can find out more from the team that wrote it.<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">> - looking at which client agent and IP were getting the 403 and/or the<br>>nonce which received 403 will give you hints about this possibility.<br><br></div><div data-marker="__QUOTED_TEXT__">Oh: it's only this one application from this one particular host. <br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>><br>>Or your network router(s) do garbage collection and terminate<br>>long-running connections to free up TCP resources?<br>> - thus forcing a lot of client re-connects at 4am, which may:<br>> a) overload the auth system/helper, or<br>> b) break a transaction that included nonce update for clients -<br>>resulting in their next request being invalid nonce.<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">I'm not aware of a 4AM connection-terminating task but will confirm. As mentioned (in answer to a), the auth system is just squid. Re: b - I hadn't thought of that. <br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">><br>><br>>Or maybe you have log processing software that does "squid -k restart"<br>>instead of the proper "squid -k rotate" to get access to the log files?<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">I hadn't checked that! But logrotate runs at 00:00 and the rotate script does use "squid -k rotate".<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>>Or maybe your auth system has a limit on how large nonce-count can become?<br>> - I notice that the working request has 0x2F uses and the forbidden<br>>has 0x35 (suspiciously close to 50 in decimal)</div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div>I don't know what that is. I didn't find a lot of helpful detail on squid's digest auth implementation. <br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">><br>>><br>>> The difference I see in the logs:? though all the digest auth info looks<br>>> okay, the %un field in the log for the usual (successful) request is the<br>>> authenticated username, while in the failed request, it's "-".?? So<br>>> though there hasn't been an authentication error or "authentication<br>>> required" in the log - and the username is in the authentication details<br>>> in the log entry -? it seems like squid isn't recognizing that username<br>>> as %un.<br>><br>><br>>Be aware that a properly behaving client will *not* send credentials<br>>until they are requested, after which it should *always* send<br>>credentials on that same connection (even if they are not requested<br>>explicitly).</div><div data-marker="__QUOTED_TEXT__"><br>>That means some requests on a multiplex/pipeline/keep-alive connection<br>>MAY arrive with credentials and be accepted(2xx)/denied(403) without<br>>authentication having occured. Entirely due to your *_access directives<br>>sequence. In these cases the log will show auth headers but no value for<br>>%un and/or %ul.<br>><br>><br>>><br>>> My squid.conf first tests a request to see if an unauthenticated request<br>>> from a particular host is permitted.? That external ACL doesn't take a<br>>> username as an argument.?? If that external ACL passes, the request is<br>>> allowed.<br>>><br>><br>>Please *show* the config lines rather than describing what you *think*<br>>they do. Their exact ordering matters *a lot*.<br></div><div data-marker="__QUOTED_TEXT__">> Obfuscation of sensitive details is okay/expected so long as you makes<br>>it easy for us to tell that value A and value B are different.<br></div><div data-marker="__QUOTED_TEXT__">><br>>FWIW; if your config file is *actually* containing only what you<br>>described it is missing a number of things necessary to have a safe and<br>>secure proxy. A look at your full config (without comments or empty<br>>lines) will help us point out any unnoticed issues for you to consider<br>>fixing.<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">Understood. Here it is:<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"> <!--StartFragment--><pre>acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl windows_net src 172.18.114.0/24
acl sample_host src 172.18.115.1/32
acl rsync port 873
acl SSL_ports port 443
acl SSL_ports port 873 #873 is rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 873
acl CONNECT method CONNECT
acl PURGE method PURGE
acl localhost src 127.0.0.1
http_access allow PURGE localhost
http_access deny PURGE
acl URN proto URN
http_access deny URN
http_access deny manager
acl API_FIREFOX dstdomain api.profiler.firefox.com
http_access deny API_FIREFOX
acl ff_browser browser ^Mozilla/5\.0
acl rma_ua browser ^RMA/1\.0.*compatible;.RealMedia
uri_whitespace encode
acl trellix_phone_cloud dstdomain amcore-ens.rest.gti.trellix.com
http_access deny trellix_phone_cloud
external_acl_type host_based_filter children-max=15 ttl=0 %ACL %DATA %SRC %>rd %>rP /PATH/TO/FILTER/SCRIPT.py
acl HostBasedRules external host_based_filter
http_access allow HostBasedRules
auth_param digest program /usr/lib/squid/digest_file_auth -c /etc/squid/passwd
auth_param digest realm squid
auth_param digest children 2
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/basic_passwd
auth_param basic children 2
auth_param basic realm squidb
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
external_acl_type custom_acl_db children-max=15 ttl=0 %ACL %DATA %ul %SRC %>rd %>rP %credentials /PATH/TO/FILTER/SCRIPT.py
acl CustomAclDB external custom_acl_db
http_access allow CustomAclDB
acl CRLs url_regex "/etc/squid/conf.d/CRL_urls.txt"
http_access allow CRLs
deny_info 303:https://abc.def.com/
http_access deny all
acl apache rep_header Server ^Apache
icp_access allow localnet
icp_access deny all
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_port 3128
coredump_dir /var/cache/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
logformat squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt
logformat squidmime %ul %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt [%>h] [%<h]
logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
logformat ip_port %ts %tu %>a %>p %<lp %<a %<p %dt %tr %un %>ru %Ss
access_log daemon:/var/log/squid/access.log combined
access_log daemon:/var/log/squid/network.log ip_port
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
access_log daemon:/var/log/squid/useragent.log useragent
visible_hostname proxy.abc.com
cache deny all</pre><!--EndFragment--> </div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>><br>>> The next line in squid.conf is<br>>><br>>> acl auth_users proxy_auth REQUIRED<br>>><br>><br>>FYI the above just means that Squid is using authentication. It says<br>>nothing about when the authentication will be (or not be) performed.<br>><br>><br>>> ... and after that, the external ACL that takes the username as well as<br>>> the other info.<br>>><br>><br>><br>>HTH<br>>Amos</div></div><div><br><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>squid-users-request@lists.squid-cache.org<br><b>To: </b>"squid-users" <squid-users@lists.squid-cache.org><br><b>Sent: </b>Thursday, May 30, 2024 8:00:01 AM<br><b>Subject: </b>squid-users Digest, Vol 117, Issue 31<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">I appreciate very much your looking at my question.<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">>Date: Thu, 30 May 2024 22:15:27 +1200<br>>From: Amos Jeffries <squid3@treenet.co.nz><br>>To: squid-users@lists.squid-cache.org<br>>Subject: Re: [squid-users] can't explain 403 denied for authenticated<br>> user<br>>Message-ID: <96746157-da51-47db-8d52-d65239f2717e@treenet.co.nz><br>>Content-Type: text/plain; charset=UTF-8; format=flowed<br>><br>>On 25/05/24 07:28, Kevin wrote:<br>>> Hi,<br>>><br>>> We have 2 external ACLs that take a request's data (IP, authenticated<br>>> username, URL, user-agent, etc) and uses that information to determine<br>>> whether a user or host should be permitted to access that URL.?? It<br>>> almost always works well, but we have a recurring occasional issue that<br>>> I can't figure out.<br>>><br>>> When it occurs, it's always around 4AM.?? This particular request occurs<br>>> often - averages about once a second throughout the day.<br>>><br>>> What we see is a "403 forbidden" for a (should be) permitted site from<br>>> an authenticated user from the same IP/user and to the same site that<br>>> gets a "202 connection established" every other time.<br>><br>>Is maybe 4am the time when your auth system refreshes all nonce?<br>> - thus making any currently in-use by the clients invalid until they<br>>re-auth. You might see a mix of 403/401/407 in a bunch at such times.<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">My auth system is just squid with one flat file each for basic and digest authentication (we allow basic only if an application doesn't support digest).<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>>Or maybe in a similar style one/some of the clients is broken and fails<br>>to update its nonce before it expires at 4am?<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">The client is a Java-based application. I can find out more from the team that wrote it.<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">> - looking at which client agent and IP were getting the 403 and/or the<br>>nonce which received 403 will give you hints about this possibility.<br><br></div><div data-marker="__QUOTED_TEXT__">Oh: it's only this one application from this one particular host. <br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>><br>>Or your network router(s) do garbage collection and terminate<br>>long-running connections to free up TCP resources?<br>> - thus forcing a lot of client re-connects at 4am, which may:<br>> a) overload the auth system/helper, or<br>> b) break a transaction that included nonce update for clients -<br>>resulting in their next request being invalid nonce.<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">I'm not aware of a 4AM connection-terminating task but will confirm. As mentioned (in answer to a), the auth system is just squid. Re: b - I hadn't thought of that. <br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">><br>><br>>Or maybe you have log processing software that does "squid -k restart"<br>>instead of the proper "squid -k rotate" to get access to the log files?<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">I hadn't checked that! But logrotate runs at 00:00 and the rotate script does use "squid -k rotate".<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>>Or maybe your auth system has a limit on how large nonce-count can become?<br>> - I notice that the working request has 0x2F uses and the forbidden<br>>has 0x35 (suspiciously close to 50 in decimal)</div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div>I don't know what that is. I didn't find a lot of helpful detail on squid's digest auth implementation. <br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">><br>>><br>>> The difference I see in the logs:? though all the digest auth info looks<br>>> okay, the %un field in the log for the usual (successful) request is the<br>>> authenticated username, while in the failed request, it's "-".?? So<br>>> though there hasn't been an authentication error or "authentication<br>>> required" in the log - and the username is in the authentication details<br>>> in the log entry -? it seems like squid isn't recognizing that username<br>>> as %un.<br>><br>><br>>Be aware that a properly behaving client will *not* send credentials<br>>until they are requested, after which it should *always* send<br>>credentials on that same connection (even if they are not requested<br>>explicitly).</div><div data-marker="__QUOTED_TEXT__"><br>>That means some requests on a multiplex/pipeline/keep-alive connection<br>>MAY arrive with credentials and be accepted(2xx)/denied(403) without<br>>authentication having occured. Entirely due to your *_access directives<br>>sequence. In these cases the log will show auth headers but no value for<br>>%un and/or %ul.<br>><br>><br>>><br>>> My squid.conf first tests a request to see if an unauthenticated request<br>>> from a particular host is permitted.? That external ACL doesn't take a<br>>> username as an argument.?? If that external ACL passes, the request is<br>>> allowed.<br>>><br>><br>>Please *show* the config lines rather than describing what you *think*<br>>they do. Their exact ordering matters *a lot*.<br></div><div data-marker="__QUOTED_TEXT__">> Obfuscation of sensitive details is okay/expected so long as you makes<br>>it easy for us to tell that value A and value B are different.<br></div><div data-marker="__QUOTED_TEXT__">><br>>FWIW; if your config file is *actually* containing only what you<br>>described it is missing a number of things necessary to have a safe and<br>>secure proxy. A look at your full config (without comments or empty<br>>lines) will help us point out any unnoticed issues for you to consider<br>>fixing.<br></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__">Understood. Here it is:<br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"> <!--StartFragment--><pre>acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl windows_net src 172.18.114.0/24
acl sample_host src 172.18.115.1/32
acl rsync port 873
acl SSL_ports port 443
acl SSL_ports port 873 #873 is rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 873
acl CONNECT method CONNECT
acl PURGE method PURGE
acl localhost src 127.0.0.1
http_access allow PURGE localhost
http_access deny PURGE
acl URN proto URN
http_access deny URN
http_access deny manager
acl API_FIREFOX dstdomain api.profiler.firefox.com
http_access deny API_FIREFOX
acl ff_browser browser ^Mozilla/5\.0
acl rma_ua browser ^RMA/1\.0.*compatible;.RealMedia
uri_whitespace encode
acl trellix_phone_cloud dstdomain amcore-ens.rest.gti.trellix.com
http_access deny trellix_phone_cloud
external_acl_type host_based_filter children-max=15 ttl=0 %ACL %DATA %SRC %>rd %>rP /PATH/TO/FILTER/SCRIPT.py
acl HostBasedRules external host_based_filter
http_access allow HostBasedRules
auth_param digest program /usr/lib/squid/digest_file_auth -c /etc/squid/passwd
auth_param digest realm squid
auth_param digest children 2
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/basic_passwd
auth_param basic children 2
auth_param basic realm squidb
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
external_acl_type custom_acl_db children-max=15 ttl=0 %ACL %DATA %ul %SRC %>rd %>rP %credentials /PATH/TO/FILTER/SCRIPT.py
acl CustomAclDB external custom_acl_db
http_access allow CustomAclDB
acl CRLs url_regex "/etc/squid/conf.d/CRL_urls.txt"
http_access allow CRLs
deny_info 303:https://abc.def.com/
http_access deny all
acl apache rep_header Server ^Apache
icp_access allow localnet
icp_access deny all
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_port 3128
coredump_dir /var/cache/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
logformat squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt
logformat squidmime %ul %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt [%>h] [%<h]
logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
logformat ip_port %ts %tu %>a %>p %<lp %<a %<p %dt %tr %un %>ru %Ss
access_log daemon:/var/log/squid/access.log combined
access_log daemon:/var/log/squid/network.log ip_port
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
access_log daemon:/var/log/squid/useragent.log useragent
visible_hostname proxy.abc.com
cache deny all</pre><!--EndFragment--> </div><div data-marker="__QUOTED_TEXT__"><br data-mce-bogus="1"></div><div data-marker="__QUOTED_TEXT__"><br>><br>>> The next line in squid.conf is<br>>><br>>> acl auth_users proxy_auth REQUIRED<br>>><br>><br>>FYI the above just means that Squid is using authentication. It says<br>>nothing about when the authentication will be (or not be) performed.<br>><br>><br>>> ... and after that, the external ACL that takes the username as well as<br>>> the other info.<br>>><br>><br>><br>>HTH<br>>Amos</div><div><br></div></div></body></html>