<div dir="ltr"><div dir="ltr"><div>Hi Amos,</div><div><br></div><div>Thanks for the info.</div><div><br></div><div>Yes, "<a href="http://mirror.aarnet.edu.au" rel="noreferrer" target="_blank">mirror.aarnet.edu.au</a>" is in the whitelist. IPv6 could be an issue as I believe AWS ELBs may not support.</div><div><br></div><div>We'll try the logging you suggest and perhaps an upgrade to 4.0 if we have no joy with 3.5.<br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,</div><div><br></div><div><div dir="ltr" style="margin-left:0pt" align="left"><table style="border:none;border-collapse:collapse"><colgroup><col width="181"><col width="229"></colgroup><tbody><tr style="height:0pt"><td style="border-left:solid #ffffff 1pt;border-right:solid #4a86e8 1pt;border-bottom:solid #ffffff 1pt;border-top:solid #ffffff 1pt;vertical-align:top;padding:5pt 5pt 5pt 5pt;overflow:hidden"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"><span style="border:none;display:inline-block;overflow:hidden;width:167px;height:130px"><img src="https://lh4.googleusercontent.com/PjzOtuo8wRwonOYtyjVEsTHphPFdwgM8H79UkQ5H--uLWS_Wje0pIvRFGgiiaYF8CohhfacA5LpIBIck7fEou91YR_e95GyEd53ubLzjKbgTVaqvhdESRKKiWZqDxUYAmOApJr47" style="margin-left:0px;margin-top:0px" width="167" height="130"></span></span></p></td><td style="border-left:solid #4a86e8 1pt;border-right:solid #ffffff 1pt;border-bottom:solid #ffffff 1pt;border-top:solid #ffffff 1pt;vertical-align:top;padding:5pt 5pt 5pt 5pt;overflow:hidden"><p dir="ltr" style="line-height:1.728;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11.5pt;font-family:Arial;color:#2e80b5;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">David Mills</span></p><p dir="ltr" style="line-height:1.728;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Senior DevOps Engineer</span></p><br><p dir="ltr" style="line-height:1.44;margin-right:5pt;margin-top:0pt;margin-bottom:5pt"><span style="font-size:9pt;font-family:Arial;color:#3388cc;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> </span><span style="font-size:9pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">E: </span><span style="font-size:9pt;font-family:Arial;color:#2e80b5;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"><a href="mailto:david.mills@acusensus.com" target="_blank">david.mills@acusensus.com</a></span></p><p dir="ltr" style="line-height:1.44;margin-right:5pt;margin-top:0pt;margin-bottom:5pt"><span style="font-size:9pt;font-family:Arial;color:#3388cc;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> </span><span style="font-size:9pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">M:</span><span style="font-size:9pt;font-family:Arial;color:#3388cc;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> </span><span style="font-size:9pt;font-family:Arial;color:#2e80b5;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">+61 411 513 404</span></p><p dir="ltr" style="line-height:1.44;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> W:</span><a href="http://acusensus.com/" style="text-decoration:none" target="_blank"><span style="font-size:9pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre;white-space:pre-wrap"> </span><span style="font-size:9pt;font-family:Arial;color:#1155cc;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre;white-space:pre-wrap">acusensus.com</span></a></p><br></td></tr></tbody></table></div><br><br></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 8 Jul 2021 at 12:19, Amos Jeffries <<a href="mailto:squid3@treenet.co.nz">squid3@treenet.co.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
On 8/07/21 11:44 am, David Mills wrote:<br>
> Hi Eliezer,<br>
> <br>
> We have:<br>
> <br>
> /etc/apt/apt.conf:<br>
> <br>
> Acquire::http::proxy<br>
> "<a href="http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/" rel="noreferrer" target="_blank">http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/</a><br>
> <<a href="http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/" rel="noreferrer" target="_blank">http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/</a>>";<br>
> Acquire::https::proxy<br>
> "<a href="http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/" rel="noreferrer" target="_blank">http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/</a><br>
> <<a href="http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/" rel="noreferrer" target="_blank">http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/</a>>";<br>
> <br>
> <br>
> /etc/apt/sources.list (comment lines removed for brevity)<br>
> <br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a>> focal main restricted<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a>> focal-updates main restricted<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a>> focal-updates universe<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a>> focal multiverse<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a>> focal-updates multiverse<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/</a>> focal-backports main<br>
> restricted universe multiverse<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal-security main restricted<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal-security universe<br>
> deb <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal-security multiverse<br>
> <br>
> <br>
> squid.conf<br>
> <br>
...<br>
> #<br>
> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS<br>
> #<br>
> <br>
> # Redirect HTTP to HTTPS<br>
> acl port_80 port 80<br>
> acl gstatic dstdomain <a href="http://www.gstatic.com" rel="noreferrer" target="_blank">www.gstatic.com</a> <<a href="http://www.gstatic.com" rel="noreferrer" target="_blank">http://www.gstatic.com</a>><br>
> http_access deny port_80 gstatic<br>
> deny_info 301:https://%H%R gstatic<br>
> <br>
> acl avpc dstdomain crop-assessment.acusensus-vpc<br>
> http_access deny port_80 avpc<br>
> deny_info 302:<company url> avpc<br>
> <br>
> <br>
> # Deny HTTP<br>
> http_access deny port_80<br>
> <br>
> # Whitelist of allowed sites<br>
> acl allowed_http_sites dstdomain "/etc/squid/squid.allowed.sites.txt"<br>
> http_access allow allowed_http_sites vpc_cidr<br>
> <br>
<br>
Is the "<a href="http://mirror.aarnet.edu.au" rel="noreferrer" target="_blank">mirror.aarnet.edu.au</a>" or a wildcard matching it listed in file <br>
squid.allowed.sites.txt ?<br>
<br>
(I assume so, but checking in case it is that simple).<br>
<br>
<br>
> # And finally deny all other access to this proxy<br>
> http_access deny all<br>
> <br>
> # Squid normally listens to port 3128<br>
> http_port 3128 ssl-bump cert=/etc/squid/cert.pem<br>
> acl allowed_https_sites ssl::server_name<br>
> "/etc/squid/squid.allowed.sites.txt"<br>
> acl step1 at_step SslBump1<br>
> acl step2 at_step SslBump2<br>
> acl step3 at_step SslBump3<br>
> ssl_bump peek step1 all<br>
> ssl_bump peek step2 allowed_https_sites<br>
> ssl_bump splice step3 allowed_https_sites<br>
> ssl_bump terminate step2 all<br>
> <br>
> # Uncomment and adjust the following to add a disk cache directory.<br>
> #cache_dir ufs /var/spool/squid 100 16 256<br>
> <br>
> # Leave coredumps in the first cache dir<br>
> coredump_dir /var/spool/squid<br>
> #<br>
> # Add any of your own refresh_pattern entries above these.<br>
> #<br>
> refresh_pattern ^ftp: 1440 20% 10080<br>
> refresh_pattern ^gopher: 1440 0% 1440<br>
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0<br>
> refresh_pattern . 0 20% 4320<br>
> <br>
> <br>
> <br>
> Squid 3.5 is running on an EC2 instance running Amazon Linux 2. I'll <br>
> answer the questions you asked Ben for extra info.<br>
> ip address:<br>
> <br>
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN<br>
> group default qlen 1000<br>
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>
> inet <a href="http://127.0.0.1/8" rel="noreferrer" target="_blank">127.0.0.1/8</a> <<a href="http://127.0.0.1/8" rel="noreferrer" target="_blank">http://127.0.0.1/8</a>> scope host lo<br>
> valid_lft forever preferred_lft forever<br>
> inet6 ::1/128 scope host<br>
> valid_lft forever preferred_lft forever<br>
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state<br>
> UP group default qlen 1000<br>
> link/ether 02:1b:15:b8:9a:06 brd ff:ff:ff:ff:ff:ff<br>
> inet <a href="http://10.0.12.111/24" rel="noreferrer" target="_blank">10.0.12.111/24</a> <<a href="http://10.0.12.111/24" rel="noreferrer" target="_blank">http://10.0.12.111/24</a>> brd 10.0.12.255<br>
> scope global dynamic eth0<br>
> valid_lft 2393sec preferred_lft 2393sec<br>
> inet6 fe80::1b:15ff:feb8:9a06/64 scope link<br>
> valid_lft forever preferred_lft forever<br>
> <br>
> <br>
> ip rule<br>
> <br>
> 0: from all lookup local<br>
> 32766: from all lookup main<br>
> 32767: from all lookup default <br>
> <br>
> <br>
> ip route show<br>
> <br>
> default via 10.0.12.1 dev eth0<br>
> <a href="http://10.0.12.0/24" rel="noreferrer" target="_blank">10.0.12.0/24</a> <<a href="http://10.0.12.0/24" rel="noreferrer" target="_blank">http://10.0.12.0/24</a>> dev eth0 proto kernel scope link<br>
> src 10.0.12.111<br>
> 169.254.169.254 dev eth0<br>
> <br>
> <br>
<br>
<br>
The traffic from Squid to the AArnet server is apparently using IPv6. Is <br>
that routing setup properly too?<br>
<br>
<br>
...<br>
<br>
> From: squid-users On Behalf Of David Mills<br>
> Sent: Wednesday, July 7, 2021 2:26 AM<br>
...<br>
> We have tried upgrading one to 20.04. Same setup. From the command<br>
> line curl or wget can happily download an Ubuntu package from the<br>
> Ubuntu Mirror site we use. But "apt update" gets lots of "IGN:"<br>
> timeouts and errors.<br>
> <br>
> The package we test curl with is<br>
> <a href="https://mirror.aarnet.edu.au/ubuntu/pool/main/c/curl/curl_7.68.0-1ubuntu2.5_amd64.deb" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/pool/main/c/curl/curl_7.68.0-1ubuntu2.5_amd64.deb</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu/pool/main/c/curl/curl_7.68.0-1ubuntu2.5_amd64.deb" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu/pool/main/c/curl/curl_7.68.0-1ubuntu2.5_amd64.deb</a>><br>
> <br>
> The Squid log shows a line the doesn't occur for the successful<br>
> 18.04 "apt updates":<br>
> 1625190959.233 81 10.0.11.191 TAG_NONE/200 0 CONNECT<br>
> <a href="http://mirror.aarnet.edu.au:443" rel="noreferrer" target="_blank">http://mirror.aarnet.edu.au:443</a> <<a href="http://mirror.aarnet.edu.au:443" rel="noreferrer" target="_blank">http://mirror.aarnet.edu.au:443</a>> -<br>
> HIER_DIRECT/2001:388:30bc:cafe::beef -<br>
> <br>
<br>
With Ubuntu 20.04 you should have received Squid-4 (v4.10 or later). <br>
Which logs a few things differently from Squid-3.5, particularly for <br>
SSL-Bump activity and client connections that lack HTTP messages.<br>
<br>
The above log line shows SSL-Bump activity. At least step2 was reached, <br>
possibly also step3. Looking at this a little closer to see if it <br>
completes fine or has unseen issues would be my next point of approach.<br>
<br>
To debug what is happening with SSL-Bump use "debug_options ALL1, 11,2 <br>
83,5" and check the resulting cache.log.<br>
<br>
<br>
<br>
> The full output of an attempt to update is:<br>
> Ign:1 <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal InRelease<br>
> Ign:2 <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal-updates InRelease<br>
> Ign:3 <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal-backports InRelease<br>
> Ign:4 <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal-security InRelease<br>
<br>
<br>
These "Ign" are fine. They just mean that apt has determined those files <br>
it has cached are up-to-date and do not need to be re-fetched right now.<br>
<br>
The below "Err" are the problem:<br>
<br>
> Err:5 <a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a><br>
> <<a href="https://mirror.aarnet.edu.au/ubuntu" rel="noreferrer" target="_blank">https://mirror.aarnet.edu.au/ubuntu</a>> focal Release<br>
> Could not wait for server fd - select (11: Resource temporarily<br>
> unavailable) [IP: 10.0.11.82 3128]...<br>
> <br>
> While running, the line<br>
> 0% [Connecting to HTTP proxy<br>
> (<a href="http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128" rel="noreferrer" target="_blank">http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128</a><br>
> <<a href="http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128" rel="noreferrer" target="_blank">http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128</a>>)]<br>
> appears often and hang for a while.<br>
> <br>
> I've tried upping the squid logging and allowing all, but they<br>
> didn't offer any additional information about the issue.<br>
> <br>
<br>
Your squid.conf looks fine, assuming the same http_access rules were <br>
used in your working version.<br>
<br>
<br>
I suspect the issue is related to one or more of:<br>
<br>
* IPv6 routing<br>
<br>
* ICMP config issues (maybe outside your network)<br>
<br>
* SSL-Bump issues processing the client or server handshake traffic<br>
typically seen with OpenSSL library version or config mismatches <br>
between Squid, client and server.<br>
<br>
* network timeouts affecting Squid<br>
<br>
<br>
HTH<br>
Amos<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div></div>
<br>
<font size="2">DISCLAIMER: Acusensus puts the privacy and security of its clients, its data and information at the core of everything we do. The information contained in this email (including attachments) is intended only for the use of the person(s) to whom it is addressed to, as it may be confidential and contain legally privileged information. If you have received this email in error, please delete all copies and notify the sender immediately. Any views or opinions presented are
solely those of the author and do not necessarily represent the views of Acusensus
Pty Ltd. Please consider the environment
before printing this email.</font>