[squid-users] Ubuntu 20.04 "apt update" issues behind a VPN and Squid proxy
David Mills
david.mills at acusensus.com
Wed Jul 7 23:44:32 UTC 2021
Hi Eliezer,
We have:
/etc/apt/apt.conf:
> Acquire::http::proxy "
> http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/";
> Acquire::https::proxy "
> http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128/";
>
/etc/apt/sources.list (comment lines removed for brevity)
> deb https://mirror.aarnet.edu.au/ubuntu/ focal main restricted
> deb https://mirror.aarnet.edu.au/ubuntu/ focal-updates main restricted
> deb https://mirror.aarnet.edu.au/ubuntu/ focal-updates universe
> deb https://mirror.aarnet.edu.au/ubuntu/ focal multiverse
> deb https://mirror.aarnet.edu.au/ubuntu/ focal-updates multiverse
> deb https://mirror.aarnet.edu.au/ubuntu/ focal-backports main restricted
> universe multiverse
> deb https://mirror.aarnet.edu.au/ubuntu focal-security main restricted
> deb https://mirror.aarnet.edu.au/ubuntu focal-security universe
> deb https://mirror.aarnet.edu.au/ubuntu focal-security multiverse
>
squid.conf
> # Debugging for your ACLs
> debug_options ALL,1
>
> # temp option for full debug logs
> #debug_options 28,2
>
> # Example rule allowing access from your local networks.
> # Adapt to list your (internal) IP networks from where browsing
> # should be allowed
> acl vpc_cidr src 10.0.0.0/16 # VPC CIDR
> acl vpc_cidr src 127.0.0.1
>
> # technician VPN source cidr
> acl technician_vpn src 10.0.104.0/22
>
>
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> #acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> #acl Safe_ports port 70 # gopher
> #acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> #acl Safe_ports port 280 # http-mgmt
> #acl Safe_ports port 488 # gss-http
> #acl Safe_ports port 591 # filemaker
> #acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
>
> #
> # Recommended minimum Access Permission configuration:
> #
> # Deny requests to certain unsafe ports
> http_access deny !Safe_ports
>
> # Deny CONNECT to other than secure SSL ports
> http_access deny CONNECT !SSL_ports
>
> # Only allow cachemgr access from localhost
> http_access allow localhost manager
> http_access deny manager
>
> # We strongly recommend the following be uncommented to protect innocent
> # web applications running on the proxy server who think the only
> # one who can access services on "localhost" is a local user
> #http_access deny to_localhost
>
> #
> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
> #
>
> # Redirect HTTP to HTTPS
> acl port_80 port 80
> acl gstatic dstdomain www.gstatic.com
> http_access deny port_80 gstatic
> deny_info 301:https://%H%R gstatic
>
> acl avpc dstdomain crop-assessment.acusensus-vpc
> http_access deny port_80 avpc
> deny_info 302:<company url> avpc
>
>
> # Deny HTTP
> http_access deny port_80
>
> # Whitelist of allowed sites
> acl allowed_http_sites dstdomain "/etc/squid/squid.allowed.sites.txt"
> http_access allow allowed_http_sites vpc_cidr
>
> # And finally deny all other access to this proxy
> http_access deny all
>
> # Squid normally listens to port 3128
> http_port 3128 ssl-bump cert=/etc/squid/cert.pem
> acl allowed_https_sites ssl::server_name
> "/etc/squid/squid.allowed.sites.txt"
> acl step1 at_step SslBump1
> acl step2 at_step SslBump2
> acl step3 at_step SslBump3
> ssl_bump peek step1 all
> ssl_bump peek step2 allowed_https_sites
> ssl_bump splice step3 allowed_https_sites
> ssl_bump terminate step2 all
>
> # Uncomment and adjust the following to add a disk cache directory.
> #cache_dir ufs /var/spool/squid 100 16 256
>
> # Leave coredumps in the first cache dir
> coredump_dir /var/spool/squid
> #
> # Add any of your own refresh_pattern entries above these.
> #
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
>
>
>
Squid 3.5 is running on an EC2 instance running Amazon Linux 2. I'll answer
the questions you asked Ben for extra info.
ip address:
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
> default qlen 1000
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP
> group default qlen 1000
> link/ether 02:1b:15:b8:9a:06 brd ff:ff:ff:ff:ff:ff
> inet 10.0.12.111/24 brd 10.0.12.255 scope global dynamic eth0
> valid_lft 2393sec preferred_lft 2393sec
> inet6 fe80::1b:15ff:feb8:9a06/64 scope link
> valid_lft forever preferred_lft forever
>
ip rule
> 0: from all lookup local
> 32766: from all lookup main
> 32767: from all lookup default
>
ip route show
> default via 10.0.12.1 dev eth0
> 10.0.12.0/24 dev eth0 proto kernel scope link src 10.0.12.111
> 169.254.169.254 dev eth0
>
ip route show table 100
>
>
iptables-save
>
>
squid -v
> Squid Cache: Version 3.5.20
> Service Name: squid
> configure options: '--build=x86_64-koji-linux-gnu'
> '--host=x86_64-koji-linux-gnu' '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include'
> '--libdir=/usr/lib64' '--libexecdir=/usr/libexec'
> '--sharedstatedir=/var/lib' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--disable-strict-error-checking'
> '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var'
> '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
> '--with-logdir=$(localstatedir)/log/squid'
> '--with-pidfile=$(localstatedir)/run/squid.pid'
> '--disable-dependency-tracking' '--enable-eui'
> '--enable-follow-x-forwarded-for' '--enable-auth'
> '--enable-auth-basic=DB,LDAP,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,SMB_LM,getpwnam'
> '--enable-auth-ntlm=smb_lm,fake'
> '--enable-auth-digest=file,LDAP,eDirectory'
> '--enable-auth-negotiate=kerberos'
> '--enable-external-acl-helpers=file_userip,LDAP_group,time_quota,session,unix_group,wbinfo_group,kerberos_ldap_group'
> '--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
> '--enable-delay-pools' '--enable-epoll' '--enable-ident-lookups'
> '--enable-linux-netfilter' '--enable-removal-policies=heap,lru'
> '--enable-snmp' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,rock,ufs'
> '--enable-wccpv2' '--enable-esi' '--enable-ecap' '--with-aio'
> '--with-default-user=squid' '--with-dl' '--with-openssl' '--with-pthreads'
> '--disable-arch-native' 'build_alias=x86_64-koji-linux-gnu'
> 'host_alias=x86_64-koji-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
> --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
> -fpie' 'LDFLAGS=-Wl,-z,relro -pie -Wl,-z,relro -Wl,-z,now' 'CXXFLAGS=-O2
> -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
> -m64 -mtune=generic -fpie'
> 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
>
uname -a
> Linux ip-10-0-12-111.ap-southeast-2.compute.internal
> 4.14.231-173.361.amzn2.x86_64 #1 SMP Mon Apr 26 20:57:08 UTC 2021 x86_64
> x86_64 x86_64 GNU/Linux
>
Regards,
David Mills
Senior DevOps Engineer
E: david.mills at acusensus.com
M: +61 411 513 404
W: acusensus.com
On Wed, 7 Jul 2021 at 20:53, Eliezer Croitoru <ngtech1ltd at gmail.com> wrote:
> Hey David,
>
> Just wondering if you have seen the apt related docs at:
>
> https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy
>
> Eliezer
>
> From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf
> Of David Mills
> Sent: Wednesday, July 7, 2021 2:26 AM
> To: squid-users at lists.squid-cache.org
> Subject: [squid-users] Ubuntu 20.04 "apt update" issues behind a VPN and
> Squid proxy
>
> Hi,
>
> We've got a collection of Ubuntu 18.04 boxes out in the field. They
> connect to an AWS OpenVPN VPN and use a Squid 3.5 AWS hosted Proxy. They
> work fine.
>
> We have tried upgrading one to 20.04. Same setup. From the command line
> curl or wget can happily download an Ubuntu package from the Ubuntu Mirror
> site we use. But "apt update" gets lots of "IGN:" timeouts and errors.
>
> The package we test curl with is
> https://mirror.aarnet.edu.au/ubuntu/pool/main/c/curl/curl_7.68.0-1ubuntu2.5_amd64.deb
>
> The Squid log shows a line the doesn't occur for the successful 18.04 "apt
> updates":
> 1625190959.233 81 10.0.11.191 TAG_NONE/200 0 CONNECT
> http://mirror.aarnet.edu.au:443 - HIER_DIRECT/2001:388:30bc:cafe::beef -
>
> The full output of an attempt to update is:
> Ign:1 https://mirror.aarnet.edu.au/ubuntu focal InRelease
>
> Ign:2 https://mirror.aarnet.edu.au/ubuntu focal-updates InRelease
>
> Ign:3 https://mirror.aarnet.edu.au/ubuntu focal-backports InRelease
>
> Ign:4 https://mirror.aarnet.edu.au/ubuntu focal-security InRelease
>
> Err:5 https://mirror.aarnet.edu.au/ubuntu focal Release
>
> Could not wait for server fd - select (11: Resource temporarily
> unavailable) [IP: 10.0.11.82 3128]
> Err:6 https://mirror.aarnet.edu.au/ubuntu focal-updates Release
>
> Could not wait for server fd - select (11: Resource temporarily
> unavailable) [IP: 10.0.11.82 3128]
> Err:7 https://mirror.aarnet.edu.au/ubuntu focal-backports Release
>
> Could not wait for server fd - select (11: Resource temporarily
> unavailable) [IP: 10.0.11.82 3128]
> Err:8 https://mirror.aarnet.edu.au/ubuntu focal-security Release
>
> Could not wait for server fd - select (11: Resource temporarily
> unavailable) [IP: 10.0.1.26 3128]
> Reading package lists... Done
>
> N: Ignoring file 'microsoft-prod.list-keep' in directory
> '/etc/apt/sources.list.d/' as it has an invalid filename extension
> E: The repository 'https://mirror.aarnet.edu.au/ubuntu focal Release'
> does not have a Release file.
> N: Updating from such a repository can't be done securely, and is
> therefore disabled by default.
> N: See apt-secure(8) manpage for repository creation and user
> configuration details.
> E: The repository 'https://mirror.aarnet.edu.au/ubuntu focal-updates
> Release' does not have a Release file.
> N: Updating from such a repository can't be done securely, and is
> therefore disabled by default.
> N: See apt-secure(8) manpage for repository creation and user
> configuration details.
> E: The repository 'https://mirror.aarnet.edu.au/ubuntu focal-backports
> Release' does not have a Release file.
> N: Updating from such a repository can't be done securely, and is
> therefore disabled by default.
> N: See apt-secure(8) manpage for repository creation and user
> configuration details.
> E: The repository 'https://mirror.aarnet.edu.au/ubuntu focal-security
> Release' does not have a Release file.
> N: Updating from such a repository can't be done securely, and is
> therefore disabled by default.
> N: See apt-secure(8) manpage for repository creation and user
> configuration details.
>
> While running, the line
> 0% [Connecting to HTTP proxy (
> http://vpn-proxy-d68aca8a8f7f81d6.elb.ap-southeast-2.amazonaws.com:3128)]
> appears often and hang for a while.
>
> I've tried upping the squid logging and allowing all, but they didn't
> offer any additional information about the issue.
>
> Any advice would be greatly appreciated.
>
> Regards,
>
>
> David Mills
> Senior DevOps Engineer
>
> E: mailto:david.mills at acusensus.com
> M: +61 411 513 404
> W:http://acusensus.com/
>
>
>
> DISCLAIMER: Acusensus puts the privacy and security of its clients, its
> data and information at the core of everything we do. The information
> contained in this email (including attachments) is intended only for the
> use of the person(s) to whom it is addressed to, as it may be confidential
> and contain legally privileged information. If you have received this email
> in error, please delete all copies and notify the sender immediately. Any
> views or opinions presented are solely those of the author and do not
> necessarily represent the views of Acusensus Pty Ltd. Please consider the
> environment before printing this email.
>
>
--
DISCLAIMER: Acusensus puts the privacy and security of its clients, its
data and information at the core of everything we do. The information
contained in this email (including attachments) is intended only for the
use of the person(s) to whom it is addressed to, as it may be confidential
and contain legally privileged information. If you have received this email
in error, please delete all copies and notify the sender immediately. Any
views or opinions presented are
solely those of the author and do not
necessarily represent the views of Acusensus
Pty Ltd. Please consider the
environment
before printing this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20210708/fcd68076/attachment-0001.htm>
More information about the squid-users
mailing list