[squid-users] Squid Cache 6.9 on Ubuntu 22.04.3 LTS. Not caching large files to disk.

Fri Apr 12 16:30:46 UTC 2024

Hi Jonathan,

No, I didn't have a refresh_pattern for .ISO/etc, so thank you. BTW, what are the "43800 100% 129600" values?

I realised that I had not actually configured "SSL Bump" in that last /etc/squid/squid.conf file I posted, as the access.log showed my https connections as being tunnelled. 🙁

I have tried to enable SSL Bump as best I understand how to and my squid.conf now looks like:

acl localnet src 0.0.0.1-0.255.255.255  # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8             # RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10          # RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16         # RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12          # RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16         # RFC 1918 local private network (LAN)
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_access allow localnet
http_access deny to_localhost
http_access deny to_linklocal
include /etc/squid/conf.d/*.conf
http_access deny all
http_port 3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myCA.pem
sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB
ssl_bump peek all
ssl_bump splice all
coredump_dir /var/spool/squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$)     43800 100% 129600
shutdown_lifetime 10 seconds
maximum_object_size 35 GB
cache_mem 256 MB
maximum_object_size_in_memory 512 KB
cache_replacement_policy heap LFUDA
range_offset_limit -1
quick_abort_min -1 KB
cache_dir aufs /var/spool/squid 150000 16 256 min-size=1048576

I read in one blog that the cache_dir had to be listed after maximum_object_size so I moved it.

I also reduced the cache_dir / min-size value from 1 GB to 1 MB for testing and switched to a smaller .ISO file as I was getting bored wating for the big one to download repeatedly.

So now:

1) A https download works, but is still tunnelled as mentioned above:

root at client1 [ /tmp ]# wget -e https_proxy=10.40.1.250:3128 --ca-certificate ~/myCA.pem https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso
--2024-04-12 15:42:44--  https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso
Connecting to 10.40.1.250:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1016070144 (969M) [application/x-iso9660-image]
Saving to: ‘ubuntu-18.04.6-live-server-amd64.iso’

ubuntu-18.04.6-live-server-amd64.iso            100%[=======================================================================================================>] 969.00M  20.0MB/s    in 53s

2024-04-12 15:43:37 (18.4 MB/s) - ‘ubuntu-18.04.6-live-server-amd64.iso’ saved [1016070144/1016070144]

and the access.log entry looks like this:

1712936617.285  52629 10.40.1.2 TCP_TUNNEL/200 1017438604 CONNECT releases.ubuntu.com:443 - HIER_DIRECT/185.125.190.40 -

2) A new http download works and is cached to disk now:

root at client1 [ /tmp ]# wget -e http_proxy=10.40.1.250:3128 http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso
--2024-04-12 15:44:15--  http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso
Connecting to 10.40.1.250:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1016070144 (969M) [application/x-iso9660-image]
Saving to: ‘ubuntu-18.04.6-live-server-amd64.iso.1’

ubuntu-18.04.6-live-server-amd64.iso.1          100%[=======================================================================================================>] 969.00M  16.0MB/s    in 52s

2024-04-12 15:45:07 (18.6 MB/s) - ‘ubuntu-18.04.6-live-server-amd64.iso.1’ saved [1016070144/1016070144]

and the access.log entry looks like this:

1712936707.689  52198 10.40.1.2 TCP_MISS/200 1016070508 GET http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso - HIER_DIRECT/185.125.190.40 application/x-iso9660-image

3) A subsequent http download of the same file does pull it from cache:

root at client1 [ /tmp ]# wget -e http_proxy=10.40.1.250:3128 http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso
--2024-04-12 15:45:23--  http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso
Connecting to 10.40.1.250:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1016070144 (969M) [application/x-iso9660-image]
Saving to: ‘ubuntu-18.04.6-live-server-amd64.iso.2’

ubuntu-18.04.6-live-server-amd64.iso.2          100%[=======================================================================================================>] 969.00M  30.4MB/s    in 36s

2024-04-12 15:45:58 (27.0 MB/s) - ‘ubuntu-18.04.6-live-server-amd64.iso.2’ saved [1016070144/1016070144]

and the access.log entry looks like this:

1712936758.943  35825 10.40.1.2 TCP_HIT/200 1016070518 GET http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso - HIER_NONE/- application/x-iso9660-image

I am making progress, I just need to understand where I am going wrong with SSL Bump for https connections. Why it is still tunnelling? If I fix that I think it will cache/pull from cache the https downloads too. #fingerscrossed

Any suggestions or decent web blogs/etc on how to configure it?

Have a great weekend,

Many Thanks
Pin

________________________________
From: Jonathan Lee <jonathanlee571 at gmail.com>
Sent: 12 April 2024 15:10
To: PinPin Poola <pinpinpoola at hotmail.com>
Cc: squid-users at lists.squid-cache.org <squid-users at lists.squid-cache.org>
Subject: Re: [squid-users] Squid Cache 6.9 on Ubuntu 22.04.3 LTS. Not caching large files to disk.

Do you have a refresh pattern for .ISO to do this. The defaults for the cache does not cache .ISO files, you have to add a custom refresh pattern for it

Something like this

refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$)                         43800 100% 129600        # RAR | JAR | GZ | TGZ | TAR | BZ2 | ISO

~~~~~
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20240412/1d3944f0/attachment-0001.htm>