[squid-users] caching apt package lists/Raspbian

[Amos Jeffries]

On 20/07/19 5:19 pm, TarotApprentice wrote:
> Recently upgraded to Raspbian Buster and squid 4.6. Since then I am
unable to cache the Packages.xz that apt uses. The various other Pis
using this proxy all end up downloading the 30MB Packages.xz every time.
Does anyone have any suggestions on how to get it to cache?
> 
> Cheers MarkJ
> 

According to both Redbot and my manual check the object is only 12MB,
not 30MB. If you are getting 30MB somebody is interfering with that
download.


It should be caching by default. The redbot tool shows the site is
providing all the required cache headers and working perfectly for
revalidation. The REFRESH_UNMODIFIED log entries show that too.

The TCP_MISS_ABORTED indicates that for that log entry there was nothing
in cache (yet) for that URL, and the client aborted the transfer with
only 2.6MB fetched.



Can you try having just one Pi do its update and seeing if the .xz
object is cached afterwards?

Alternatively try the command:
  squidclient
http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz

It the object is cacheable, but your environment tends to have the Pi's
all fetching at the same time (eg before the first finishes), then you
may find collapsed_forwarding feature of use. That helps with caching
parallel fetches of objects.

Amos


> squid -v
> Squid Cache: Version 4.6
> Service Name: squid
> Raspbian linux
> 
> 
> access.log
> 
> 1563597855.786    605 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 15306 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease - HIER_DIRECT/93.93.128.193 -
> 
> 1563597855.811    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 25429 GET http://archive.raspberrypi.org/debian/dists/buster/InRelease - HIER_DIRECT/93.93.128.133 -
> 
> 1563597857.486    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 205801 GET http://archive.raspberrypi.org/debian/dists/buster/main/binary-armhf/Packages.gz - HIER_DIRECT/93.93.128.133 application/x-gzip
> 
> 1563597936.436  80026 192.168.1.73 TCP_MISS_ABORTED/200 2641974 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz - HIER_DIRECT/93.93.128.193 application/x-xz
> 
> 
> config file
> 
...
> acl hiddenwasp2 dstdomain http://103.206.123.13

The above "http://" is not a valid domain name.

> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny ads
> http_access deny malware
> http_access deny malware2
> http_access deny hiddenwasp
> http_access deny hiddenwasp2
> http_access allow l500-020b manager
> http_access deny manager


'dst' ACL is quite slow and resource intensive. You should put these
manager rules above the "malware2" denial to protect against DoS better.

...
> http_port 3128
> cache_mem 448 MB
> maximum_object_size 320 MB
> memory_replacement_policy lru
> cache_replacement_policy heap LFUDA
> cache_dir aufs /var/spool/squid 18432 32 256
> quick_abort_min -1 KB
> client_request_buffer_max_size 128 KB

...

> refresh_pattern (\.deb|\.udeb)$ 1440    80%     10080
> refresh_pattern ^ftp:           1440    20%     10080
> refresh_pattern ^gopher:        1440    0%      1440
> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
> refresh_pattern .               0       20%     4320


Amos