[squid-users] caching apt package lists/Raspbian

[Mark James]

Doing an “apt update” on the squid machine got another TCP_MISS_ABORTED for ::1 and then subsequent IPv4 requests from other Pis get the TCP_REQUEST_UNMODIFIED.

Packages.xz was 13MB.


> On 21 Jul 2019, at 12:36 am, Amos Jeffries <squid3 at treenet.co.nz> wrote:
> 
>> On 20/07/19 5:19 pm, TarotApprentice wrote:
>> Recently upgraded to Raspbian Buster and squid 4.6. Since then I am
> unable to cache the Packages.xz that apt uses. The various other Pis
> using this proxy all end up downloading the 30MB Packages.xz every time.
> Does anyone have any suggestions on how to get it to cache?
>> 
>> Cheers MarkJ
>> 
> 
> According to both Redbot and my manual check the object is only 12MB,
> not 30MB. If you are getting 30MB somebody is interfering with that
> download.
> 
> 
> It should be caching by default. The redbot tool shows the site is
> providing all the required cache headers and working perfectly for
> revalidation. The REFRESH_UNMODIFIED log entries show that too.
> 
> The TCP_MISS_ABORTED indicates that for that log entry there was nothing
> in cache (yet) for that URL, and the client aborted the transfer with
> only 2.6MB fetched.
> 
> 
> 
> Can you try having just one Pi do its update and seeing if the .xz
> object is cached afterwards?
> 
> Alternatively try the command:
>  squidclient
> http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz
> 
> It the object is cacheable, but your environment tends to have the Pi's
> all fetching at the same time (eg before the first finishes), then you
> may find collapsed_forwarding feature of use. That helps with caching
> parallel fetches of objects.
> 
> Amos
> 
> 
>> squid -v
>> Squid Cache: Version 4.6
>> Service Name: squid
>> Raspbian linux
>> 
>> 
>> access.log
>> 
>> 1563597855.786    605 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 15306 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease - HIER_DIRECT/93.93.128.193 -
>> 
>> 1563597855.811    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 25429 GET http://archive.raspberrypi.org/debian/dists/buster/InRelease - HIER_DIRECT/93.93.128.133 -
>> 
>> 1563597857.486    620 192.168.1.73 TCP_REFRESH_UNMODIFIED/200 205801 GET http://archive.raspberrypi.org/debian/dists/buster/main/binary-armhf/Packages.gz - HIER_DIRECT/93.93.128.133 application/x-gzip
>> 
>> 1563597936.436  80026 192.168.1.73 TCP_MISS_ABORTED/200 2641974 GET http://raspbian.raspberrypi.org/raspbian/dists/buster/main/binary-armhf/Packages.xz - HIER_DIRECT/93.93.128.193 application/x-xz
>> 
>> 
>> config file
>> 
> ...
>> acl hiddenwasp2 dstdomain http://103.206.123.13
> 
> The above "http://" is not a valid domain name.
> 
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access deny ads
>> http_access deny malware
>> http_access deny malware2
>> http_access deny hiddenwasp
>> http_access deny hiddenwasp2
>> http_access allow l500-020b manager
>> http_access deny manager
> 
> 
> 'dst' ACL is quite slow and resource intensive. You should put these
> manager rules above the "malware2" denial to protect against DoS better.
> 
> ...
>> http_port 3128
>> cache_mem 448 MB
>> maximum_object_size 320 MB
>> memory_replacement_policy lru
>> cache_replacement_policy heap LFUDA
>> cache_dir aufs /var/spool/squid 18432 32 256
>> quick_abort_min -1 KB
>> client_request_buffer_max_size 128 KB
> 
> ...
> 
>> refresh_pattern (\.deb|\.udeb)$ 1440    80%     10080
>> refresh_pattern ^ftp:           1440    20%     10080
>> refresh_pattern ^gopher:        1440    0%      1440
>> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
>> refresh_pattern .               0       20%     4320
> 
> 
> Amos
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users