[squid-users] Cache dir
Francesco Chemolli
gkinkie at gmail.com
Sat Jan 11 13:24:43 UTC 2025
On Fri, Jan 10, 2025 at 5:39 PM Jonathan Lee <jonathanlee571 at gmail.com>
wrote:
> Thanks for the reply
>
>
> What OS are you using? How many CPU cores do you want to dedicate to
> Squid? How much memory?
>
>
> I am using FreeBSD variant 4GB ram 2 CPUs pfSense plus
>
> Starting CPU 1 (1)
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> cpulist0: <Open Firmware CPU Group> on ofwbus0
> cpu0: <Open Firmware CPU> on cpulist0
> cpu1: <Open Firmware CPU> on cpulist0
> e6000sw0: CPU port at 5
> CPU 0: ARM Cortex-A53 r0p4 affinity: 0
> CPU 1: ARM Cortex-A53 r0p4 affinity: 1
> I do not know how to dedicate specific CPU cores to Squid I do not think I
> can with pfSense plus.
>
It's more about how many squid workers to start. Then the OS will do the
allocation
> What filesystem are you using? For modern filesystems (ext4, btrfs, apfs)
> this parameter is much less meaningful than 10 years ago as they store
> directories as trees instead of lists.
>
>
> => 1 250069679 ada0 MBR (119G)
> 1 532480 1 efi (260M)
> 532481 131072 2 fat32 (64M)
> 663553 249406127 3 freebsd [active] (119G)
>
> => 0 249406127 ada0s3 BSD (119G)
> 0 16 - free - (8.0K)
> 16 235528175 1 freebsd-zfs (112G)
> 235528191 13877248 2 freebsd-swap (6.6G)
> 249405439 688 - free - (344K)
>
> => 40 500118112 nda0 GPT (238G)
> 40 2008 - free - (1.0M)
> 2048 16777216 1 freebsd-swap (8.0G)
> 16779264 482344960 2 freebsd-ufs (230G)
> 499124224 993928 - free - (485M)
>
> ada0 is for the host os
> nda0 is my cache is uses freebsd-ufs I use the command *mount_msdosfs
> /dev/nda0p2 /nvme/**LOGS_Octane*
>
huh? From the output above, it would seem that nda0 is using FreeBD UFS,
not msdos.
Might be related to the partition table type you're using? I'm not a
freebsd expert tho
> The only way to mount the NVMe drive I have found is with mount_msdosfs
> maybe this causes a slow down I do not know but I can write and save to the
> drive this way. it is on nda0p2 230GB
>
> Geom name: nda0
> modified: false
> state: OK
> fwheads: 255
> fwsectors: 63
> last: 500118151
> first: 40
> entries: 128
> scheme: GPT
> Providers:
> 1. Name: nda0p1
> Mediasize: 8589934592 (8.0G)
> Sectorsize: 512
> Stripesize: 0
> Stripeoffset: 1048576
> Mode: r1w1e2
> efimedia: HD(1,GPT,04d31fb2-c0fd-11ef-8536-90ec770dda25,0x800,0x1000000)
> rawuuid: 04d31fb2-c0fd-11ef-8536-90ec770dda25
> rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
> label: swapUSB
> length: 8589934592
> offset: 1048576
> type: freebsd-swap
> index: 1
> end: 16779263
> start: 2048
> 2. Name: nda0p2
> Mediasize: 246960619520 (230G)
> Sectorsize: 512
> Stripesize: 0
> Stripeoffset: 8590983168
> Mode: r1w1e1
> efimedia: HD(2,GPT,d84dfc00-cb1c-11ef-afd9-90ec770dda25,0x1000800,0x1cc00000)
> rawuuid: d84dfc00-cb1c-11ef-afd9-90ec770dda25
> rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
> label: LOG
> length: 246960619520
> offset: 8590983168
> type: freebsd-ufs
> index: 2
> end: 499124223
> start: 16779264
> Consumers:
> 1. Name: nda0
> Mediasize: 256060514304 (238G)
> Sectorsize: 512
> Mode: r2w2e5
>
>
> my memory replacement policy is LRU for memory cache it seems to run
>> better with that, my options for memory replacement policy are HEAP GDSF I
>> assume any HEAP will require more memory, HEAP LFUDF, HEAP LRU and just LRU.
>
>
> I think so but shouldn't be significantly more
>
>
> Should I change Memory Replacement Policy from LRU to HEAP LRU? I have
> tried every one again there is also the Cache Replacement Policy: Currently
> set to HEAP LFUDA Should memory replacement policy and cache replacement
> policy both be the same, and or does one cause any performance issues
> with the other?
>
They should not be too different in terms of memory useage.
>
> Current Config
>
> # This file is automatically generated by pfSense
> # Do not edit manually !
>
> http_port 192.168.1.1:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
>
> http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
>
> https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
>
> icp_port 0
> digest_generation off
> dns_v4_first on
> pid_filename /var/run/squid/squid.pid
> cache_effective_user squid
> cache_effective_group proxy
> error_default_language en
> icon_directory /usr/local/etc/squid/icons
> visible_hostname Lee_Family.home.arpa
> cache_mgr jonathanlee571 at gmail.com
> access_log /nvme/LOGS_Optane/Squid_Logs/access.log
> cache_log /nvme/LOGS_Optane/Squid_Logs/cache.log
> cache_store_log none
> netdb_filename /nvme/LOGS_Optane/Squid_Logs/netdb.state
> pinger_enable on
> pinger_program /usr/local/libexec/squid/pinger
> sslcrtd_program /usr/local/libexec/squid/security_file_certgen -s /var/squid/lib/ssl_db -M 4MB -b 2048
> tls_outgoing_options cafile=/usr/local/share/certs/ca-root-nss.crt
> tls_outgoing_options capath=/usr/local/share/certs/
> tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
> tls_outgoing_options cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
> sslcrtd_children 10
>
> logfile_rotate 10
> debug_options rotate=10
> shutdown_lifetime 3 seconds
> # Allow local network(s) on interface(s)
> acl localnet src 192.168.1.0/27
> forwarded_for delete
> via off
> httpd_suppress_version_string on
> uri_whitespace strip
>
> acl block_hours time 00:30-05:00
> ssl_bump terminate all block_hours
> http_access deny all block_hours
> icp_port 0
> htcp_port 0
> snmp_port 0
> icp_access deny all
> htcp_access deny all
> snmp_access deny all
> acl getmethod method GET
> acl to_ipv6 dst ipv6
> acl from_ipv6 src ipv6
>
> #tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
> tls_outgoing_options options=NO_SSLv3,NO_TLSv1,NO_TLSv1_1,NO_TICKET,SINGLE_DH_USE,SINGLE_ECDH_USE
> #tls_outgoing_options default-ca=on
>
> acl HttpAccess dstdomain '/usr/local/pkg/http.access'
> acl windowsupdate dstdomain '/usr/local/pkg/windowsupdate'
> #acl rewritedoms dstdomain '/usr/local/pkg/desdom'
>
> #store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
> #store_id_children 10 startup=5 idle=1 concurrency=0
> #always_direct allow all
> #store_id_access deny connect
> #store_id_access deny !getmethod
> #store_id_access allow rewritedoms
> #store_id_access deny all
>
> refresh_all_ims on
> reload_into_ims on
> max_stale 20 years
> minimum_expiry_time 0
>
> #refresh_pattern -i ^http.*squid.internal.* 43200 100% 79900 override-expire override-lastmod ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth
>
> refresh_pattern -i windowsupdate.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) <http://windowsupdate.com/.*.(cab%7Cexe%7Cms%5Bi%7Cu%7Cf%7Cp%5D%7C%5Bap%5Dsf%7Cwm%5Bv%7Ca%5D%7Cdat%7Czip%7Cpsf)> 43200 80% 129600 reload-into-ims
> refresh_pattern -i microsoft.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) <http://microsoft.com/.*.(cab%7Cexe%7Cms%5Bi%7Cu%7Cf%7Cp%5D%7C%5Bap%5Dsf%7Cwm%5Bv%7Ca%5D%7Cdat%7Czip%7Cpsf)> 43200 80% 129600 reload-into-ims
> refresh_pattern -i windows.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) <http://windows.com/.*.(cab%7Cexe%7Cms%5Bi%7Cu%7Cf%7Cp%5D%7C%5Bap%5Dsf%7Cwm%5Bv%7Ca%5D%7Cdat%7Czip%7Cpsf)> 43200 80% 129600 reload-into-ims
> refresh_pattern -i microsoft.com.akadns.net/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) <http://microsoft.com.akadns.net/.*.(cab%7Cexe%7Cms%5Bi%7Cu%7Cf%7Cp%5D%7C%5Bap%5Dsf%7Cwm%5Bv%7Ca%5D%7Cdat%7Czip%7Cpsf)> 43200 80% 129600 reload-into-ims
> refresh_pattern -i deploy.akamaitechnologies.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) <http://deploy.akamaitechnologies.com/.*.(cab%7Cexe%7Cms%5Bi%7Cu%7Cf%7Cp%5D%7C%5Bap%5Dsf%7Cwm%5Bv%7Ca%5D%7Cdat%7Czip%7Cpsf)> 43200 80% 129600 reload-into-ims
>
>
> acl https_login url_regex -i ^https.*(login|Login).*
> cache deny https_login
>
> #range_offset_limit 512 MB windowsupdate
> range_offset_limit 0 !windowsupdate
> quick_abort_min -1 KB
> cache_mem 64 MB
> maximum_object_size_in_memory 256 KB
> memory_replacement_policy lru
> cache_replacement_policy heap LFUDA
> minimum_object_size 0 KB
> maximum_object_size 512 MBcache_dir aufs /nvme/LOGS_Optane/Squid_Cache 32000 16 256
> offline_mode off
> cache_swap_low 90
> cache_swap_high 95
> acl donotcache dstdomain '/var/squid/acl/donotcache.acl'
> cache deny donotcache
> cache allow all
> # Add any of your own refresh_pattern entries above these.
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|?) 0 0% 0
> refresh_pattern . 0 20% 4320
>
>
> #Remote proxies
>
>
> # Setup some default acls
> # ACLs all, manager, localhost, and to_localhost are predefined.
> acl allsrc src all
> acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 8080 3128 3129 1025-65535
> acl sslports port 443 563 8080 5223 2197
>
> acl purge method PURGE
> acl connect method CONNECT
>
> # Define protocols used for redirects
> acl HTTP proto HTTP
> acl HTTPS proto HTTPS
>
> # SslBump Peek and Splice
> # http://wiki.squid-cache.org/Features/SslPeekAndSplice
> # http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit
> # Match against the current step during ssl_bump evaluation [fast]
> # Never matches and should not be used outside the ssl_bump context.
> #
> # At each SslBump step, Squid evaluates ssl_bump directives to find
> # the next bumping action (e.g., peek or splice). Valid SslBump step
> # values and the corresponding ssl_bump evaluation moments are:
> # SslBump1: After getting TCP-level and HTTP CONNECT info.
> # SslBump2: After getting TLS Client Hello info.
> # SslBump3: After getting TLS Server Hello info.
> # These ACLs exist even when 'SSL/MITM Mode' is set to 'Custom' so that
> # they can be used there for custom configuration.
> acl step1 at_step SslBump1
> acl step2 at_step SslBump2
> acl step3 at_step SslBump3
> acl banned_hosts src '/var/squid/acl/banned_hosts.acl'
> acl blacklist dstdom_regex -i '/var/squid/acl/blacklist.acl'
> http_access allow manager localhost
>
> http_access deny manager
> http_access allow purge localhost
> http_access deny purge
> http_access deny !safeports
> http_access deny CONNECT !sslports
>
> # Always allow localhost connections
> http_access allow localhost
>
> quick_abort_min 0 KB
> quick_abort_max 0 KB
> quick_abort_pct 95
> request_body_max_size 0 KBdelay_pools 1
> delay_class 1 2
> delay_parameters 1 -1/-1 -1/-1
> delay_initial_bucket_level 100
> delay_access 1 allow allsrc
>
> # Reverse Proxy settings
>
> deny_info TCP_RESET allsrc
>
> # Package Integration
> url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
> url_rewrite_bypass off
> url_rewrite_children 25 startup=12 idle=8 concurrency=0
>
> # Custom options before auth
> #host_verify_strict on
>
> # These hosts are banned
> http_access deny banned_hosts
> # Block access to blacklist domains
> http_access deny blacklist
> # List of domains allowed to logging in to Google services
> request_header_access X-GoogApps-Allowed-Domains deny all
> request_header_add X-GoogApps-Allowed-Domains consumer_accounts
> # Set YouTube safesearch restriction
> acl youtubedst dstdomain -n www.youtube.com m.youtube.com youtubei.googleapis.com youtube.googleapis.com www.youtube-nocookie.com
> request_header_access YouTube-Restrict deny all
> request_header_add YouTube-Restrict none youtubedst
> # Custom SSL/MITM options before auth
> acl wpad urlpath_regex ^/wpad.dat$
> acl wpad urlpath_regex ^/proxy.pac$
> acl wpad urlpath_regex ^/wpad.da$
> deny_info TCP_RESET wpad
> #deny_info 200:/etc/squid/wpad.dat wpad
> reply_header_access Content-Type deny wpad
> http_access deny wpad
> http_access deny !safeports
> http_access deny CONNECT !sslports
> cachemgr_passwd disable offline_toggle reconfigure shutdown
> cachemgr_passwd redacted all
> eui_lookup on
> acl no_miss url_regex -i gateway.facebook.com/ws/realtime?
> acl no_miss url_regex -i web-chat-e2ee.facebook.com/ws/chat
> acl CONNECT method CONNECT
> acl wuCONNECT dstdomain www.update.microsoft.com
> acl wuCONNECT dstdomain sls.microsoft.com
> http_access allow CONNECT wuCONNECT localnet
> http_access allow CONNECT wuCONNECT localhost
> http_access allow CONNECT windowsupdate localnet
> http_access allow CONNECT windowsupdate localhost
> http_access allow CONNECT HttpAccess localnet
> http_access allow CONNECT HttpAccess localhost
> http_access deny to_ipv6
> http_access deny from_ipv6
>
> acl BrokenButTrustedServers dstdomain '/usr/local/pkg/dstdom.broken'
> acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
> sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
> sslproxy_cert_error deny all
>
> acl splice_only_ip src 192.168.1.8
> acl splice_only_ip src 192.168.1.10
> acl splice_only_ip src 192.168.1.11
> acl splice_only_ip src 192.168.1.15
> acl splice_only_ip src 192.168.1.16
> :::: = redacted mac address
> acl splice_only_mac arp :::::
> acl splice_only_mac arp :::::
> acl splice_only_mac arp :::::
> acl splice_only_mac arp :::::
> acl splice_only_mac arp :::::
>
> acl NoSSLIntercept ssl::server_name_regex -i '/usr/local/pkg/reg.url.nobump'
> acl NoBumpDNS dstdomain -n '/usr/local/pkg/dns.nobump'
> acl SSL_Intercept_Terminate dstdomain -n '/usr/local/pkg/url.bump'
>
> acl active_use annotate_client active=true
>
> acl bump_only_ip src 192.168.1.3
> acl bump_only_ip src 192.168.1.4
> acl bump_only_ip src 192.168.1.5
> #acl bump_only_ip src 192.168.1.6
> acl bump_only_ip src 192.168.1.9
> acl bump_only_ip src 192.168.1.13
>
> acl bump_only_mac arp :::::
> acl bump_only_mac arp :::::
> acl bump_only_mac arp :::::
> acl bump_only_mac arp :::::
> acl bump_only_mac arp :::::
> #acl bump_only_mac arp :::::
>
> coredump_dir /nvme/LOGS_Optane/Squid_Dump
>
> acl splice_group any-of https_login NoBumpDNS NoSSLIntercept
> acl splice_only_local_group all-of splice_only_mac splice_only_ip
> acl splice_main any-of splice_group splice_only_local_group
> acl bump_main all-of bump_only_mac bump_only_ip
>
> ssl_bump peek step1
> ssl_bump terminate SSL_Intercept_Terminate
> miss_access deny no_miss active_use
> ssl_bump splice splice_main active_use
> ssl_bump bump bump_main active_use
> acl activated note active_use true
> ssl_bump terminate !activated
>
> # Setup allowed ACLs
> # Allow local network(s) on interface(s)
> http_access allow localnet
> # Default block all to be sure
> http_access deny allsrc
>
> Does delay pool setting cause any issues? They seem to be default values
> one pool.
>
If you need them, you need them.
There doesn't seem to be anything obviously wrong with the highlighted
parts of your config, except maybe that the memory cache is tiny.
How much RAM is the OS reporting that squid is using after a few hours or
days of use?
--
Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20250111/e71ccd49/attachment-0001.htm>
More information about the squid-users
mailing list