[squid-users] Cache dir
Jonathan Lee
jonathanlee571 at gmail.com
Fri Jan 10 17:39:32 UTC 2025
Thanks for the reply
> What OS are you using? How many CPU cores do you want to dedicate to Squid? How much memory?
I am using FreeBSD variant 4GB ram 2 CPUs pfSense plus
Starting CPU 1 (1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
cpu1: <Open Firmware CPU> on cpulist0
e6000sw0: CPU port at 5
CPU 0: ARM Cortex-A53 r0p4 affinity: 0
CPU 1: ARM Cortex-A53 r0p4 affinity: 1
I do not know how to dedicate specific CPU cores to Squid I do not think I can with pfSense plus.
> What filesystem are you using? For modern filesystems (ext4, btrfs, apfs) this parameter is much less meaningful than 10 years ago as they store directories as trees instead of lists.
=> 1 250069679 ada0 MBR (119G)
1 532480 1 efi (260M)
532481 131072 2 fat32 (64M)
663553 249406127 3 freebsd [active] (119G)
=> 0 249406127 ada0s3 BSD (119G)
0 16 - free - (8.0K)
16 235528175 1 freebsd-zfs (112G)
235528191 13877248 2 freebsd-swap (6.6G)
249405439 688 - free - (344K)
=> 40 500118112 nda0 GPT (238G)
40 2008 - free - (1.0M)
2048 16777216 1 freebsd-swap (8.0G)
16779264 482344960 2 freebsd-ufs (230G)
499124224 993928 - free - (485M)
ada0 is for the host os
nda0 is my cache is uses freebsd-ufs I use the command mount_msdosfs /dev/nda0p2 /nvme/LOGS_Octane
The only way to mount the NVMe drive I have found is with mount_msdosfs maybe this causes a slow down I do not know but I can write and save to the drive this way. it is on nda0p2 230GB
Geom name: nda0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 500118151
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: nda0p1
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 1048576
Mode: r1w1e2
efimedia: HD(1,GPT,04d31fb2-c0fd-11ef-8536-90ec770dda25,0x800,0x1000000)
rawuuid: 04d31fb2-c0fd-11ef-8536-90ec770dda25
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: swapUSB
length: 8589934592
offset: 1048576
type: freebsd-swap
index: 1
end: 16779263
start: 2048
2. Name: nda0p2
Mediasize: 246960619520 (230G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 8590983168
Mode: r1w1e1
efimedia: HD(2,GPT,d84dfc00-cb1c-11ef-afd9-90ec770dda25,0x1000800,0x1cc00000)
rawuuid: d84dfc00-cb1c-11ef-afd9-90ec770dda25
rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
label: LOG
length: 246960619520
offset: 8590983168
type: freebsd-ufs
index: 2
end: 499124223
start: 16779264
Consumers:
1. Name: nda0
Mediasize: 256060514304 (238G)
Sectorsize: 512
Mode: r2w2e5
>> my memory replacement policy is LRU for memory cache it seems to run better with that, my options for memory replacement policy are HEAP GDSF I assume any HEAP will require more memory, HEAP LFUDF, HEAP LRU and just LRU.
>
> I think so but shouldn't be significantly more
Should I change Memory Replacement Policy from LRU to HEAP LRU? I have tried every one again there is also the Cache Replacement Policy: Currently set to HEAP LFUDA Should memory replacement policy and cache replacement policy both be the same, and or does one cause any performance issues with the other?
Current Config
# This file is automatically generated by pfSense
# Do not edit manually !
http_port 192.168.1.1:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
icp_port 0
digest_generation off
dns_v4_first on
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname Lee_Family.home.arpa
cache_mgr jonathanlee571 at gmail.com
access_log /nvme/LOGS_Optane/Squid_Logs/access.log
cache_log /nvme/LOGS_Optane/Squid_Logs/cache.log
cache_store_log none
netdb_filename /nvme/LOGS_Optane/Squid_Logs/netdb.state
pinger_enable on
pinger_program /usr/local/libexec/squid/pinger
sslcrtd_program /usr/local/libexec/squid/security_file_certgen -s /var/squid/lib/ssl_db -M 4MB -b 2048
tls_outgoing_options cafile=/usr/local/share/certs/ca-root-nss.crt
tls_outgoing_options capath=/usr/local/share/certs/
tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
tls_outgoing_options cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
sslcrtd_children 10
logfile_rotate 10
debug_options rotate=10
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src 192.168.1.0/27
forwarded_for delete
via off
httpd_suppress_version_string on
uri_whitespace strip
acl block_hours time 00:30-05:00
ssl_bump terminate all block_hours
http_access deny all block_hours
icp_port 0
htcp_port 0
snmp_port 0
icp_access deny all
htcp_access deny all
snmp_access deny all
acl getmethod method GET
acl to_ipv6 dst ipv6
acl from_ipv6 src ipv6
#tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls_outgoing_options options=NO_SSLv3,NO_TLSv1,NO_TLSv1_1,NO_TICKET,SINGLE_DH_USE,SINGLE_ECDH_USE
#tls_outgoing_options default-ca=on
acl HttpAccess dstdomain '/usr/local/pkg/http.access'
acl windowsupdate dstdomain '/usr/local/pkg/windowsupdate'
#acl rewritedoms dstdomain '/usr/local/pkg/desdom'
#store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
#store_id_children 10 startup=5 idle=1 concurrency=0
#always_direct allow all
#store_id_access deny connect
#store_id_access deny !getmethod
#store_id_access allow rewritedoms
#store_id_access deny all
refresh_all_ims on
reload_into_ims on
max_stale 20 years
minimum_expiry_time 0
#refresh_pattern -i ^http.*squid.internal.* 43200 100% 79900 override-expire override-lastmod ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth
refresh_pattern -i windowsupdate.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i windows.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com.akadns.net/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i deploy.akamaitechnologies.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
acl https_login url_regex -i ^https.*(login|Login).*
cache deny https_login
#range_offset_limit 512 MB windowsupdate
range_offset_limit 0 !windowsupdate
quick_abort_min -1 KB
cache_mem 64 MB
maximum_object_size_in_memory 256 KB
memory_replacement_policy lru
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 512 MB
cache_dir aufs /nvme/LOGS_Optane/Squid_Cache 32000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
acl donotcache dstdomain '/var/squid/acl/donotcache.acl'
cache deny donotcache
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320
#Remote proxies
# Setup some default acls
# ACLs all, manager, localhost, and to_localhost are predefined.
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 8080 3128 3129 1025-65535
acl sslports port 443 563 8080 5223 2197
acl purge method PURGE
acl connect method CONNECT
# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
# SslBump Peek and Splice
# http://wiki.squid-cache.org/Features/SslPeekAndSplice
# http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit
# Match against the current step during ssl_bump evaluation [fast]
# Never matches and should not be used outside the ssl_bump context.
#
# At each SslBump step, Squid evaluates ssl_bump directives to find
# the next bumping action (e.g., peek or splice). Valid SslBump step
# values and the corresponding ssl_bump evaluation moments are:
# SslBump1: After getting TCP-level and HTTP CONNECT info.
# SslBump2: After getting TLS Client Hello info.
# SslBump3: After getting TLS Server Hello info.
# These ACLs exist even when 'SSL/MITM Mode' is set to 'Custom' so that
# they can be used there for custom configuration.
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
acl banned_hosts src '/var/squid/acl/banned_hosts.acl'
acl blacklist dstdom_regex -i '/var/squid/acl/blacklist.acl'
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports
# Always allow localhost connections
http_access allow localhost
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 95
request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc
# Reverse Proxy settings
deny_info TCP_RESET allsrc
# Package Integration
url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
url_rewrite_bypass off
url_rewrite_children 25 startup=12 idle=8 concurrency=0
# Custom options before auth
#host_verify_strict on
# These hosts are banned
http_access deny banned_hosts
# Block access to blacklist domains
http_access deny blacklist
# List of domains allowed to logging in to Google services
request_header_access X-GoogApps-Allowed-Domains deny all
request_header_add X-GoogApps-Allowed-Domains consumer_accounts
# Set YouTube safesearch restriction
acl youtubedst dstdomain -n www.youtube.com m.youtube.com youtubei.googleapis.com youtube.googleapis.com www.youtube-nocookie.com
request_header_access YouTube-Restrict deny all
request_header_add YouTube-Restrict none youtubedst
# Custom SSL/MITM options before auth
acl wpad urlpath_regex ^/wpad.dat$
acl wpad urlpath_regex ^/proxy.pac$
acl wpad urlpath_regex ^/wpad.da$
deny_info TCP_RESET wpad
#deny_info 200:/etc/squid/wpad.dat wpad
reply_header_access Content-Type deny wpad
http_access deny wpad
http_access deny !safeports
http_access deny CONNECT !sslports
cachemgr_passwd disable offline_toggle reconfigure shutdown
cachemgr_passwd redacted all
eui_lookup on
acl no_miss url_regex -i gateway.facebook.com/ws/realtime?
acl no_miss url_regex -i web-chat-e2ee.facebook.com/ws/chat
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow CONNECT windowsupdate localnet
http_access allow CONNECT windowsupdate localhost
http_access allow CONNECT HttpAccess localnet
http_access allow CONNECT HttpAccess localhost
http_access deny to_ipv6
http_access deny from_ipv6
acl BrokenButTrustedServers dstdomain '/usr/local/pkg/dstdom.broken'
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all
acl splice_only_ip src 192.168.1.8
acl splice_only_ip src 192.168.1.10
acl splice_only_ip src 192.168.1.11
acl splice_only_ip src 192.168.1.15
acl splice_only_ip src 192.168.1.16
:::: = redacted mac address
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl NoSSLIntercept ssl::server_name_regex -i '/usr/local/pkg/reg.url.nobump'
acl NoBumpDNS dstdomain -n '/usr/local/pkg/dns.nobump'
acl SSL_Intercept_Terminate dstdomain -n '/usr/local/pkg/url.bump'
acl active_use annotate_client active=true
acl bump_only_ip src 192.168.1.3
acl bump_only_ip src 192.168.1.4
acl bump_only_ip src 192.168.1.5
#acl bump_only_ip src 192.168.1.6
acl bump_only_ip src 192.168.1.9
acl bump_only_ip src 192.168.1.13
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
#acl bump_only_mac arp :::::
coredump_dir /nvme/LOGS_Optane/Squid_Dump
acl splice_group any-of https_login NoBumpDNS NoSSLIntercept
acl splice_only_local_group all-of splice_only_mac splice_only_ip
acl splice_main any-of splice_group splice_only_local_group
acl bump_main all-of bump_only_mac bump_only_ip
ssl_bump peek step1
ssl_bump terminate SSL_Intercept_Terminate
miss_access deny no_miss active_use
ssl_bump splice splice_main active_use
ssl_bump bump bump_main active_use
acl activated note active_use true
ssl_bump terminate !activated
# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc
Does delay pool setting cause any issues? They seem to be default values one pool.
> On Jan 10, 2025, at 00:25, Francesco Chemolli <gkinkie at gmail.com> wrote:
>
>
>
> On Fri, Jan 10, 2025 at 7:22 AM Jonathan Lee <jonathanlee571 at gmail.com <mailto:jonathanlee571 at gmail.com>> wrote:
>>
>> After trying every setting inside of Squid I thought I should ask I have 4bg ram and 128GB M.2 SSD onboard disk I am using a NVMe secondary Intel Optane M.2 drive for my cache.
>
> What OS are you using? How many CPU cores do you want to dedicate to Squid? How much memory?
>
>> What is a good recommendation for Hard Drive Cache System I use UFS but AUFS
>
> UFS is the slowest option; AUFS or rock are considered the fastest
>
>> inside of the Squid definitive guide says it is way faster like formula one versus UFS the options I have are UFS AUFS DISKD, I have 16 Level 1 directories
>
> What filesystem are you using? For modern filesystems (ext4, btrfs, apfs) this parameter is much less meaningful than 10 years ago as they store directories as trees instead of lists.
>
>> my memory replacement policy is LRU for memory cache it seems to run better with that, my options for memory replacement policy are HEAP GDSF I assume any HEAP will require more memory, HEAP LFUDF, HEAP LRU and just LRU.
>
> I think so but shouldn't be significantly more
>
>> I also have a Cache Replacement policy with the same options I have it set to Heap LFUDA that is the default. Squid Memory cache Size I have set to default 64MB with max object size 256kb for the memory, for disk I have 256GB available I only have it set to 32000MB or 3.2GB for fear of overloading the RAM when it fills up.
>
> 4gb is plenty of memory; what other workloads do you want to run on that machine? You can also tune these parameters after checking behaviour in practice, no need to fix them once and for all now
>
>> for level 1 directories I can have 4,8,16,32,64,128,256 each layer one containers 256 sub directories so this could hog memory if you did
>> 256*256=65,536 I imagine not ok with only 4GB I have onboard memory I can’t make it any bigger. I use this with SSL intercept it does cache and works well I just want to get rid of the lag on news websites.
>
> number of directories has no impact on memory use. Just be aware that if you change it, you need to wipe and rebuild your cache.
>
>> rewrite process children I have it set max 25 with process children startup at 12 and idle at 8
>> SSL certificate deamon children I have it set to start 10
>
> Sure.
>
>> it runs well I have tried many different things as you know from all the emails, I am sorry it is the most fascinating software to me. Code that runs as fast as the internet. Is there any thing I can do to make it go faster? Some website have a lag fox news yahoo only do on the SSL intercept devices the splice devices never have any issues, its lighting fast for them. I thought I should finally ask after 4-5 years of doing changes. I have got it to work as fast as I can on my own, time to ask the community.
>
> I think it boils down mainly to how much memory you're willing to dedicate to squid. More memory more performance. Apart from that, they main advice is to chang from UFS to just about anything else
>
> --
> Francesco
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20250110/e5c0959a/attachment-0001.htm>
More information about the squid-users
mailing list