[squid-users] squid cache Large rock with aufs optimization for bandwidth saving
Ahmed Allzaeem
ahmed.zaeem at netstream.ps
Sun Nov 16 07:07:45 UTC 2014
Hi developers .
Im using squid
Squid Cache: Version 3.HEAD-20141105-r13687
With options :
Service Name: squid
configure options: '--prefix=/usr' '--includedir=/include'
'--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc'
'--enable-cachemgr-hostname=drx' '--localstatedir=/var'
'--libexecdir=/lib/squid' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--disable-silent-rules' '--srcdir=.'
'--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
'--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8'
'--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap'
'--enable-delay-pools' '--enable-cache-digests' '--enable-underscores'
'--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam
,squid_radius_auth,multi-domain-NTLM' '--enable-ntlm-auth-helpers=smb_lm'
'--enable-digest-auth-helpers=ldap,password'
'--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-arp-acl'
'--enable-esi' '--disable-translation' '--with-logdir=/var/log/squid'
'--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=131072'
'--with-large-files' '--with-default-user=squid' '--enable-linux-netfilter'
'CFLAGS=-g -O2 -g -Wall -O2' 'LDFLAGS=' 'CPPFLAGS=' 'CXXFLAGS=-g -O2 -g
-Wall -O2' '--enable-ltdl-convenience'
and my squid.conf is :
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
machines
acl mysubnet src xxxxxxxxxxxxx
acl xxxxxxxxxx
acl localip src xxxxxxx
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
######################################################
# Recommended minimum Access Permission configuration:
# Deny requests to certain unsafe ports
##############################################################
http_access allow xxx
http_access allow manager xxx
http_access deny !Safe_ports
http_access allow localnet
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
#######################################################
###########SMP oPTIONS#####################
dns_v4_first on
workers 6
cache_mem 6000 MB
maximum_object_size 4 MB
maximum_object_size_in_memory 4 MB
minimum_object_size 50 KB
########################################################
#cpu_affinity_map process_numbers=5,6,7 cores=1,2,7
http_port 127.0.0.1:400${process_number}
if ${process_number} = 2
###################################
cache_mem 25000 MB
maximum_object_size 4 MB
maximum_object_size_in_memory 4 MB
minimum_object_size 50 KB
###################################
cache_dir aufs /ssd1 90000 64 256
cache_log stdio:/var/log/squid/cache5.log
access_log /var/log/squid/access5.log
endif
####################
####################
if ${process_number} = 6
###################################
cache_mem 6000 MB
maximum_object_size 4 MB
maximum_object_size_in_memory 4 MB
minimum_object_size 50 KB
###################################
cache_dir aufs /ssd2 90000 64 256
cache_log stdio:/var/log/squid/cache6.log
access_log /var/log/squid/access6.log
endif
#################
#################
if ${process_number} = 4
###################################
###################################
cache_dir aufs /ssd3 90000 64 256
cache_log stdio:/var/log/squid/cache7.log
access_log /var/log/squid/access7.log
endif
#######################################
if ${process_number} = 5
###################################
cache_mem 6000 MB
maximum_object_size 4 MB
maximum_object_size_in_memory 4 MB
minimum_object_size 50 KB
###################################
cache_dir aufs /ssd4 40000 64 256
cache_log stdio:/var/log/squid/cache8.log
access_log /var/log/squid/access8.log
endif
#########################################
visible_hostname Largerock-squid
####################################################
####Filterring##############
acl blockkeywords dstdomain "/etc/squid/koko.acl"
http_access deny blockkeywords
#################################################
##Regix filtering##########
acl xacl dstdom_regex "/etc/squid/squid-regex.acl"
http_access deny xacl
##############################
cache_log /var/log/squid/cache.log
access_log /var/log/squid/access.log
###############################################
http_port xxxxx
http_port xxxxxxx tproxy
###
http_access allow mysubnet
# add user authentication and similar options here
http_access allow manager localhost
http_access allow manager localip
http_access allow manager mysubnet
http_access allow mysubnet manager
http_access deny manager
##########################
cache_mem 6000 MB
maximum_object_size 4 MB
maximum_object_size_in_memory 4 MB
minimum_object_size 0 KB
cache_swap_low 90
cache_swap_high 95
###########################################################
#########################################################
quick_abort_pct 95
fqdncache_size 65535
ipcache_size 65535
###########################################################
ipcache_low 98
ipcache_high 99
#####################################################
########### WCCP2 Config#############
wccp2_router xxxxxx
wccp2_rebuild_wait off
wccp_version 2
wccp2_forwarding_method 2
wccp2_return_method 2
wccp2_assignment_method 2
wccp2_service dynamic 80
wccp2_service_info 80 protocol=tcp flags=src_ip_hash priority=250 ports=80
wccp2_service dynamic 90
wccp2_service_info 90 protocol=tcp flags=dst_ip_hash,ports_source
priority=250 ports=80
######################################################
cache_effective_user squid
cache_effective_group squid
#######################################################
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
#####################################################
dns_nameservers x.x.x.x
#######################################################
http_access deny all
##############################
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
######
memory_pools off
pconn_timeout 2 minutes
persistent_request_timeout 1 minute
read_ahead_gap 128 KB
[root at Largerock-squid ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 13G 35G 27% /
tmpfs 100G 5.2G 95G 6% /dev/shm
/dev/sdb1 99G 82G 12G 88% /ssd1
/dev/sdc1 99G 56G 38G 60% /ssd2
/dev/sdd1 99G 75G 19G 81% /ssd3
/dev/sde1 45G 28G 15G 67% /ssd4
But still wondering , why after im using squid3.head that support large rock
and using aufs with 4 processes with object size about 4 M configured in
each prcess .the mean size is about 38 KB ???
This is my biggest problem
I could load the traffic which is about 1000 req/second among my cores
equally ..but I cant save a lot of BW ???!!
Here is cache mamager info :
Cache Manager menu
<http://176.58.67.242:9090/cgi-bin/cachemgr.cgi?host=176.58.67.242&port=6500
0&user_name=a at a&operation=menu&auth=MTc2LjU4LjY3LjI0MnwxNDE2MDg1MzcwfGFAYXxh
>
_____
_____
Squid Object Cache: Version 3.HEAD-20141105-r13687
Build Info:
Service Name: squid
Start Time:
Sat, 15 Nov 2014 06:47:15 GMT
Current Time:
Sat, 15 Nov 2014 21:02:50 GMT
Connection information for squid:
Number of clients accessing cache: 29344
Number of HTTP requests received: 36422429
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 42570.5
Average ICP messages per minute since start: 0.0
Select loop called: 1117931059 times, 0.303 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 15.8%, 60min: 16.0%
Hits as % of bytes sent: 5min: 3.0%, 60min: 2.6%
Memory hits as % of hit requests: 5min: 27.1%, 60min: 28.2%
Disk hits as % of hit requests: 5min: 39.5%, 60min: 40.2%
Storage Swap size: 250049956 KB
Storage Swap capacity: 78.8% used, 21.2% free
Storage Mem size: 4697152 KB
Storage Mem capacity: 76.5% used, 23.5% free
Mean Object Size: 38.16 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.18416 0.17590
Cache Misses: 0.25616 0.24208
Cache Hits: 0.00061 0.00061
Near Hits: 0.07278 0.08295
Not-Modified Replies: 0.00015 0.00030
DNS Lookups: 0.00000 0.00000
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 51334.692 seconds
CPU Time: 95558.407 seconds
CPU Usage: 186.15%
CPU Usage, 5 minute avg: 269.60%
CPU Usage, 60 minute avg: 270.21%
Maximum Resident Size: 145364816 KB
Page faults with physical i/o: 5
Memory accounted for:
Total accounted: 3392882 KB
memPoolAlloc calls: 1736
memPoolFree calls: 10645847682
File descriptor usage for squid:
Maximum number of file descriptors: 786432
Largest file desc currently in use: 8453
Number of file desc currently in use: 38409
Files queued for open: 0
Available number of file descriptors: 748023
Reserved number of file descriptors: 600
Store Disk files open: 124
Internal Data Structures:
6556333 StoreEntries
3738 StoreEntries with MemObjects
73065 Hot Object Cache Items
6552722 on-disk objects
Any help ??????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20141115/7bc6a6a4/attachment-0001.html>
More information about the squid-users
mailing list