[squid-users] Performance issue /cache_dir / cache_mem / SMP workers

Thu Nov 22 12:52:48 UTC 2018

Hello people,

@Alex, thanks for your help, we took your advise and change the deployment
to SMP, some issues were solved, but others appeared :-(.

The config files are attached, in case anybody could help or to help others
with our same issues, as we have been searching several days and we didn't
found any solution or a recent configuration guide.

df_-hT.txt
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/df_-hT.txt>  
LVM_info.txt
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/LVM_info.txt>  
fdisk_-l.txt
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/fdisk_-l.txt>  
fstab.fstab
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/fstab.fstab>  
dev_shm_permissions.txt
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/dev_shm_permissions.txt>  
cat_proc_cpuinfo.txt
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/cat_proc_cpuinfo.txt>  
systecl_-a.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/systecl_-a.log>  
99-sysctl.conf
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/99-sysctl.conf>  
squid.conf
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/squid.conf>  
frontend.conf
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/frontend.conf>  
backend.conf
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend.conf>  

1. VMWare - Hardware changes
	Changed the CPU's servers from 2 CPU with 2 sockets (2*2) to 5 CPU with 1
socket each (5*1). 
	Changed from 8 GB to 32 GB RAM
	LVM with an OS disk of 30 GB, and changed the 8TB to several disks of 2 TB,
as suggested, fewer smaller disks better than a big disk.

2. Server - LVM config disks
	Attached are the LVM info, df -hT, fstab, fdisk -l.
	4 disks of 2 TB. 
	Changed the file system from initial xfs to ext4.
	We checked the Reiserfs suggestion, we tried to change it, but found
several problems (discontinuity, system not compatible), so we decided to
remain in ext4.
	Mounted ext4 with noatime.

3. Server - CPU affinity
	Attached is cat /proc/cpuinfo
	I suppose the processor id in /proc/cpuinfo is the id that should be
referenced in the cpu_affinity_map, isn't it?

	cpu_affinity_map process_numbers=1,2,3,4 cores=1,2,3,4

4. systctl
	Taking into account several posts founded, we changed some settings in the
sysctl to improve the server's performance.

	Something very important are the file descriptors, though we configured
this parameter in the squid.conf.
	This actually solve the main problem, as we checked they were being quickly
exhausted.

	There are several parameters related to the memory, but we found some
discrepancies in the examples, so we haven't changed them.

5. shm
	Attached dev_shm permissions. 

	We checked the following error in the /var/log/messages at the squid
start...
		Ipc::Mem::Segment::open failed to shm_open(/squid-cache4_spaces.shm): (2)
No such file or directory	

	So we add the following line to the /etc/fstab file:
		shm        /dev/shm    tmpfs    nodev,nosuid,noexec    0    0

	The errors continued, so we created the files (squid-cache2_spaces.shm,
etc) that appear to be missing, and the errors changed to...
		FATAL: Ipc::Mem::Segment::attach failed to mmap(/squid-cache4_spaces.shm):
(22) Invalid argument

	Regarding this error, we have disabled the SELinux, as suggested in some
post, but the errors continued appearing.

6. Squid configuration
	Attached the squid.conf, frontend.conf and backend.conf.
	We configured them as the Wiki example, although it appears that this
example is too much old, furthermore it is suggesting to use aufs, when it
is not SMP aware.
	Then we changed several things as suggested in several posts of the Nabble
forum.

	We found several posts where issues are reported and Alex suggest to
update, so we updated Squid from 3.5.28 to 4.1, but the errors continued.
	We don't know whether it is coincidence, but almost everybody that is
reporting these issues have our same OS, CentOS.
	Our system is CentOS Linux release 7.5.1804 (Core).

Furthermore, we have attached the log files, in case it helps.

var_log_messages.txt
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/var_log_messages.txt>  
frontend.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/frontend.log>  
backend0.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend0.log>  
backend2.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend2.log>  
backend3.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend3.log>  
backend4.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend4.log>  
backend5.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend5.log>  
backend6.log
<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377599/backend6.log>  

In summary, we don't know how to continue the troubleshooting, please help.

Thanks in advance and sorry for the brick.
Best regards,
Paco.

References:
https://wiki.squid-cache.org/Features/SmpScale
https://wiki.squid-cache.org/ConfigExamples/SmpCarpCluster
https://wiki.squid-cache.org/Features/RockStore
https://wiki.squid-cache.org/BestOsForSquid
http://vietlux.blogspot.com/2012/07/squid-proxy-tuning-for-high-perfomance.html
https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php
http://squid-web-proxy-cache.1019090.n4.nabble.com/FATAL-Ipc-Mem-Segment-create-failed-to-shm-open-squid-cf-queues-shm-17-File-exists-td4680157.html
http://squid-web-proxy-cache.1019090.n4.nabble.com/FATAL-shm-open-squid-ssl-session-cache-shm-td4683398.html

--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html