[squid-users] Squid3: 100 % CPU load during object caching

[Amos Jeffries]

On 24/07/2015 7:49 p.m., Jens Offenbach wrote:
> I have found something out... Hopefully, it helps to reproduce and solve the issue. 
> 
> I got it working with a good download rate, but very high CPU usage on Squid 3.3.8 and Squid 3.5.6. There seems to be problem with large files that get cached on disk in combination with memory caching. When I use these settings, memory usage of Squid grows step-by-step with 100% CPU usage and 500 KB/sec download rate:
> 
> # MEMORY CACHE OPTIONS
> # -----------------------------------------------------------------------------
>   maximum_object_size_in_memory 1 GB
>   memory_replacement_policy heap LFUDA
>   cache_mem 4 GB
> 
> # DISK CACHE OPTIONS
> # -----------------------------------------------------------------------------
>   maximum_object_size 10 GB
>   cache_replacement_policy heap GDSF
>   cache_dir aufs /var/cache/squid3 25600 16 256
> 
> I decided to turn off memory caching completely and used the following settings:
> 
> # MEMORY CACHE OPTIONS
> # -----------------------------------------------------------------------------
>   maximum_object_size_in_memory 0 GB
>   memory_replacement_policy heap LFUDA
>   cache_mem 0 GB
> 
> # DISK CACHE OPTIONS
> # -----------------------------------------------------------------------------
>   maximum_object_size 10 GB
>   cache_replacement_policy heap GDSF
>   cache_dir aufs /var/cache/squid3 25600 16 256
> 
> Now, I get stable and high download rates even on a cache miss.
> 

Damn. That gives me ~90% confidence its the mem_node walking as new 4KB
chunks of memory are appended to the memory copy of the object.


I would expect to see a reduced effect in 3.5 that kicks in around
maximum_object_size_in_memory. Since the memory copies are now split
into cache_mem objects, vs transients (disk cache only, or totally
non-cacheable). With the transients getting their unnecessary in-memory
sections pruned away regularly.

Amos