[squid-users] Bug: Missing MemObject::storeId value

Tue Sep 26 04:57:11 UTC 2017

Yeah, sounds like I need to prove that ssl-bump is not eating memory
before I start worrying about caching.    Then slowly add features
until I find the smoking gun and focus on that.

I'm curious, does anyone have a suggestion of what modern high traffic
volume squid deployments look like? Seems like lots of the suggestions
are a bit out dated.  I'm trying to go with the KISS principle and not
do any fancy ICP/etc or multi-layer proxy config since that seems much
more difficult to deploy and benchmark.  Instead we're using haproxy
to have cache affinity across systems.  Obviously this may result in
some hot spotting, but it seems like we'll need enough servers that
hopefully the pain will be distributed.

The reason I'm looking at squid is that I've got a small server farm
of ~850 web clients which will be making ~10M page requests/day.
Right now I'm estimating about 50% of my traffic is SSL so bumping SSL
connections is pretty important.

--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune

On Mon, Sep 25, 2017 at 9:21 PM, Eliezer Croitoru <eliezer at ngtech.co.il> wrote:
> Hey Aaron,
>
> Consider the comments from Amos and Alex first before moving forward.
> And again we need to clear out the current doubt's for both you and us.
> We don't know if the issue is related to rock cache_dir or to squid-cache in general.
> Currently for SMP aware caches the best disk cache is rock but you need to understand that the situation is that disk cache is a second level of caching and not the main goal.
> You first need to make sure that squid works for you and then to make sure rock works good enough for you.
> Also take into account that you actually "all in" for disk caching and it's not clear if you even need all this cache.
> Before you decide that the disk cache is for you and that you really need it start low and aim higher, then in small steps move forward.
> Start with a simple squid with ssl-bump without caching at all, then when you see it's stable enough from basic memory perspective for a period of 24 to 72 hours.
> Then and only then when you see it's stable enough for you and the machine can take the load try to see if adding memory cache into the picture makes sense.
> Check squid with it's default settings of cache_object sizes and try to analyze the cache logs to verify what are the most hot sites and objects that in use of your cache.
> Only when you will have a clear view what is the demand from your cache proxy service you should consider moving forward to start investigating the usage of disk cache(with default cache object sizes).
> Take into account that there is a possibility that squid will write object to the disk cache but will not use then and this is a very good reason to first test and analyze before going all in or out with squid.
> Also start with a small disk cache(10GB max) and only after verifying that indeed the setup is working good enough try to find the right memory and disk cache utilization for your setup.
>
> The above is my recommended recipe for a good and smooth start with squid in production environment.
> You are not the first and probably not the last to receive this recommendation and I believe that some articles and resources that can be fetched from the Internet can miss-lead a Linux system administrator expectation from squid-cache or any cache.
>
> Please test Squid-Cache one step at a time and do not get tempted to try to "cache all" since it's practically not possible.
> Update us as you move forward with your tests.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: eliezer at ngtech.co.il
>
>
>
> -----Original Message-----
> From: Aaron Turner [mailto:synfinatic at gmail.com]
> Sent: Monday, September 25, 2017 22:57
> To: Eliezer Croitoru <eliezer at ngtech.co.il>
> Cc: squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] Bug: Missing MemObject::storeId value
>
> So is v4 stable?  I was the impression it was beta?  That said, if v4
> has better memory tuning options then I'm all ears.  Right now I'm
> fighting OOM errors (and the kernel OOM reaper) under sustained load.
> I've come to realize 6GB is way way too much for my 14GB RAM systems,
> but finding even 1GB is too much since each squid process is exceeding
> 4GB.  About to try 500MB now.
>
> I can disable rock cache, but I need some disk cache- is there a better option?
>
> As for haproxy, I actually don't care about the client IP... I'm
> running haproxy locally on the servers where the clients reside.
> Mostly I'm using it for squid failover and cache affinity so I don't
> have to make all my caches peers of each other.
>
>
> --
> Aaron Turner
> https://synfin.net/         Twitter: @synfinatic
> My father once told me that respect for the truth comes close to being
> the basis for all morality.  "Something cannot emerge from nothing,"
> he said.  This is profound thinking if you understand how unstable
> "the truth" can be.  -- Frank Herbert, Dune
>
>
> On Mon, Sep 25, 2017 at 11:45 AM, Eliezer Croitoru <eliezer at ngtech.co.il> wrote:
>> Hey Aaron,
>>
>> Just to clear out the doubt's, what happen when you use squid-cache without rock cache_dir? Is the problem appearing again?
>> Also, there is a possibility of a bug which is related to squid ssl-bump termination code on 3.5.X.
>> Testing 4.0.21 would be the best to understand if the issue is 3.5 local or if it was fixed in 4.X+ but, from my memory I think you will need to adapt your squid.conf ssl_bump configurations.
>> You can get the latest beta and stable binaries from my repo and the beta repo details are at:
>> https://wiki.squid-cache.org/action/edit/KnowledgeBase/CentOS#Squid_Beta_release
>>
>> Also, since you are using haproxy in front of squid I would suggest you to use the proxy protocol(v1) which is the best way to pass the source ip addresses to the proxy.
>> I have tested squid to work with the proxy protocol v1 but yet to test v2.
>>
>> All The Bests,
>> Eliezer
>>
>> ----
>> Eliezer Croitoru
>> Linux System Administrator
>> Mobile: +972-5-28704261
>> Email: eliezer at ngtech.co.il
>>
>>
>>
>> -----Original Message-----
>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Aaron Turner
>> Sent: Saturday, September 23, 2017 02:19
>> To: squid-users at lists.squid-cache.org
>> Subject: [squid-users] Bug: Missing MemObject::storeId value
>>
>> Version: 3.5.26 on CentOS 7.3 on AWS EC2 m3.xlarge and 2x 100GB EBS
>> volumes for rock cache.
>>
>> Doing some basic system tests and we're seeing a bunch of errors like:
>>
>> 2017/09/22 22:43:15 kid1| Bug: Missing MemObject::storeId value
>> 2017/09/22 22:43:15 kid1| mem_hdr: 0x7f169d0a2a70 nodes.start() 0x7f169c6cc9d0
>> 2017/09/22 22:43:15 kid1| mem_hdr: 0x7f169d0a2a70 nodes.finish() 0x7f169dae4e40
>> 2017/09/22 22:43:15 kid1| MemObject->start_ping: 0.000000
>> 2017/09/22 22:43:15 kid1| MemObject->inmem_hi: 20209
>> 2017/09/22 22:43:15 kid1| MemObject->inmem_lo: 0
>> 2017/09/22 22:43:15 kid1| MemObject->nclients: 0
>> 2017/09/22 22:43:15 kid1| MemObject->reply: 0x7f167ee60db0
>> 2017/09/22 22:43:15 kid1| MemObject->request: 0
>> 2017/09/22 22:43:15 kid1| MemObject->logUri:
>> 2017/09/22 22:43:15 kid1| MemObject->storeId:
>> 2017/09/22 22:43:15 kid1| Bug: Missing MemObject::storeId value
>> 2017/09/22 22:43:15 kid1| mem_hdr: 0x7f16a0388760 nodes.start() 0x7f16a6a4a500
>> 2017/09/22 22:43:15 kid1| mem_hdr: 0x7f16a0388760 nodes.finish() 0x7f16a6a4a4d0
>> 2017/09/22 22:43:15 kid1| MemObject->start_ping: 0.000000
>> 2017/09/22 22:43:15 kid1| MemObject->inmem_hi: 50265
>> 2017/09/22 22:43:15 kid1| MemObject->inmem_lo: 0
>> 2017/09/22 22:43:15 kid1| MemObject->nclients: 0
>> 2017/09/22 22:43:15 kid1| MemObject->reply: 0x7f169f83d7d0
>> 2017/09/22 22:43:15 kid1| MemObject->request: 0
>> 2017/09/22 22:43:15 kid1| MemObject->logUri:
>> 2017/09/22 22:43:15 kid1| MemObject->storeId:
>>
>> I did some googling and seems like a lot of comments about this with
>> Rock (we're using) and ICP/HTCP (not using).  Curious if this the same
>> bug or something new?  Are there config changes we can make to prevent
>> this (perhaps switching away from rock cache??)
>>
>> We have a bunch of clients behind haproxy which is load balancing to
>> 4x Squid.  Config of the squids is as:
>>
>> http_access allow localhost manager
>> http_access deny manager
>>
>> external_acl_type client_ip_map_0 %>ha{Our-Client}
>> /usr/lib64/squid/user_loadbalance.py 0 4
>> external_acl_type client_ip_map_1 %>ha{Our-Client}
>> /usr/lib64/squid/user_loadbalance.py 1 4
>> external_acl_type client_ip_map_2 %>ha{Our-Client}
>> /usr/lib64/squid/user_loadbalance.py 2 4
>> external_acl_type client_ip_map_3 %>ha{Our-Client}
>> /usr/lib64/squid/user_loadbalance.py 3 4
>>
>> acl client_group_0 external client_ip_map_0
>> acl client_group_1 external client_ip_map_1
>> acl client_group_2 external client_ip_map_2
>> acl client_group_3 external client_ip_map_3
>>
>> http_access allow client_group_0
>> http_access allow client_group_1
>> http_access allow client_group_2
>> http_access allow client_group_3
>> http_access deny all
>>
>> tcp_outgoing_address 10.93.2.41 client_group_0
>> tcp_outgoing_address 10.93.2.76 client_group_1
>> tcp_outgoing_address 10.93.2.198 client_group_2
>> tcp_outgoing_address 10.93.3.178 client_group_3
>>
>> cache_dir rock /var/lib/squid/cache1 51200
>> cache_dir rock /var/lib/squid/cache2 51200
>> coredump_dir /var/spool/squid
>> maximum_object_size_in_memory 8 MB
>> maximum_object_size 8 MB
>>
>> cache_mem 6 GB
>> memory_cache_shared on
>> workers 4
>>
>> refresh_pattern . 0 100% 30
>>
>> http_port squid0001:3128 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=400MB cert=/etc/squid/ssl_cert/myCA.pem
>> http_port localhost:3128
>> ssl_bump bump all
>>
>> request_header_access Our-Client deny all
>> request_header_access Via deny all
>> forwarded_for delete
>>
>> visible_hostname squid0001.lab.company.com
>> logformat adttest %tg %6tr %>a %Ss/%03>Hs %<st %rm %>ru %[un %Sh/%<a %mt %ea
>> access_log daemon:/var/log/squid/access.${process_number}.log adttest
>> icon_directory /usr/share/squid/icons
>>
>> sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
>> sslcrtd_children 32 startup=2 idle=2
>> sslproxy_session_cache_size 100 MB
>> sslproxy_cert_error allow all
>> sslproxy_flags DONT_VERIFY_PEER
>>
>>
>> --
>> Aaron Turner
>> https://synfin.net/         Twitter: @synfinatic
>> My father once told me that respect for the truth comes close to being
>> the basis for all morality.  "Something cannot emerge from nothing,"
>> he said.  This is profound thinking if you understand how unstable
>> "the truth" can be.  -- Frank Herbert, Dune
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>