[squid-users] CPU Load 100% after implementing SSL Bump ....

[Amos Jeffries]

On 23/05/2016 6:27 p.m., Sagar Malve wrote:
> Hi Team,
> 
> System Config:
> 
> Intel S2400SC2 Motherboard
> Intel Xeon ES 2407 V2 CPU
> RAM 32 GB
> 

What Squid version?

> 
> http_port 3127
> http_port 3128 intercept
> https_port 3129 intercept ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_certs/squid.crt
> key=/etc/squid/ssl_certs/squid.key options=NO_SSLv3
> tls-dh=/etc/squid/dhparam.pem
> sslproxy_capath /etc/ssl/certs
> 
> 
> # FILTERING HTTPS
> acl 1 dstdomain .fbcdn.net .akamaihd.net .fbsbx.com
> #acl 2a dstdomain .mahadana.com .mql4.com .metaquotes.net
> acl 2 url_regex -i ^https?:\/\/attachment\.fbsbx\.com\/.*\?(id=[0-9]*).*
> acl 2 url_regex -i
> \.fbsbx\.com\/.*\/(.*\.(unity3d|pak|zip|exe|dll|jpg|png|gif|swf)/)$
> acl 2 url_regex -i ^https?:\/\/.*\.ytimg\.com(.*\.(webp|jpg|gif))
> acl 2 url_regex -i ^https?:\/\/([^\.]*)\.yimg\.com\/(.*)
> acl 2 url_regex -i ^https?:\/\/.*\.gstatic\.com\/images\?q=tbn\:(.*)
> acl 2 url_regex -i
> ^https?:\/\/.*\.reverbnation\.com\/.*\/(ec_stream_song|download_song_direct|stream_song)\/([0-9]*).*
> acl 2 url_regex -i
> ^https?:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|.exoclick\.com|interclick.\com|\.googlesyndication\.com|\.auditude\.com|.visiblemeasures\.com|yieldmanager|cpxinteractive)(.*)
> acl 2 url_regex -i ^https?:\/\/(.*?)\/(ads)\?(.*?)
> acl 2 url_regex -i ^https?:\/\/.*steampowered\.com\/.*\/([0-9]+\/(.*))
> acl 3 url_regex -i
> ^https?:\/\/(.*?)\/speedtest\/.*\.(jpg|txt|png|gif|swf)\?.*
> acl 3 url_regex -i speedtest\/.*\.(jpg|txt|png|gif|swf)\?.*
> acl 4 url_regex -i reverbnation.*audio_player.*ec_stream_song.*$
> acl 5 url_regex -i utm.gif.*
> acl 6 url_regex -i c.android.clients.google.com.market.GetBinary.GetBinary.*
> acl 7 url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204).*$
> acl 7 url_regex -i
> \.c\.(youtube|google)\.com\/(get_video|videoplayback|videoplay).*$
> acl 7 url_regex -i (youtube|google).*\/videoplayback\?.*
> acl 8 http_status 302
> acl getmethod method GET
> 

Using .* on the beginning or end of a regex does nothing but cause more
CPU workload for Squid.

If you put it inside (.*), or with an anchor ^.* or .*$ just makes the
CPU usage worse.

What http_access rules are using those?

> 
> ssl_bump splice localhost
> acl 9 at_step SslBump1
> acl 10 at_step SslBump2
> acl 11 at_step SslBump3
> ssl_bump peek 9 all
> ssl_bump bump 10 all
> ssl_bump bump 11 all

Step3 of bumping process will never happen. You told Squid to begin
decryption at step2.

Have you disabled "via"?


> 
> ----------------------------------------------------------------------------------------------
> 
> Is there any way where it can Cache SSL Certificate for all HTTPS Traffic
> ....
> Because SSL Cert & Squid process were using 99% of CPU Load ....

Er, what do you think caching does exactly?

Caching HTTPS will have no effect on your described CPU problem. Might
make it worse even.


Between them?

How much is each process using?

How may concurrent connections are being handled by Squid to get that
loading ?


Check whether Squid is finished loading its cache_dir indexes, or if any
of them are undergoing a "DIRTY" rebuild. That can use a lot of CPU
while its happening and caching cannot be fully operational until its
finished either.


> 
> We have approx 200 users ....
> 
> I have set the open file limit to 100000

FYI: SSL-Bump with your configuration will use 3 FD for each client
inbound HTTPS request. That 100K limit will restrict your users to 150
concurrent connections each.
A browser using Happy eyeballs will open 16 connections to each domain.
Average web page on the most popular sites involve around 100 objects
spread over 10+ domains.
  => ~160 FD needed to load an average page.

I'd double that limit, if you expect this proxy to have much traffic.

> 
> Could you please let us know if there is any way to Cache the HTTPS Request
> in Squid .....
> 

You are already SSL-Bumping traffic. That removes the 'S' from HTTPS.
Leaving Squid with regular HTTP messages, which already are cached if it
can.

Amos