[squid-users] very slow squid response

[Amos Jeffries]

On 19/09/17 23:54, Iraj Norouzi wrote:
> hi Antony
> thanks for you reply
>> i setup squid on ubuntu and centos
> 
> Why both?
> because of test and because i not get the result
> 
>> with tproxy and wccp for 6 gb/s traffic
> 
> What hardware are you using for that sort of traffic flow?
> i use hp DL360 with 2 6 core processor with 3 GHZ and 64 GB RAM and 1 TB HDD
> 
>> but when i try to test squid with 40 mb/s traffic
> 
> How are you generating "40 mb/s traffic"?  I'm assuming that your Internet
> connection is 6Gbps as stated above, so how are you restricting this down to
> 40Mbps for testing?
> i redirect one class of ip address with 40 mb/s traffic for test of 
> squid and i am going decide to redirect whole of traffic to squid after 
> getting fast browsing
> 

Squid is designed to optimize and reduce *bandwidth*. "fast browsing" is 
just a nice side effect of caching. It is a mistake to think that Squid 
will always produce faster browsing.

This is especially true during the initial cache warm-up period where 
DNS and HTTP objects are being fetched for the first time. Speed can 
only come from future fetches being reduced by the cache.

So when you are testing for speed with real traffic make sure there has 
been at least a few hrs for the caches to warm up.

Since you are testing with RAM-only caching right now be aware that 
every time you restart Squid *all* its caches (for all data types) get 
erased back to "cold"/empty.


>> it response very slow
> 
> Numbers please.
> websites load at 2 or 1 second by direct browsing and load at 10 second 
> or not load by squid
> 
>> while when i use direct browsing i can browse websites very fast
> 
> Is the direct traffic still being routed through the Squid server (you say
> you're using tproxy, so I assume this is an intercept machine with the 
> traffic
> going through it between client and server)?
> no, HTTP traffic redirect to squid by wccp and access-list config on Cisco

Cisco is tunneling the packets with WCCP to the Squid machine, which is 
intercepting the traffic with TPROXY.

So actually "yes", if not something would be terribly broken in your 
WCCP and TPROXY setup.


> ip wccp 80 redirect-list wccp
> ip wccp 90 redirect-list wccp_to_inside
> 
> ip access-list extended wccp
>   remark Permit http access from clients
>   permit tcp x.x.x.x 0.0.0.255 any eq www
>   deny   ip any any
> ip access-list extended wccp_to_inside
>   permit tcp any eq www x.x.x.x 0.0.0.255
>   deny   ip any any
> 
>> i used tcpdump for tracing connections arrive time and there was no problem,
> 
> Arrival time where?  From the origin server to Squid?  From Squid to the
> client?  What are you actually measuring?
> yes, arriving source packets from clients to squid interface,by time 
> that i push enter on browser address bar and getting packets on tcpdump 
> immediately

Packets arriving at Squid from the client is only the first ~1% of 
things that are going on. It would be a big problem if they took 
anything more than a few ms to arrive.

Once the packets arrive there are DNS lookups to do. Delay in DNS 
lookups is the most common cause of overall delays.

Then there is HTTP processing to find a source for the response. If the 
request has never been seen before that means a fair amount of logic to 
select potential upstream servers and attempt connections to them (maybe 
several or even all of them).

Then the server request has to be generated, and wait for a response.
Only when that server response comes back can stuff for the client 
response start to happen.

When the cache is involved the total time could be under 1ms, or 
somewhat around 50ms. If there are lots of server things to do the time 
can be hundreds of ms.


> 
>> i used watch -d for tracing packets match by iptables rules and it was ok,
> 
> Please be more specific - what did you measure and what does "OK" mean?
> i add rule to iptables for tracing one website packets and i saw them on 
> kern.log that matched with the rule, so if you need please tell me to 
> send commands that i used.

Which packets. As I detailed above, there are a minimum of 2 TCP 
connections involved with delivering a MISS object (client->Squid, and 
Squid->server) potentially many more if there are network issues 
connecting to server(s).


> 
> ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 
> <http://0.0.0.0/0> dev enp3s0f0 table 100
> 
> iptables -t mangle -A PREROUTING -d $LOCALIP -j ACCEPT

You also need a rule before the above one which blocks traffic to
   "-d $LOCALIP -p 3129 -j REJECT"

> iptables -t 
> mangle -N DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 
> iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING 
> -p tcp -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp 
> --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129
> 
> watch -d iptables -t mangle -vnL
> 
> Did you compare with and without Squid in place to see what differs?
> no, as i told when i browsing directly it is well and packets not coming 
> to squid and exit from cisco to internet and arrive to clients from 
> cisco and because of traffic on cisco i can't enable debugging on it but 
> when i browse from squid i get latency so i suppose the problem is squid 
> or server that squid running on it

Your test is incomplete.

To eliminate WCCP and routing issues being the cause of problems you 
need to compare:

A) normal browsing without WCCP  or TPROXY.

B) normal browsing with WCCP tunneling traffic to the Squid machine (no 
Squid running and no TPROXY).

C) normal browsing with WCCP tunneling traffic to the Squid machine 
where TPROXY diverts the traffic into Squid.

If things work any slower at test (B) than test (A) your problem is the 
WCCP + Squid machine setup, not Squid itself.


> 
>> i also used iptables trace command for tracing matching iptables rules,
>> there was no problem except i had latency on arriving packets on iptables
>> rule while tcpdump captured packets fast, it happened when my browsing was
>> so slow, at some times that my browsing was fast there was no latency on
>> iptables trace log.
> 
> That description is too vague to know exactly what you were measuring 
> and what
> results you got.
> 
> iptables -t raw -A PREROUTING -s x.x.x.x -j TRACE
> 
> iptables -t raw -A OUTPUT -s x.x.x.x -j TRACE
> 
> tailf /var/log/kern.log
> 
> tcpdump -e -i enp3s0f0 -d x.x.x.x dst port 80
> 
> tcpdump -e -i enp3s0f0 -s x.x.x.x src port 80
> 

"and what results you got" ?


>> i also used tcp and linux enhancement configurations
> 
> Details?
> net.core.wmem_default=524288
> net.core.wmem_max=16777216
> net.core.rmem_default=524288
> net.core.rmem_max=16777216
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_rmem = 66560 524288 16777216
> net.ipv4.tcp_wmem = 66560 524288 16777216
> net.core.somaxconn=4000
> net.ipv4.tcp_timestamps=0
> net.ipv4.tcp_sack=0
> net.ipv4.tcp_fin_timeout=20
> net.ipv4.ip_local_port_range=10240 65000
> net.ipv4.tcp_keepalive_time = 900
> net.ipv4.tcp_keepalive_intvl = 900
> net.ipv4.tcp_keepalive_probes = 9
> net.core.somaxconn = 5000
> net.core.netdev_max_backlog = 8000
> net.ipv4.tcp_max_syn_backlog = 8096
> net.ipv4.tcp_slow_start_after_idle = 0
> net.ipv4.tcp_tw_reuse = 1
> 


Okay, so no improvement with those means they were not useful, please 
ensure they are back at defaults so as not to add problems to other 
experimental changes.


>> but nothing happened.
>> wccp send packets very well and tcpdump show capturing packets too but
>> browsing with squid is very slow.
> 
> Firstly, please define "slow" - do you mean it takes a long time for new web
> pages / images / etc to appear (but once they start, they arrive quickly)

 >> Iraj:
> browse in 10 second or not browsing, webpages that i browse for first 
> time or i browse for multiple times

For that sort of delay:

* check the DNS response times for lookups made by the Squid machine. 
The details of Squid's lookups can be found in the "idns" cache manager 
report.

*


 >> Antony:
> , or do you mean that a continuous stream of data (a "download") arrives 
> more
> slowly when going through Squid than going direct (and if so, what are the
> different speeds)?

 >> Iraj:
> no just browsing,
> 

 >> Antony:
> Secondly, what are you trying to achieve with Squid - what is its purpose in
> your network?

 >> Iraj:
> caching pages and their objects for better internet browsing by clients 
> and save bandwith.
> 
> squid.conf
> dns_v4_first on
> acl fanava_net src x.x.x.x/32
> acl Safe_ports port 80# http
> acl Safe_ports port 280# http-mgmt
> acl Safe_ports port 488# gss-http
> acl Safe_ports port 777# multiling http
> acl CONNECT method CONNECT
> http_access allow localhost manager
> http_access deny manager
> http_access allow localhost
> http_access allow fanava_net
> http_access deny all
> http_port 0.0.0.0:3128
> http_port 0.0.0.0:3129 tproxy
> cache_mem 50 GB

Does this machine actually have at least 52 GB of free RAM for Squid to use?

> maximum_object_size_in_memory 100 MB
> minimum_object_size 2 KB
> maximum_object_size 6 GB
> #cache_dir ufs /var/spool/squid 1024000 256 512
> cache_swap_low 90
> cache_swap_high 92
> coredump_dir /var/spool/squid
> # Image files
> refresh_pattern -i \.(png|gif|jpg|jpeg|bmp|tif|tiff)$ 10080 90% 43200
> # Compressed files
> refresh_pattern -i \.(zip|rar|tar|gz|tgz|z|arj|lha|lzh|iso|deb|rpm)$ 
> 10080 90% 43200
> # Binary files
> refresh_pattern -i \.(exe|msi)$ 10080 90% 43200
> # Multimedia files
> refresh_pattern -i 
> \.(mp3|wav|mid|midi|ram|avi|wmv|mpg|mpeg|mp4|swf|flv|)$ 10080 90% 43200
> # Document files
> refresh_pattern -i \.(pdf|ps|doc|ppt|xls|pps)$ 10080 90% 43200
> # HTML patterns
> refresh_pattern -i \.index.(html|htm)$ 0 40% 10080
> refresh_pattern -i \.default.(html|htm)$ 0 40% 10080
> refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320
> # Default patterns
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320  # Fanava First 
> clients range to cache
> refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880
> cache_mgr ******
> wccp2_router x.x.x.x
> wccp_version 2
> wccp2_rebuild_wait on
> wccp2_forwarding_method 2
> wccp2_return_method 2
> wccp2_service dynamic 80
> wccp2_service_info 80 protocol=tcp flags=src_ip_hash priority=240 ports=80
> wccp2_service dynamic 90
> wccp2_service_info 90 protocol=tcp flags=dst_ip_hash,ports_source 
> priority=240 ports=80
> 
> *Regards,
> Iraj Norouzi*
> 

Amos