[squid-users] squid logging disable based on ACL & kernel: Out of memory

[Alex Rousskov]

On 5/6/20 8:58 AM, Akshay Hegde wrote:

> 1. Is there any way to filter HTTPS URLs without importing CA
> certificates on client side?

No, there is no way for a proxy to look at request URLs without the
browser trusting the proxy certificate. There are other ways to police
traffic (e.g., browser plugins), but they all require fiddling with the
client environment.


> 2. for 16GB RAM, 4 core CPU, 8GB Swap, expected to have 10GB cache,  how
> to calculate configurations parameters, is there any thumb rule ?

I believe there is some related advice on Squid wiki:
https://wiki.squid-cache.org/SquidFaq/SquidMemory

HTH,

Alex.


> # config
> cache_mgr webmaster
> cache deny QUERY
> cache_mem 256 MB
> cache_swap_low 90
> cache_swap_high 95
> maximum_object_size 4 MB
> minimum_object_size 0 KB
> maximum_object_size_in_memory 512 kB
> ipcache_size 2048
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 1024
> cache_replacement_policy lru
> memory_replacement_policy lru
> cache_dir ufs /var/spool/squid 10000 16 256
> cache_effective_user squid
> cache_effective_group squid
> cache_log /var/log/squid/cache.log
> cache_store_log /var/log/squid/store.log
> memory_pools on
> memory_pools_limit 5 MB
> 
> # SSL-Bump -working but not feasible.
> http_port 3128 ssl-bump cert=/etc/squid/sslcert/proxyCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> sslcrtd_program /usr/lib64/squid/security_file_certgen -s
>  /var/spool/squid/ssl_db -M 4MB
> sslcrtd_children 5
> acl step1 at_step SslBump1
> ssl_bump peek step1
> ssl_bump bump all
> 
> ------------------------------------ My New Environment --------------------
> # squid -v
> Squid Cache: Version 4.4
> Service Name: squid
> 
> # cat /etc/redhat-release
> CentOS Linux release 8.1.1911 (Core)
> 
> 
> # Tested ACLs
> logformat test_log %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %>ru %[un
> %Sh/%<a %mt
> acl test_sites dstdomain "/etc/squid/acls/test_sites.acl"
> access_log /var/log/squid/test_site.log test_log test_sites
> 
> # tail -f /var/log/squid/test_site.log
> 1588678050.178   3247 10.0.2.15 TCP_TUNNEL/200 28073 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588678050.189   3942 10.0.2.15 TCP_TUNNEL/200 24000 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588678050.355   2552 10.0.2.15 TCP_TUNNEL/200 788 CONNECT
> nav.sciencedirect.com:443 <http://nav.sciencedirect.com:443> akshay
> HIER_DIRECT/91.235.133.74 <http://91.235.133.74> -
> 1588681419.635    647 10.0.2.15 TCP_MISS/200 402 POST
> http://scratchpads.eu/modules/statistics/statistics.php akshay
> HIER_DIRECT/157.140.2.32 <http://157.140.2.32> text/html
> 1588681420.055   1069 10.0.2.15 TCP_MISS/200 46772 GET
> http://scratchpads.eu/sites/all/themes/scratchpads_eu/images/shrimp-202px.png
> akshay HIER_DIRECT/157.140.2.32 <http://157.140.2.32> image/png
> 
> 
> 
> 
> On Sat, May 2, 2020 at 1:00 AM Alex Rousskov
> <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>> wrote:
> 
>     On 5/1/20 12:43 PM, Akshay Hegde wrote:
> 
>     > I have below option globally, which I don't want to make "off"
>     > strip_query_terms on
> 
>     > acl track dstdomain "/etc/squid/sites_track.txt"
>     > access_log /var/log/squid/full_site_links.log squid_custom track
> 
>     > however for specific ACL I would like to log full URL with query
>     > parameters, how this can be done ?
> 
>     I have not tested this, and the results may be version-dependent, but
>     according to logformat documentation[1], %ru honors strip_query_terms
>     while %>ru does not:
> 
>         logformat strippedFormat %ts... %ru ...
>         access_log ... strippedFormat track !specific_ACL
> 
>         logformat detailedFormat %ts... %>ru ...
>         access_log ... detailedFormat track specific_ACL
> 
>     [1] http://www.squid-cache.org/Doc/config/logformat/
> 
> 
>     HTH,
> 
>     Alex.
> 
>     > On Fri, May 1, 2020 at 7:05 PM Alex Rousskov wrote:
>     >
>     >     On 5/1/20 1:20 AM, Akshay Hegde wrote:
>     >
>     >     > *1. How to disable logging of few ACLs ?
>     >
>     >     Use "access_log none aclX" to prevent creation of access.log
>     records for
>     >     transactions matching aclX. See
>     >   
>      http://lists.squid-cache.org/pipermail/squid-users/2020-April/021876.html
>     >     for
>     >     some related caveats.
>     >
>     >
>     >     > *2. Kernel Out of Memory
>     >
>     >     This problem is most likely unrelated to logging. If your Squid is
>     >     gradually leaking memory (rather than just being overwhelmed with
>     >     traffic), then the first step towards removing those memory
>     leaks would
>     >     be to upgrade your Squid from the unsupported and buggy v3.1.10.
>     >