[squid-users] squid with quota limit using external helper problem !

[Amos Jeffries]

On 04/09/17 07:49, --Ahmad-- wrote:
> Hello squid folks .
> 
> I’m trying to use squid external helper to get quote to ips or users.
> 
> I’m following the wiki :
> 
> http://www.mikealeonetti.com/wiki/index.php?title=Squid_Arms_and_Tentacles:_Bandwidth_quotas
> 
> i have done everything my side on squid .
> 
> i have tested the connection :
> root at localhost:~# /usr/local/bin/bandwidth_calculate 
> /etc/squid/bandwidth_rules
> root at localhost:~#
> 
> no errors above !
> 
> #######################################
> 
> the issue I’m not sure if I’m using squid config file integration 
> correctly .
> 
> here is my squid.conf file :
> 
> dns_v4_first on
> acl localnet src all

You have defined your LAN to be the entire Internet. Don't do that.

Define localnet to be your actual network ranges.

Use the provided 'all' ACL to refer to things that are allowed/denied to 
everyone online. Most of the time 'all' is unnecessary.

If you expect clients from the general web to access your proxy and some 
access control to apply to them, then simply do not limit those access 
controls with the 'localnet' ACL.


> auth_param basic program /lib/squid/basic_ncsa_auth  /etc/squid/squid_user
> acl ncsa_users proxy_auth REQUIRED
> auth_param basic children 1000

How many users do expect exactly?

Squid de-duplicated overlapping Basic auth logins so one user can login 
multiple times at once (ie login bursts when a Browser starts up) with 
only one query sent to the auth helper. NCSA is also extremely fast lookups.

If you bumped that up because of the WARNING logged, then please change 
your practices to fix ERRORs before WARNINGs.
* WARNINGs are logged for things Squid can workaround but needs help to 
fix properly,
* ERRORs are things Squid cannot do anything about and need your attention,
* FATALs are things that are absolutely critical to fix if you are going 
to use Squid at all.


> external_acl_type bandwidth_check ttl=60 %SRC /usr/local/bin/bandwidth_check

The ttl= parameter needs to be 0 for accurate bandwidth results. With 
the above the helper is only checked once per minute, not on every request.
Keep in mind that you are only controlling whether new requests can 
start, and once started they will complete. So regular re-checking is 
required to minimize overages.

NP: negative_ttl= control how often Squid re-checks results from the 
helper once users go over their quota. This is the option that you will 
want to tune with non-0 values to reduce helper load, but also keep it 
low enough not to block clients for too long after their quota renews.


> acl bandwidth_auth external bandwidth_check
> http_access allow localnet bandwidth_auth
> http_access deny  localnet !bandwidth_auth

The wiki is documenting the above two rules as *alternatives*. I suggest 
you go back and read their descriptions, then pick the one that does 
what you need.


> ###################################################
> cache_effective_user squid
> cache_effective_group squid
> ###########################################
> http_access allow ncsa_users

This will only login users that broadcast their credentials. It will not 
require credentials from clients, and none of your below rules require 
login to have happened.

Best practice for authentication is to place the rules applying to 
non-authenticate clients first, then have:

   http_access deny !ncsa_users

... then to follow that with any rules applying to authenticated clients.


> ############################
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443         # https
> acl Safe_ports port 70          # gopher
> acl Safe_ports port 210         # wais
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280         # http-mgmt
> acl Safe_ports port 488         # gss-http
> acl Safe_ports port 591         # filemaker
> acl Safe_ports port 777         # multiling http
> acl CONNECT method CONNECT
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports

These Safe_ports and CONNECT rule need to be *above* all of your custom 
rules. Otherwise they will have zero ability to protect your proxy 
against the DoS and hijacking attacks they are supposed to prevent.

<snip>
> 
> here is errors i get :
> 
> 
> 2017/09/03 19:32:38 kid1| WARNING: external ACL 'bandwidth_check' queue 
> overload. Request rejected '11.13.209.12'.
> 2017/09/03 19:38:31 kid1| WARNING: external ACL 'bandwidth_check' queue 
> overload. Request rejected '11.13.209.12'.
> 2017/09/03 19:44:46 kid1| WARNING: external ACL 'bandwidth_check' queue 
> overload. Request rejected '148.161.111.42'.
> 2017/09/03 19:44:47 kid1| WARNING: external ACL 'bandwidth_check' queue 
> overload. Request rejected '148.161.111.42’.
> 
> 
> 
> but I’m sure 100 % that the ips above not blacklisted bec i check them 
> over the helper :

Please re-read the WARNING message.

IPs are *not* being rejected because they are listed. They are being 
rejected because the helper lookup queue is overloaded and no OK is 
received.

> 
> here is squid when it run :
> 
> root at localhost:~# tailf /var/log/squid/cache.log
> 2017/09/03 19:32:33 kid1| ERROR: Failed to create helper child read FD: 
> TCP [::1]

Fix that ERROR. The WARNING's about the helper and ACL checking are all 
side effects of there not actually being a helper running.

There are several ways to do that:

1) fix the helpers IPv6 support. It seems not to have any, or if it does 
is somehow still only using the IPv4-only address of localhost. Squid is 
trying to contact it over an IPv6-v4-mapped address for localhost.


2) add the 'ipv4' option to your external_acl_type, to make Squid 
temporarily be IPv4-only when talking to this helper.

While (2) is very tempting and easy, you will probably find that an 
IPv4-only helper like this has errors when it gets told the IP address 
of an IPv6 client. So (1) is the better option and I see the wiki page 
author goes on about being happy to fix problem with their helper - just 
get in touch.


Amos