[squid-users] Best practice for adding or removing ACLs dynamically ?

Sun Nov 8 01:08:03 UTC 2020

Following up.

Well, github and google is full of horrible ways of implementing the tcp_outgoing_address selection so..
I have written a sketch of the concept I was writing about based on ncsa_auth which I must admit is far more .. simple then using a DB like mysql or LDAP.

The config I found was using either:
- multiple inbound squid ports -> mapped each to a single tcp_outgoing_addrees
- multiple static usernames mapped to a specific address(based on either username or usernamer+dst-domain.. else ACLs types)
- Others which are not relevant since these break the internet

What I am offering is an example which offloads the actual selection code/algorithm and config into an external_acl_type helper.
The result would be that squid will only listen(when possible and required) only on one port.
(which means that adding more ports can multiply/power up the service config "capacity", like vlans ontop of vlans)
Also the authentication mechanism would be ncsa_auth which is good for up to 1000s++ of users since the helpers store the htaccess file In RAM.
Even if the file will contain 10k users and the auth helpers in use would be let say 40 and up, the proxy would still consume less Memory on the
storage of the htaccess file in RAM compared to a MySQL or LDAP DB.

To change an address for a specific existing user you can use a simple sed command:
sed -i -E "s at user80[\ \t]+[0-9]+ at user80 1 at g" /etc/squid/user-to-ip.txt

and to change password for a user you can use a simple htaccess command:
htpasswd -b /etc/squid/htpasswd  user80 1234

Within my external_acl_helper I have used a similar approach to the one the ncsa_auth helper uses to reload modified files from disk.
(If the mtime stamp is being changed from the old one the helper would reload the mapping config file from disk)
* https://github.com/elico/vagrant-squid-outgoing-addresses/blob/master/shared/note.rb

To demonstrate this piece of code I have used Vagrant on-top of Virtualbox.
(I have verified that on the current Windows 10 PRO installations both Virtualbox and Hyper-v can be used simultaneously with CPU Virtualization support ON)

The example lab is at:
https://github.com/elico/vagrant-squid-outgoing-addresses

The test is composed of two nodes:
* Squid
* Web

The Squid instance will be populated with more then 20 IP addresses on the same subnet.
For simplicity I have added a simple /32 address to the Squid+Web private/closed network interface.
There are other ways which can be used in production when Routing daemons are in the picture.

To start the demo/test you can use:
vagrant up
vagrant ssh squid --command /vagrant/test-random-ips-assignment.sh

The user to ip config mapping file is at:
/etc/squid/user-to-ip.txt

I have used a simple "<user> <tab or space> <ip integer id>" mapping syntax.

Technically speaking this setup can work with scales of 1000s and up of addresses and clients.
I am pretty sure it's good enough for proxy services which uses multiple routing tunnels and/or BGP feeds.

If anyone wants to ask about this setup I will be more than happy to receive emails about it here in the list.

All The Bests,

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd at gmail.com

-----Original Message-----
From: Eliezer Croitor <ngtech1ltd at gmail.com> 
Sent: Sunday, November 1, 2020 2:41 AM
To: squid-users at lists.squid-cache.org
Cc: 'roee klinger' <roeeklinger60 at gmail.com>
Subject: RE: [squid-users] Best practice for adding or removing ACLs dynamically ?

Hey Roee,

I am trying to understand what part of squid.conf you want to be able to change without a reconfigure/reload?
If you have many users, ie above 50 you should probably not use a simple ncsa_auth although  it's possible and in more then one case is preferable.
You could probably write your own basic auth helper that will interact with a DB which will probably simplify your whole setup.
(You can use existing basic auth helpers with mysql or ldap)

As for the tcp_outgoing_address, it’s a whole different story.
Since it's a "fast" acl type the options to do something dynamic with it are an issue.
(Maybe eCAP/ICAP service or a "pre-cooked" note or other factor to the acl can be used)

I am pretty sure that if an authentication service can reply with a note ie connection annotation then it can be used for the address selection.
One issue with it is that It will be valid for the next X ttl seconds/minutes/hours.

I do believe that there should be a way to allow something like external_acl helper to affect this squid feature.
I was thinking that an eCAP or an ICAP service or an external_acl helper can add a note for a connection based on couple other factors like:
* src ip
* auth username
* request domain or request sni
* ...

So let say the proxy will have a set of 100 addresses, each will have a single specific matching acl for a request header or connection annotation/note.
This way the selection of a tcp_outgoing_address would be a little less complex the it is today.

I have couple other ideas for implementations which I have experimented with but the proxy admin need to learn how these work which might be
a bit complicated some times.

Eliezer
----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd at gmail.com

-----Original Message-----
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of roee klinger
Sent: Saturday, October 31, 2020 2:35 AM
To: squid-users at lists.squid-cache.org
Subject: [squid-users] Best practice for adding or removing ACLs dynamically ?

Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to change it in squid.conf and also in htpassword and then I run "squid -k reconfigure", if I don't reconfigure the old user still has access to the proxy and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, and also I would like to blacklist websites often and on the fly, so I was searching for a better way to manage this without reconfiguring every time, since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this currently or there is a better way to change users/ACLs on the fly without reloading Squid?

Thanks,
Roee Klinger
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users