[squid-users] Cache Peers and traffic handling
Alex Rousskov
rousskov at measurement-factory.com
Wed Apr 14 21:09:07 UTC 2021
On 4/14/21 2:49 PM, koshik moshik wrote:
> First of all thank you for trying to help me. Let me describe my current
> issue: I have 5000 proxies and would like to hide them. My plan was
> using another proxy server with 5000 cache peers and 5000 users. Each
> user would get one peer and one proxy attached to that peer. So
> basically the outer world would not see my "main proxy" and only the one
> from the new proxy server.
> Is there any better solution than cache peers for that?
Probably. It sounds like you need dynamic HTTP proxy routing (where the
first proxy computes the address of the second proxy/hop from the user
name rather than selects from a huge set of fixed cache_peers using a
huge set of fixed ACLs). AFAIK, Squid does not support such routing
without source code modifications. Even ICAP/eCAP cannot do that alone.
Alex.
> On Wed, Apr 14, 2021 at 8:37 PM Alex Rousskov wrote:
>
> On 4/14/21 2:29 AM, koshik moshik wrote:
> > Thank you! Yes, it works fine with 5 peers. So, what would be the best
> > solution to handle 5000 peers?
>
> As you can tell by other responses, you might be asking the wrong
> question. However, I will still try to answer your question. The best
> option may be to add support for a new Squid configuration parameter
> that tells Squid to limit cache_peer candidate accumulation to N peers,
> effectively making all those linear searches fast.
>
> https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
> <https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F>
>
> Alex.
>
>
> > On Mon, Apr 12, 2021 at 6:03 PM Alex Rousskov wrote:
> >
> > On 4/10/21 5:03 PM, koshik moshik wrote:
> >
> > > I am trying to run a Squid proxy Server witth about 5000 cache
> > peers. I
> > > am running a dedicated server with 6 cores and 32GB RAM on
> Ubuntu 16.
> > >
> > >
> > > Could you tell me what else is needed / not needed in my
> > squid.config? I
> > > am encountering a high CPU usage and would like to create a very
> > > efficient proxy server.
> >
> > IIRC, Squid code is not optimized for handling a large number of
> > cache_peers: Several cache peer selection steps involve linear
> searches.
> >
> > I do not know what exactly causes high CPU usage in your
> environment but
> > it could be those linear searches. You can test that
> (indirectly) by
> > decreasing the number of cache_peers from 5000 to, say, 5.
> That is a
> > weak test, of course, because other cache_peer-related
> overheads could
> > be to blame, but I would start there.
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> >
> > > Down below you can find my squid.config(I deleted the other
> cache_peer
> > > lines):
> > >
> > > -----------
> > >
> > > http_port 3128
> > >
> > > dns_v4_first on
> > >
> > > acl SSL_ports port 1-65535
> > >
> > > acl Safe_ports port 1-65535
> > >
> > > acl CONNECT method CONNECT
> > >
> > > http_access deny !Safe_ports
> > >
> > > http_access deny CONNECT !SSL_ports
> > >
> > > auth_param basic program /usr/lib/squid/basic_ncsa_auth
> > /etc/squid/.htpasswd
> > >
> > > auth_param basic children 5
> > >
> > > auth_param basic realm Squid Basic Authentication
> > >
> > > auth_param basic credentialsttl 5 hours
> > >
> > > acl password proxy_auth REQUIRED
> > >
> > > http_access allow password
> > >
> > > #http_access deny all
> > >
> > > cache allow all
> > >
> > > never_direct allow all
> > >
> > > ident_access deny all
> > >
> > >
> > >
> > >
> > >
> > > cache_mem 1 GB
> > >
> > > maximum_object_size_in_memory 16 MB
> > >
> > >
> > >
> > >
> > >
> > > # Leave coredumps in the first cache dir
> > >
> > > coredump_dir /var/spool/squid
> > >
> > >
> > > #Rules to anonymize http headers
> > >
> > > forwarded_for off
> > >
> > > request_header_access Allow allow all
> > >
> > > request_header_access Authorization allow all
> > >
> > > request_header_access WWW-Authenticate allow all
> > >
> > > request_header_access Proxy-Authorization allow all
> > >
> > > request_header_access Proxy-Authenticate allow all
> > >
> > > request_header_access Cache-Control allow all
> > >
> > > request_header_access Content-Encoding allow all
> > >
> > > request_header_access Content-Length allow all
> > >
> > > request_header_access Content-Type allow all
> > >
> > > request_header_access Date allow all
> > >
> > > request_header_access Expires allow all
> > >
> > > request_header_access Host allow all
> > >
> > > request_header_access If-Modified-Since allow all
> > >
> > > request_header_access Last-Modified allow all
> > >
> > > request_header_access Location allow all
> > >
> > > request_header_access Pragma allow all
> > >
> > > request_header_access Accept allow all
> > >
> > > request_header_access Accept-Charset allow all
> > >
> > > request_header_access Accept-Encoding allow all
> > >
> > > request_header_access Accept-Language allow all
> > >
> > > request_header_access Content-Language allow all
> > >
> > > request_header_access Mime-Version allow all
> > >
> > > request_header_access Retry-After allow all
> > >
> > > request_header_access Title allow all
> > >
> > > request_header_access Connection allow all
> > >
> > > request_header_access Proxy-Connection allow all
> > >
> > > request_header_access User-Agent allow all
> > >
> > > request_header_access Cookie allow all
> > >
> > > request_header_access All deny all
> > >
> > >
> > >
> > >
> > >
> > > #
> > >
> > > # Add any of your own refresh_pattern entries above these.
> > >
> > > #
> > >
> > > #refresh_pattern ^ftp: 1440 20% 10080
> > >
> > > #refresh_pattern ^gopher: 1440 0% 1440
> > >
> > > #refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> > >
> > > #refresh_pattern (Release|Packages(.gz)*)$ 0 20%
> 2880
> > >
> > > #refresh_pattern . 0 20% 4320
> > >
> > >
> > > ################################
> > >
> > > acl me proxy_auth ye-1
> > >
> > > cache_peer my.proxy.com <http://my.proxy.com>
> <http://my.proxy.com <http://my.proxy.com>>
> > <http://my.proxy.com/ <http://my.proxy.com/>
> <http://my.proxy.com/ <http://my.proxy.com/>>> parent 31280
> > > login=user1:password1 no-query name=a1
> > >
> > > cache_peer_access a1 allow me
> > >
> > > cache_peer_access a1 deny all
> > >
> > >
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> > >
> >
>
More information about the squid-users
mailing list