[squid-users] allow request to cloudfront after 302 redirection.

Thu May 27 12:16:34 UTC 2021

Thanks Alex,

I assume you do remember that 301 is not the same as 302.
Depends on the status code ie 301/302 307/307 etc the ttl of the cached location would be decides.
The basic idea is to use either redis or memcached  on another key-value DB to persist the Location response.
(This key-value DB would only be a "cache" with a ttl)

For a 301 the basic assumption is that the Location header contains the static/permanent redirection.
For a 302 it's not a static, I think it's important to have the status code sent to the helper.

About the http_reply_access, I actually forgot it exists and indeed it makes more sense...

I am working on an example.
Eliezer

-----Original Message-----
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Alex Rousskov
Sent: Monday, May 24, 2021 9:26 PM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] allow request to cloudfront after 302 redirection.

On 5/24/21 5:52 AM, Eliezer Croitoru wrote:

> Following up this thread I was wondering about an example how to do
> that with an external_acl helper. With ICAP I can do that easily to
> some degree. With an external_acl helper I am not sure what values to
> send.

AFAICT, the external ACL helper(s) should be sent the Location URI
values the helper(s) should cache (at response time) and the request URI
values the helper(s) should examine (at request time). More information
may be sent/cached, of course, depending on the exact decision logic.

> I would guess that the response code and response Location header
> might be the ones which should be passed to the helper?

I do not see the value in sending the response status code (because
Squid can check that itself), but the correct answer depends on the
exact decision logic.

> What do you think about the next acls, should do the trick?

> acl redirect http_status 301-308
> acl gitlab_package dstdomain package.gitlab.com

> external_acl_type openlocation children=15 %DST %SRC %<{Location} /usr/local/bin/location-openner.rb
> acl location_openner external openlocation
> http_access deny gitlab_package redirect location_openner
> http_access allow location_openner

The above sketch does not make sense to me because it uses response
information (e.g., %<h{Location}) in a request-time directive (i.e.
http_access). FWIW, I would expect something along these lines:

  http_access allow toPreviouslySeenLocation

  http_reply_access allow isRedirect storeSeenLocation
  http_reply_access allow all

... where toPreviouslySeenLocation and storeSeenLocation are external
ACLs (using one shared or two dedicated external ACL helpers).
storeSeenLocation always matches and is used for its side effect.

HTH,

Alex.

> -----Original Message-----
> From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Alex Rousskov
> Sent: Wednesday, April 21, 2021 8:49 PM
> To: squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] allow request to cloudfront after 302 redirection.
> 
> On 4/21/21 12:48 PM, Miroslaw Malinowski wrote:
>> Is it possible to create a whitelist that allows cloudfront 302
>> redirections, e.g. gitlab is using cloudfront as CDN and when we
>> whitelist package.gitlab.com the URL is redirected (302) to
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c15d
> 
> 
> Yes, it is possible to allow future requests to Location-listed URLs,
> but since we are talking about two (or more) independent HTTP
> transactions, on two (or more) TCP connections, you will need to store
> the allowed Location values (at least) somewhere, maintain that storage
> (e.g., remove stale entries), and (optionally) determine whether the
> request for an allowed cloudfront URL came from the same user agent as
> the gitlab request that was redirected to that URL.
> 
> Storing, maintenance, and checking of allowed Locations/etc. can be done
> using external ACLs and/or eCAP/ICAP adaptation services. It cannot be
> reliably done using built-in ACLs alone AFAICT.
> 
> 
> HTH,
> 
> Alex.
> 
> 
>> I could whitelist a whole .cloudfront.net <http://cloudfront.net> domain
>> or url_regex, but what I would like to achieve, I don't know if
>> possible, is a chain of events like:
>> If packages.gitlab.com <http://packages.gitlab.com> return 302 Location
>> .cloudfront, then allow
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c
>> <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb?t=1619023239_a63698472b6bebeaee980e7c030632d97a29c>
>> request.
>> I've been playing around with http_reply_access and rep_headers, but I
>> can only go as far as allow replay of the first request to
>> package.gitlab.com <http://package.gitlab.com>, but then a GET to
>> cloudfront is blocked anyway as it's not on our whitelist.
>> e.g.
>> 1619022938.916   423 172.16.230.237 NONE/200 0 CONNECT 54.153.54.194:443
>> <http://54.153.54.194:443> - ORIGINAL_DST/54.153.54.194
>> <http://54.153.54.194> -
>> 1619022939.074   153 172.16.230.237 TCP_MISS/302 758 GET
>> https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb
>> <https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/bionic/gitlab-ee_11.0.1-ee.0_amd64.deb/download.deb>
>> - ORIGINAL_DST/54.153.54.194 <http://54.153.54.194> text/html
>> 1619022939.108    20 172.16.230.237 NONE/200 0 CONNECT 52.84.90.34:443
>> <http://52.84.90.34:443> - ORIGINAL_DST/52.84.90.34 <http://52.84.90.34> -
>> 1619022939.114     2 172.16.230.237 TCP_DENIED/403 19053 GET
>> https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb <https://d20rj4el6vkp4c.cloudfront.net/7/11/ubuntu/package_files/35938.deb>?
>> - HIER_NONE/- text/html
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 

_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users