[squid-users] offline mode not working for me

Fri Jan 19 13:05:23 UTC 2024

Hi, thanks so much for the detailed response.  I chose to test option 2
from your recommendations as I am new to squid and I do not understand how
to set it up as a reverse proxy anyway.  I made the change to my squid.conf
:

#ssl_bump peek step1

ssl_bump bump step1

ssl_bump bump all

This made it work - which is great news.   My curl requests now are
satisfied by the cache when the pc is offline!

I do have 1 followup question which I think is unrelated, let me know if
etiquette demands I create a new post for this.  When I test using
chromium browser, chromium sends OPTION requests - which I think is
something to do with CORS.   These always cause cache MISS  from squid,.. I
think because the return code is 204...?

1705669236.776    113 ::1 TCP_MISS/204 680 OPTIONS
https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_DIRECT/
3.135.146.17 application/json

I can prevent my chromium instance from making these (pointless?) OPTIONS
calls using the following args, but I would rather not have to do this.

--disable-web-security  --disable-features=IsolateOrigins,site-per-process

Any way I can get squid to cache these calls?

Thanks again and all the best,

Robin

On Thu, 18 Jan 2024 at 16:03, Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 2024-01-18 09:53, Robin Carlisle wrote:
>
> > My expectation/hope is that squid would return the cached object on
> > any network failure in between ubuntu-pc and the AWS endpoint - and
> > continue to return this cached object forever.   Is this something
> > squid can do? It would seem that offline_mode should do this?
>
> Yes and yes. The reason you are getting errors are not related to cache
> hits or misses. Those errors happen _before_ Squid gets the requested
> resource URL and looks up that resource in Squid cache.
>
> > ssl_bump peek step1
> > ssl_bump bump all
>
> To get that URL (in your configuration), Squid must bump the connection.
> To bump the connection at step2, Squid must contact the origin server.
> When the cable is unplugged, Squid obviously cannot do that: The attempt
> to open a Squid-AWS connection fails.
>
>  > .../200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT
>  > .../503 4087 GET https://stuff.amazonaws.com/api/... - HIER_NONE
>
> Squid reports bumping errors to the client using HTTP responses. To do
> that, Squid remembers the error response, bumps the client connection,
> receives GET from the client on that bumped connection, and sends that
> error response to the client. This is why you see both CONNECT/200 and
> GET/503 access.log records. Note that Squid does not check whether the
> received GET request would have been a cache hit in this case -- the
> response to that request has been preordained by the earlier bumping
> failure.
>
>
> Solution candidates to consider include:
>
> * Stop bumping: https_port 443 cert=/etc/squid/stuff.pem
>
> Configure Squid as (a reverse HTTPS proxy for) the AWS service. Use
> https_port. No SslBump rules/options! The client would think that it is
> sending HTTPS requests directly to the service. Squid will forward
> client requests to the service. If this works (and I do not have enough
> information to know that this will work in your specific environment),
> then you will get a much simpler setup.
>
>
> * Bump at step1, before Squid contacts AWS: ssl_bump bump all
>
> Bugs notwithstanding, there will be no Squid-AWS connection for cache
> hits. The resulting certificate will not be based on AWS service info,
> but it looks like your client is ignorant enough to ignore related
> certificate problems.
>
>
> HTH,
>
> Alex.
>
>
> > Hi, Hoping someone can help me with this issue that I have been
> > struggling with for days now.   I am setting up squid on an ubuntu PC to
> > forward HTTPS requests to an API and an s3 bucket under my control on
> > amazon AWS.  The reason I am setting up the proxy is two-fold...
> >
> > 1) To reduce costs from AWS.
> > 2) To provide content to the client on the ubuntu PC if there is a
> > networking issue somewhere in between the ubuntu PC and AWS.
> >
> > Item 1 is going well so far.   Item 2 is not going well.   Setup details
> ...
> >
> > *# squid - setup cache folder*
> > mkdir -p /var/cache/squid
> > chown -R proxy:proxy  /var/cache/squid
> >
> > *# ssl - generate key*
> > apt --yes install squid-openssl libnss3-tools
> > openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \
> >    -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com
> > <http://www.example.com>" \
> >    -keyout /etc/squid/stuff.pem -out /etc/squid/stuff.pem
> > chown root:proxy /etc/squid/stuff.pem
> > chmod 644  /etc/squid/stuff.pem
> >
> > *# ssl - ssl DB*
> > mkdir -p /var/lib/squid
> > rm -rf /var/lib/squid/ssl_db
> > /usr/lib/squid/security_file_certgen -c -s /var/lib/squid/ssl_db -M 4MB
> > chown -R proxy:proxy /var/lib/squid/ssl_db
> >
> > *# /etc/squid/squid.conf :*
> > acl to_aws dstdomain .amazonaws.com <http://amazonaws.com>
> > acl from_local src localhost
> > http_access allow to_aws
> > http_access allow from_local
> > cache allow all
> > cache_dir ufs /var/cache/squid 1024 16 256
> > offline_mode on
> > http_port 3129 ssl-bump cert=/etc/squid/stuff.pem
> > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> > sslcrtd_program /usr/lib/squid/security_file_certgen -s
> > /var/lib/squid/ssl_db -M 4MB
> > acl step1 at_step SslBump1
> > ssl_bump peek step1
> > ssl_bump bump all
> > sslproxy_cert_error deny all
> > cache_store_log stdio:/var/log/squid/store.log
> > logfile_rotate 0
> >
> > *# /usr/bin/proxy-test :*
> > #!/bin/bash
> > curl --proxy http://localhost:3129 <http://localhost:3129> \
> >    --cacert /etc/squid/stuff.pem \
> >    -v "https://stuff.amazonaws.com/api/v1/stuff/stuff.json
> > <https://stuff.amazonaws.com/api/v1/stuff/stuff.json>" \
> >    -H "Authorization: token MYTOKEN" \
> >    -H "Content-Type: application/json" \
> >    --output "/tmp/stuff.json"
> >
> >
> >
> > When network connectivity is GOOD, everything works well and I get cache
> > HITS ...
> >
> > *# /var/log/squid/access.log*
> > 1705587538.837    238 127.0.0.1 NONE_NONE/200 0 CONNECT
> > stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> -
> > HIER_DIRECT/3.136.246.238 <http://3.136.246.238> -
> > 1705587538.838      0 127.0.0.1 TCP_MEM_HIT/200 32818 GET
> > https://stuff.amazonaws.com/api/v1/stuff/stuff.json
> > <https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/-
> > application/json
> >
> > *# extract from /usr/bin/proxy-test output*
> > < HTTP/1.1 200 OK
> > < Date: Thu, 18 Jan 2024 13:38:01 GMT
> > < Content-Type: application/json
> > < Content-Length: 32187
> > < x-amzn-RequestId: 8afba80e-6df7-4d5b-a34b-a70bd9b54380
> > < Last-Modified: 2024-01-03T11:23:19.000Z
> > < Access-Control-Allow-Origin: *
> > < x-amz-apigw-id: RvN1CF2_iYcEokA=
> > < Cache-Control: max-age=2147483648,public,stale-if-error
> > < ETag: "53896156c4e8e26933188a092c4e40f1"
> > < X-Amzn-Trace-Id: Root=1-65a929b9-3bd3285934151c1a2495481a
> > < Age: 2578
> > < Warning: 110 squid/5.7 "Response is stale"
> > < X-Cache: HIT from ubuntu-pc
> > < X-Cache-Lookup: HIT from ubuntu-pc:3129
> > < Via: 1.1 ubuntu-pc (squid/5.7)
> > < Connection: keep-alive
> >
> >
> > When network connectivity is BAD, I get errors and a cache MISS.   In
> > this test case I unplugged the ethernet cable from the back on the
> > ubuntu-pc ...
> >
> > *# /var/log/squid/access.log*
> > 1705588717.420     11 127.0.0.1 NONE_NONE/200 0 CONNECT
> > stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> -
> > HIER_DIRECT/3.135.162.228 <http://3.135.162.228> -
> > 1705588717.420      0 127.0.0.1 NONE_NONE/503 4087 GET
> > https://stuff.amazonaws.com/api/v1/stuff/stuff.json
> > <https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/-
> > text/html
> >
> > *# extract from /usr/bin/proxy-test output*
> > < HTTP/1.1 503 Service Unavailable
> > < Server: squid/5.7
> > < Mime-Version: 1.0
> > < Date: Thu, 18 Jan 2024 14:38:37 GMT
> > < Content-Type: text/html;charset=utf-8
> > < Content-Length: 3692
> > < X-Squid-Error: ERR_CONNECT_FAIL 101
> > < Vary: Accept-Language
> > < Content-Language: en
> > < X-Cache: MISS from ubuntu-pc
> > < X-Cache-Lookup: NONE from ubuntu-pc:3129
> > < Via: 1.1 ubuntu-pc (squid/5.7)
> > < Connection: close
> >
> > I have also seen it error in a different way with a 502 but with the
> > same ultimate result.
> >
> > My expectation/hope is that squid would return the cached object on any
> > network failure in between ubuntu-pc and the AWS endpoint - and continue
> > to return this cached object forever.   Is this something squid can do?
> >    It would seem that offline_mode should do this?
> >
> > Hope you can help,
> >
> > Robin
> >
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> > https://lists.squid-cache.org/listinfo/squid-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20240119/06298d11/attachment-0001.htm>