[squid-users] Data not being cached

[Darwin O'Connor]

I run a transit prediction web app <https://www.transsee.ca/>. It 
connects to a variety of web APIs to collect the real time transit data 
it needs. The app's activities are split among many processes. They 
currently uses libcurl to connect to squid for caching (often for as 
little as 10-30 seconds) and benefits of connection sharing.

There is still cases where data isn't being cached no matter what I do. 
It is https data, but I am able to cache other https pages like 
https://cdn.mbta.com/realtime/Alerts.pb

The refresh_pattern:

refresh_pattern .               60      99999%  7200 override-expire 
override-lastmod reload-into-ims ignore-reload ignore-no-cache 
ignore-no-store ignore-private ignore-auth store-stale

The http headers from curl of an example where it is not being cached:

*   Trying 127.0.0.1:3128...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
 > GET https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses HTTP/1.1
Host: 127.0.0.1:3128
User-Agent: curl/7.77.0 (+https://www.transsee.ca/)
Accept: */*
Accept-Encoding: gzip
Authorization: apikey 2eYEqXXxOPEDChnpeF7sZL2aR8moD2DtdNmn
Cache-Control: max-age=60
Content-Encoding: aes128gcm

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sun, 20 Jun 2021 17:52:14 GMT
< Content-Type: application/protobuf
< Content-Length: 7455
< Set-Cookie: 
AWSALB=EEAv0EXNaGzBkgAPS1JzrAiQZi7aKDk063qjM4ApeOuIArbwQ+s17/jimuq7AzzuC+a35t5Lx/y+H5YMqTC9bL6LqS4Y75auumXt7LWnvSFE/SMXQd/OFa6TSCW9; 
Expires=Sun, 27 Jun 2021 17:52:14 GMT; Path=/
< Set-Cookie: 
AWSALBCORS=EEAv0EXNaGzBkgAPS1JzrAiQZi7aKDk063qjM4ApeOuIArbwQ+s17/jimuq7AzzuC+a35t5Lx/y+H5YMqTC9bL6LqS4Y75auumXt7LWnvSFE/SMXQd/OFa6TSCW9; 
Expires=Sun, 27 Jun 2021 17:52:14 GMT; Path=/; SameSite=None; Secure
< Server: Apache-Coyote/1.1
< X-Powered-By: Express
< Access-Control-Allow-Credentials: true
< ETag: W/"ab70-8SI2GdBV4SJG4edSc4E5W8LBJWk"
< Vary: Accept-Encoding
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Pop: SYD1-C1
< X-Amz-Cf-Id: hCoQckLsNONQMpgPr2kXJVdTDHu98jxl-rPXqV_PHB2vTCEomAd-Nw==
< Age: 35
< Access-Control-Allow-Origin: *
< Content-Encoding: gzip
< X-Cache: MISS from transsee
< X-Cache-Lookup: MISS from transsee:3128
< Via: 1.1 359a113ca166631b42f31a0f2e6a1aab.cloudfront.net (CloudFront), 
1.1 transsee (squid/4.15)
< Connection: keep-alive

Here is a sample from the Squid access log:

1624212034.891    246 127.0.0.1 59216 TCP_MISS/200 8517 GET 
https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses - 
HIER_DIRECT/52.65.222.24 application/protobuf