[squid-users] Data not being cached

[Darwin O'Connor]

On 2021-06-22 8:00 a.m., squid-users-request at lists.squid-cache.org wrote:

> On 21/06/21 6:41 am, Darwin O'Connor wrote:
>> I run a transit prediction web app <https://www.transsee.ca/>. It
>> connects to a variety of web APIs to collect the real time transit data
>> it needs. The app's activities are split among many processes. They
>> currently uses libcurl to connect to squid for caching (often for as
>> little as 10-30 seconds) and benefits of connection sharing.
>>
>> There is still cases where data isn't being cached no matter what I do.
>> It is https data, but I am able to cache other https pages like
>> https://cdn.mbta.com/realtime/Alerts.pb
>>
>> The refresh_pattern:
>>
>> refresh_pattern .?????????????? 60????? 99999%? 7200 override-expire
>> override-lastmod reload-into-ims ignore-reload ignore-no-cache
>> ignore-no-store ignore-private ignore-auth store-stale
>>
> Please be aware that several of those options may be causing you more
> problems than they solve:
>
> * ignore-no-cache - no longer exists.
>
> * ignore-reload - contradicts reload-into-ims and that can cause
> inconsistent behaviour between initial MISS and followup HIT responses.
>
> * override-lastmod - replaces all Last-Modification (L-M) headers with
> values indicating the object is almost brand new. That prevents caches
> detecting that objects have stuck around long enough not to need 
> replacing.
> Luckily the example response has no L-M header, so that is not the
> problem here. But it may be affecting other traffic.

I have removed these options. My refresh_pattern is now

refresh_pattern .               60      99999%  7200 override-expire 
ignore-no-store ignore-private ignore-auth store-stale

Here are two requests, one after another. The headers match the actual 
client apps.

*   Trying 127.0.0.1:3128...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
> GET https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses HTTP/1.1
Host: 127.0.0.1:3128
User-Agent: curl/7.77.0 (+https://www.transsee.ca/)
Accept: */*
Accept-Encoding: gzip
Authorization: apikey 4sc5KapPRVzX6OeQm74FPCx9O0bFsIB7lei3
Cache-Control: max-age=60

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 22 Jun 2021 23:23:06 GMT
< Content-Type: application/protobuf
< Content-Length: 7317
< Set-Cookie: 
AWSALB=uydcNa4qcva/6YErLw+Ztl55Xvjtkf1N6HQiEfXOrPv46pjWn6uHEnqcRlPMWjcHXu6vAsbYjB/1WdxrVIYu0AAbrtIcz00zyAIiimoWQhWUlQFb6PO9120gf46N; 
Expires=Tue, 29 Jun 2021 23:23:06 GMT; Path=/
< Set-Cookie: 
AWSALBCORS=uydcNa4qcva/6YErLw+Ztl55Xvjtkf1N6HQiEfXOrPv46pjWn6uHEnqcRlPMWjcHXu6vAsbYjB/1WdxrVIYu0AAbrtIcz00zyAIiimoWQhWUlQFb6PO9120gf46N; 
Expires=Tue, 29 Jun 2021 23:23:06 GMT; Path=/; SameSite=None; Secure
< Server: Apache-Coyote/1.1
< X-Powered-By: Express
< Access-Control-Allow-Credentials: true
< ETag: W/"9b9e-32oicwXN7rJop21q+Y++RZhtnac"
< Vary: Accept-Encoding
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Pop: SYD1-C
< X-Amz-Cf-Id: fQeOCQLj5CiuSv2JV06y7O0r5rM0LvNchlXKFSBDKkES0-tUrmaorQ==
< Age: 31
< Access-Control-Allow-Origin: *
< Content-Encoding: gzip
< X-Cache: MISS from transsee
< X-Cache-Lookup: MISS from transsee:3128
< Via: 1.1 e57fe70b9ed429fb51b4b2432cadc67b.cloudfront.net (CloudFront), 
1.1 transsee (squid/4.15)
< Connection: keep-alive

*   Trying 127.0.0.1:3128...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
> GET https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses HTTP/1.1
Host: 127.0.0.1:3128
User-Agent: curl/7.77.0 (+https://www.transsee.ca/)
Accept: */*
Accept-Encoding: gzip
Authorization: apikey 4sc5KapPRVzX6OeQm74FPCx9O0bFsIB7lei3
Cache-Control: max-age=60

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 22 Jun 2021 23:23:13 GMT
< Content-Type: application/protobuf
< Content-Length: 7317
< Set-Cookie: 
AWSALB=zjeR+TBhpLTKE+mr042891Cxkb7caSUqya7wvG82Ca7cK86Z7iQco0ahsj/9H9WtdgJuiX/D9++AOYiQf3+9yryXrZscG5eX+X0+vVjI+t2fI/86DjV27tXLyt3o; 
Expires=Tue, 29 Jun 2021 23:23:13 GMT; Path=/
< Set-Cookie: 
AWSALBCORS=zjeR+TBhpLTKE+mr042891Cxkb7caSUqya7wvG82Ca7cK86Z7iQco0ahsj/9H9WtdgJuiX/D9++AOYiQf3+9yryXrZscG5eX+X0+vVjI+t2fI/86DjV27tXLyt3o; 
Expires=Tue, 29 Jun 2021 23:23:13 GMT; Path=/; SameSite=None; Secure
< Server: Apache-Coyote/1.1
< X-Powered-By: Express
< Access-Control-Allow-Credentials: true
< ETag: W/"9b9e-32oicwXN7rJop21q+Y++RZhtnac"
< Vary: Accept-Encoding
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Pop: SYD1-C1
< X-Amz-Cf-Id: cL-w8TjYWTb6cjEOLsyfwyIvZvQYTI7Hn4R8Uz6cppHEJwVUWejO1w==
< Age: 38
< Access-Control-Allow-Origin: *
< Content-Encoding: gzip
< X-Cache: MISS from transsee
< X-Cache-Lookup: MISS from transsee:3128
< Via: 1.1 e57fe70b9ed429fb51b4b2432cadc67b.cloudfront.net (CloudFront), 
1.1 transsee (squid/4.15)
< Connection: keep-alive

Here is the corresponding records from the access log. Before the 
records where from different fetches.

1624404186.746    298 127.0.0.1 59540 TCP_MISS/200 8380 GET 
https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses - 
HIER_DIRECT/3.105.153.49 application/protobuf

1624404193.862    269 127.0.0.1 59886 TCP_MISS/200 8380 GET 
https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses - 
HIER_DIRECT/3.105.153.49 application/protobuf