[squid-users] TCP_MISS only
Andy Armstrong
andy.armstrong at uk.ibm.com
Fri Oct 21 12:50:08 UTC 2022
Hi Amos, All,
I continue to struggle with only receiving TCP_MISS when using squid-cache. I have now implemented a server which is responding to HTTP GETs. I have checked the endpoint with redbot.org and it states that :
1/The response allows all caches to store it
2/The response is fresh until 7 days from now
The reason may still be served by a cache once it becomes stale.
Therefore I believe the server side of this is now valid as a test for squid cache.
My client, is routing calls to squidcache, and when it arrives there, I see the following log entry:
1666355825.841 106 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
Infact every single client call I make results in this TCP_MISS.
I therefore think this may be to do with my squid.conf.
Please see my squid conf below – paying particular attention to the refresh patterns I added specifically for this call (because it wasn’t working without these so I thought I would add these to see if it would work).
Squid.conf:
http_port 3128
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 1025-65535 # unregistered ports
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
http_access allow localnet
http_access allow localhost
http_access deny all
coredump_dir /squid/var/cache/squid
cache_dir ufs /var/spool/squid 1024 16 256
refresh_pattern -i http:\/\/192.168.20.1:3001\/.* 10080 100% 43200 override-lastmod
refresh_pattern -i http:\/\/192.168.20.1:3004\/.* 10080 100% 43200 override-lastmod
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
The complete output from the logs is :
admin at zcxsys01:~$ docker logs squid-container
2022/10/21 12:16:58| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2022/10/21 12:16:58| Set Current Directory to /etc/squid
2022/10/21 12:16:58| Creating missing swap directories
2022/10/21 12:16:58| /var/spool/squid exists
2022/10/21 12:16:58| /var/spool/squid/00 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/00
2022/10/21 12:16:58| /var/spool/squid/01 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/01
2022/10/21 12:16:58| /var/spool/squid/02 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/02
2022/10/21 12:16:58| /var/spool/squid/03 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/03
2022/10/21 12:16:58| /var/spool/squid/04 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/04
2022/10/21 12:16:59| /var/spool/squid/05 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/05
2022/10/21 12:16:59| /var/spool/squid/06 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/06
2022/10/21 12:16:59| /var/spool/squid/07 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/07
2022/10/21 12:16:59| /var/spool/squid/08 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/08
2022/10/21 12:16:59| /var/spool/squid/09 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/09
2022/10/21 12:16:59| /var/spool/squid/0A exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0A
2022/10/21 12:16:59| /var/spool/squid/0B exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0B
2022/10/21 12:16:59| /var/spool/squid/0C exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0C
2022/10/21 12:16:59| /var/spool/squid/0D exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0D
2022/10/21 12:16:59| /var/spool/squid/0E exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0E
2022/10/21 12:16:59| /var/spool/squid/0F exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0F
2022/10/21 12:16:59| Removing PID file (/run/squid.pid)
2022/10/21 12:16:59| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2022/10/21 12:16:58| Set Current Directory to /etc/squid
2022/10/21 12:16:58| Creating missing swap directories
2022/10/21 12:16:58| /var/spool/squid exists
2022/10/21 12:16:58| /var/spool/squid/00 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/00
2022/10/21 12:16:58| /var/spool/squid/01 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/01
2022/10/21 12:16:58| /var/spool/squid/02 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/02
2022/10/21 12:16:58| /var/spool/squid/03 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/03
2022/10/21 12:16:58| /var/spool/squid/04 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/04
2022/10/21 12:16:59| /var/spool/squid/05 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/05
2022/10/21 12:16:59| /var/spool/squid/06 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/06
2022/10/21 12:16:59| /var/spool/squid/07 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/07
2022/10/21 12:16:59| /var/spool/squid/08 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/08
2022/10/21 12:16:59| /var/spool/squid/09 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/09
2022/10/21 12:16:59| /var/spool/squid/0A exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0A
2022/10/21 12:16:59| /var/spool/squid/0B exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0B
2022/10/21 12:16:59| /var/spool/squid/0C exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0C
2022/10/21 12:16:59| /var/spool/squid/0D exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0D
2022/10/21 12:16:59| /var/spool/squid/0E exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0E
2022/10/21 12:16:59| /var/spool/squid/0F exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0F
2022/10/21 12:16:59| Removing PID file (/run/squid.pid)
2022/10/21 12:16:59| Set Current Directory to /etc/squid
2022/10/21 12:16:59| Starting Squid Cache version 5.2 for s390x-ibm-linux-gnu...
2022/10/21 12:16:59| Service Name: squid
2022/10/21 12:16:59| Process ID 11
2022/10/21 12:16:59| Process Roles: master worker
2022/10/21 12:16:59| With 1048576 file descriptors available
2022/10/21 12:16:59| Initializing IP Cache...
2022/10/21 12:16:59| DNS Socket created at 0.0.0.0, FD 8
2022/10/21 12:16:59| Adding nameserver 10.11.5.1 from /etc/resolv.conf
2022/10/21 12:16:59| Adding nameserver 10.11.5.2 from /etc/resolv.conf
2022/10/21 12:16:59| Adding domain test.sys.one from /etc/resolv.conf
2022/10/21 12:16:59| Logfile: opening log daemon:/var/log/squid/access.log
2022/10/21 12:16:59| Logfile Daemon: opening log /var/log/squid/access.log
2022/10/21 12:17:00| Unlinkd pipe opened on FD 14
2022/10/21 12:17:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2022/10/21 12:17:00| Store logging disabled
2022/10/21 12:17:00| Swap maxSize 1048576 + 262144 KB, estimated 100824 objects
2022/10/21 12:17:00| Target number of buckets: 5041
2022/10/21 12:17:00| Using 8192 Store buckets
2022/10/21 12:17:00| Max Mem size: 262144 KB
2022/10/21 12:17:00| Max Swap size: 1048576 KB
2022/10/21 12:17:00| Rebuilding storage in /var/spool/squid (dirty log)
2022/10/21 12:17:00| Using Least Load store dir selection
2022/10/21 12:17:00| Set Current Directory to /etc/squid
2022/10/21 12:17:00| Finished loading MIME types and icons.
2022/10/21 12:17:00| HTCP Disabled.
2022/10/21 12:17:00| Pinger socket opened on FD 19
2022/10/21 12:17:00| Squid plugin modules loaded: 0
2022/10/21 12:17:00| Adaptation support is off.
2022/10/21 12:17:00| Accepting HTTP Socket connections at conn2 local=0.0.0.0:3128 remote=[::] FD 17 flags=9
2022/10/21 12:17:00| Done reading /var/spool/squid swaplog (0 entries)
2022/10/21 12:17:00| Store rebuilding is 0.00% complete
2022/10/21 12:17:00| Finished rebuilding storage from disk.
2022/10/21 12:17:00| 0 Entries scanned
2022/10/21 12:17:00| 0 Invalid entries.
2022/10/21 12:17:00| 0 With invalid flags.
2022/10/21 12:17:00| 0 Objects loaded.
2022/10/21 12:17:00| 0 Objects expired.
2022/10/21 12:17:00| 0 Objects cancelled.
2022/10/21 12:17:00| 0 Duplicate URLs purged.
2022/10/21 12:17:00| 0 Swapfile clashes avoided.
2022/10/21 12:17:00| Took 0.12 seconds ( 0.00 objects/sec).
2022/10/21 12:17:00| Beginning Validation Procedure
2022/10/21 12:17:00| Completed Validation Procedure
2022/10/21 12:17:00| Validated 0 Entries
2022/10/21 12:17:00| store_swap_size = 0.00 KB
2022/10/21 12:17:00| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2022/10/21 12:17:00| pinger: Initialising ICMP pinger ...
2022/10/21 12:17:00| pinger: ICMP socket opened.
2022/10/21 12:17:00| pinger: ICMPv6 socket opened
2022/10/21 12:17:01| storeLateRelease: released 0 objects
1666355216.966 213 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355220.808 93 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355221.369 66 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355222.770 254 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355224.514 82 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355225.518 80 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355225.963 65 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355226.366 64 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355226.715 69 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355227.371 315 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355227.700 56 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355239.913 207 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355244.729 96 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355245.014 63 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355408.390 208 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355426.040 197 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355426.650 63 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355427.196 65 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355427.565 60 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355427.915 64 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355824.295 209 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355825.153 120 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355825.841 106 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
I suspect something is wrong with my config – perhaps the rules are not working correctly. Please can you offer some advice as to why I can’t get this response to cache in squid-cache.
Kind regards,
Andy Armstrong
安迪 阿姆斯特朗
Principal Specialist for Z Technologies
EMEA Squad Leader for Hybrid Cloud
Worldwide Community Leader for Hybrid Cloud
Member of the CTO Office Server & Storage EMEA
Distinguished Technical Specialist – The Open Group
IBM Master Inventor
Mobile: +447500103874
From: squid-users <squid-users-bounces at lists.squid-cache.org> on behalf of Andy Armstrong <andy.armstrong at uk.ibm.com>
Date: Thursday, 29 September 2022 at 15:00
To: Amos Jeffries <squid3 at treenet.co.nz>, squid-users at lists.squid-cache.org <squid-users at lists.squid-cache.org>
Subject: [EXTERNAL] Re: [squid-users] TCP_MISS only
Hi, Excellent I understand and agree with what you are saying. Is this behaviour documented within the Squid documentation anywhere, or is this more ‘how does the HTTP specification handle caching’?
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi,
Excellent I understand and agree with what you are saying. Is this behaviour documented within the Squid documentation anywhere, or is this more ‘how does the HTTP specification handle caching’?
I am moving forward with a HTTP GET to see if that works per my use case. I assume therefore that any other verb is simply not going to work out the box?
Kind regards,
Andy Armstrong
安迪 阿姆斯特朗
Principal Specialist for Z Technologies
EMEA Squad Leader for Hybrid Cloud
Worldwide Community Leader for Hybrid Cloud
Member of the CTO Office Server & Storage EMEA
Distinguished Technical Specialist – The Open Group
IBM Master Inventor
Mobile: +447500103874
From: squid-users <squid-users-bounces at lists.squid-cache.org> on behalf of Amos Jeffries <squid3 at treenet.co.nz>
Date: Thursday, 29 September 2022 at 13:06
To: squid-users at lists.squid-cache.org <squid-users at lists.squid-cache.org>
Subject: [EXTERNAL] Re: [squid-users] TCP_MISS only
On 28/09/22 07:56, Andy Armstrong wrote:
> Okay – but what happens if you are communicating with a non REST
> endpoint.
You are still communicating over HTTP. To interact with and benefit from
HTTP agents like caches you need to comply to the HTTP semantics they use.
IMO, REST is just a useful tool to define (in abstract) an API's
operation when considering what/how it needs to be implemented.
> Consider a Web services endpoint for example where a request
> is only interacted with via POST but the operation for example may
> frequently be a read based function akin to a HTTP GET?
That is by definition a broken implementation of HTTP. The agent is
using a *delivery* API (POST) for retrieval (GET).
If you can separate the delivery and fetch operations HTTP becomes much
easier to use.
> Is Squid just
> simply not going to help cache those requests?
Not *by default*, no.
POST implies changing some arbitrary resource *other* than the URL
presented. Based on data and logic which may not be provided in the
request message URL+headers.
To use POST with caching both the client *and* the server have to
explicitly tell the HTTP cache agent(s) what to do on every single HTTP
message.
- The client has to tell the cache whether a stored response is able
to be produced as reply, what object-ID it is trying to retrieve, what
object-ID's it already knows about (if any), and how old the stored
object is allowed to be.
- The server has to tell the cache whether the response can be stored,
what to use for a unique-ID of the reply object, how old it already is,
how long it can be stored for, how and when to update it when it becomes
stale.
The Squid refresh_pattern can provide defaults for the storage times
when they are omitted. But all the ID related things and whether to use
cache at all can only come from the client/server.
As you can see by limiting yourself to POST-only you have imposed a huge
amount of complexity. Using GET instead for fetches makes all the above
*optional* where now it is mandatory.
> It is only helpful for
> more strict alignment to REST principles?
>
You lost me here. Squid implements HTTP.
REST is a very abstract simplification of basic HTTP/1.0 semantics. So
the closer ones code aligns to REST the *easier* it is to implement HTTP
properly. But HTTP/1.1+ are vastly more than REST.
HTH
Amos
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users<http://lists.squid-cache.org/listinfo/squid-users>
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20221021/bf01ba01/attachment-0001.htm>
More information about the squid-users
mailing list