[squid-users] Using Digests to reduce traffic between peers, Parent - Sibling configuration question

[Alex Rousskov]

On 10/26/2015 10:14 PM, Jester Purtteman wrote:

> I have been wrestling with squid for a while and my reading has brought
> “Cache-Digests” to my attention.  I suspect the answer is “that would be
...
> As far as I can tell from (very limited) experimenting and reading, this
> doesn’t **appear** to be how it works,


Hello Jester,

    With a few configuration adjustments and code modifications, you can
make Cache Digests help with your use case. Cache Digests make decisions
based on request URLs. It sounds like you want to make decisions based
on response body as well. It is possible to change the code to do that,
but it will be a lot of non-trivial work and there will always be some
false positives because Cache Digests are not meant to give
always-precise answers.


As Amos mentioned, there are existing/standard HTTP mechanisms that are
meant to decrease pointless fetches across expensive links. However,
just like Cache Digests, "as is", they may not work well in your use
case. Those mechanisms make decisions based on origin-server-supplied
headers such as ETags. As you said, that information may be missing or
false in many responses.

Just like with Cache Digests, with a few configuration adjustments and
code modifications, you can make those standard mechanisms work better
for you. For example, you can teach Squid to generate its own ETag-like
content+header checksums that can be used in conditional HTTP requests
that Squid understands. None of this is easy, but it is doable.


There have been many proposals on how to solve this problem. I do not
think there is a single winning approach. Everybody seem to experiment
with their own tweaks of the existing tools and standards.

If you are looking for a solution that will cost you a few days/weeks of
development and sysadmin work, I do not think there is one. If you are
willing and able to invest a lot more, then I recommend that you
estimate the expected savings _before_ you invest in an expensive
solution. Getting reliable estimates is a complicated project on its
own, but it is still a lot cheaper than investing months into a solution
that does not meet your needs.


HTH,

Alex.