[squid-users] Squid 2.7.s9 HTTPS-proxying - hint welcome

[Amos Jeffries]

On 18/08/2016 11:00 a.m., Torsten Kuehn wrote:
> Thank you for your quick reply!
> 
> On 17/08/2016 6:01 p.m., Amos Jeffries wrote:
> 
>>> I am forced to stuck with 2.X
>> Then you cannot decrypt the HTTPS in order to cache it. Squid older than
>> 3.2 simply do not have any of the functionality to do so.
> 
> I.e. not cacheable at all?

Correct.

> May sound stupid but I imagine layouts where the
> encrypted traffic itself gets stored and a client could re-use it later.

There is no way for Squid to know what is contained inside the
encryption. Therefore no way to know if a previous encrypted stream is
reusable. To squid it is just a random stream of opaque bytes.


> 
>> Cache is not an archive. Everything in a cache is by definition *not*
>> valuable and subject to be erased at any time.
>> [...] "of personal value" data is at high risk of being erased with
>> every request sent through that proxy
> 
> Archiving is a different matter, no question. But I prefer not to erase
> objects from my cache, unless requested. My refresh_patterns etc. may look
> horrible for administrators who try to provide most recent content:
>     authenticate_ttl 359996400 seconds

Lookup "credentials replay attack". Thats what this enables. There is no
other use for this directive than attacking your clients. We have
dropped it in Squid-3.

Use the "auth_param ... credentialsttl" to extend credentials lifespan.
It is still not good to have large values there, but way better than the
global TTL feature.


>     hierarchy_stoplist cgi-bin
>     maximum_object_size 1073741824 bytes
>     refresh_pattern -i /cgi-bin/ 5258880 100% 5258880

Please use the pattern settings:  "-i (/cgi-bin/|\?) 0 0% 0"

This pattern is designed to work around an issue with truly ancient CGI
scripts that are so old or broken that they do not produce any
cache-control headers. Such scripts are guaranteed to be dynamically
changing but being so old /broken they are incapable of telling the
proxy what their outputs expiry should be. Thus we need to use 0's to be
sure its accurate.

NP: this does not affect ASP or PHP type dynamic CGI scripts used in
modern sites. These languages are new enough that the scripting engines
automatically add appropriate cache-controls and the 0 values do not get
used on their outputs.


>     refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880
>     refresh_pattern . 5258880 100% 5258880
>         override-expire override-lastmod ignore-no-cache ignore-private
>     positive_dns_ttl 359996400 seconds

Meaning whenever any domain name moves hosting service you get cutoff
from it completely for ~11 years or until you fully restart Squid.

When you go to Squid-3, removing both these DNS settings entirely would
be best. But if you really insist on caching the DNS positive results
for a long time ~48hrs should work just as well and be much safer as well.


>     negative_dns_ttl 60 seconds
>     vary_ignore_expire on
>     reload_into_ims on
> 
> This setup is that robust that both force-reload

This is done solely by "override-expire". The rest of your config is
unrelated. Even the "reload_into_ims" does nothing because Squid is
explicitly ignoring the max-age=0 value, aka. 'force reload'.


> and PURGE fail unless

In Squid-2 this is more efficiently done by:
 acl PURGE method PURGE
 http_access deny PURGE

Squid-3 does that automatically by default and even disables all the
PURGE functionality *unless* you have "PURGE" method in an ACL like
above. It is a good idea for performance reasons to remove all mention
of "PURGE" when upgrading to Squid-3.


> objects are deleted manually (resulting in "truncate: Warning: DosOpen
> Error 110, OPEN_FAILED, file ...") or the ugly "reload_into_ims on" option
> is set which violates standards.

reload_into_ims is not a violation of current HTTP/1.1 standards. It is
simply undefined behaviour - therefore a bit risky.

The big(est) problem with it is that Squid-2 is HTTP/1.0 software and
both reload and IMS revalidation are not defined in that version of the
protocol. Adding two risky undefined things together multiplies the
dangers. The risk being that the server or client is not properly doing
the revalidate actions in HTTP/1.0 messaging.



Overall the approach to caching *at any cost* is doing more harm than
good, both to yourself and to many others. I have seen far worse
configurations though.

The only really terrible things you are doing is the authenticate_ttl
and positive_dns_ttl values. They are not relevant to the HTTP caching
behaviour and are doing a huge amount of harm to you.


Amos