[squid-users] How to avoid Squid disclosing the origin server IP when there is an error

[Xen]

Again, impressed by your knowledge. But I'm not really arguing against 
your knowledge. It is basically a principle choice to /call/ one thing 
security and the other privacy based on the impression or experience that 
the one thing provides actual defenses or benefits in certain common 
scenario's and the other doesn't. Perhaps that is pertinent to software 
security, but in that case it is a very specific field and you are going 
to define "security" in a very constrained way.

Basically, it is then more of a normative statement "what do me and my 
buddies consider good enough" rather than a statement of definition.

You are basically arguing that in (all) real world scenarios (of 
software/web/server security) the obscurity thing tends to converge on 
irrelevance. But even that is true, it is still not a defining 
characteristic, so to speak.

And I am just not that experienced in securty. I barely know about side 
channel attacks because I did read one paper on a memory access (latency) 
"localhost" attack on I believe OpenSSH or something that required a local 
account to manipulate cache hits/misses.

And my math was just not up to par to understand it :p.

But I must say the way they narrowed down the possible set (or chains) of 
keys based on known information was very interesting.


On Sun, 27 Sep 2015, Amos Jeffries wrote:

>>> 1) security by obscurity does not work.
>> 
>> Seriously, that is just dogma being repeated over and over by people 
>> who just heard it one time and accepted it as their own truth, without 
>> really thinking it over. Security by obscurity works very well,

> This tells me you dont have much actual experience with security, or not 
> low-level enough. Real experience with attacks and working on 0-day 
> tools is better than even just "thinking it over". Thats what I based my 
> statement on. Though I do adopt the industry phrase to get the idea 
> across clearly.

But it's just not true. If all the world thinks that way and then devises 
systems and setups where it is practically useless, then it is still not 
true. Also, obscurity is not intended to defend against 
single-point-of-interest attacks. Or anything that makes you a really 
interesting target to those who already know you.

The point and the problem is that this industry phrase is a judgement that 
really closes off thinking. You might even start missing out on 
opportunies that might help you. I mean, to give this example again.

Perhaps it is not relevant when you have (or someone has) excellent 
scanning tools and all that. But usually, in the physical world, and in 
many many situations, thereof, there is certainly knowing or not knowing 
"the terrain". If you know where you are, for example, but your pursuer 
does not, it is obviously easier to escape. Now, if that pursuer was using 
helicopters, satellites, automatic camera surveillances and analysis, and 
tracking dogs to top it off :p, it would become a kind of different 
situation.

So perhaps in (computer) security the status quo is that attackers and 
defenders both have access to that level of sophistication in every 
instance. And if information is almost never capable of being hidden, 
well, then perhaps you'd better become a brick wall, or better yet, a 
nuclear shelter.

In lesser situations though it becomes perfectly clear that if you know 
how something works and someone else doesn't, you can use that to your 
advantage. You may not want to call that "security" but I would call it 
"escaping with my life".

There is something about masculine 'security' and feminine 'security'. The 
one you are advocating is like masculine. Strong defenses. Not relying on 
luck, or surrendering to the flow of things. Certainty at every level. 
Guarantees. But often in life you also have to proceed based on confidence 
and intuition alone, and you may need to trust that conditions will meet 
where you want to be, when you get there.

If you are exposed anyway because you are a big business and all that, 
there is already a design choice being made. Within certain choices 
already having been made, the resulting 'security landscape' is also 
informed by that choice or those choices.

Then, given that practical reality, you end up in a status quo where some 
truths seem to hold. One of these conclusions may be "security by 
obscurity is not security". But that still doesn't make it true. It makes 
it dogma that is only true provided some well-established conditions 
remain in force. That means it is not universal but more of a "rule of 
fist" (Dutch phrase) -- a best practice or guideline you can always rely 
on.

But it is actually a vulnerability. An attacker that is creative enough 
will see your (that) (not wanting to insult here) fixed mindset and know 
the blind spots that result from this fixation in thinking. This person 
would think "hey, that's curious". "They left this possibility open". "It 
just doesn't occur to them that anyone would do that".

I'm sorry, I was just like responding to that phrase alone, not 
necessarily to this subject of CDN networks and this particular error 
message, although the same holds true. My apologies.

It's just that that industry phrase, as you term it, is repeated so often 
that even a security layman as myself must have heard it uttered at least 
a dozen times in my own experience on mailing lists and perhaps more on 
the web. It is a mantra, but the fact that people feel it needs to be 
repeated constantly means that it doesn't come intuitive or instinctive. 
And that means that in general people don't agree with it. These 
disagreeing people are then called stupid in one way or another.

Sometimes you just have to stick to what makes sense intuitively to know 
the real truth. Because you might learn something new that people 
overlook.

And even if you know ...like nothing about security. You may still be able 
to devise a better system. If you understand the fundamentals better. If 
you can avoid the mistakes that were made prior to it. If you can unlock a 
secret. If you can travel to the moon through a gateway no one knew about 
;-).

> Not for sane security. For privacy. The two are different, though
> complimentary.

Then you (or the industry, as per you) has just reverted to calling the 
masculine type of security "security" and the feminine type of security 
"privacy" and warped both concepts in the process. It is more of a 
statement as to what you consider "worthy" than that it is about what it 
really is or comes down to.

All I know is that disregarding "privacy" can make you feel very /unsafe/. 
I will agree that this is not the same as feeling "secure" or "insecure". 
Security to my mind has more to do with having set up the required 
prerequisites required to counter any attack or problem. Security then 
revolves around feeling content that you have done what is necessary in 
the event of something happening. Privacy is meant to make the likelihood 
of that event happening, less or lower. Your situation can be extremely 
"insecure" in that sense but you can still make it out "unscathed" due to 
your level of privacy.

There is some line from some book about not telling your left hand what 
your right hand knows, that pertains to this ;-).

> To take your analogy; walking into the town square keeping your mouth 
> closed dressed in a cape and mask while someone is firing a machine gun 
> all over the place will see you just as dead.
> They won't know who you are, but you'll still be dead.

If that is the environment you operate in, well, good luck :p.

Most of the time in the real (physical) world the situation is that there 
are a lot of *potential* attackers (sometimes just about everyone) but 
attack only happens as soon as you are 'identified'. Cloak and dagger 
might not be that bad. I would agree that this is not an ideal situation 
to have for a 'living' but in this condition your real 'security' is going 
to be rather fragile regardless. You always need *some* level of "real" 
security, even if it is nothing more than your hideout in the mountains. I 
knew a guy who put traps in his vehicle and residences.

Person breaks into his car and starts to ignite it. But the ignition has 
been altered and the chambre is flooded with a sleeping gas. ;-).

Real world too. They are just Russian. Someone broke into his "vacation 
house" and the room automatically locked itself, flooded with water up to 
neck level, and alerted the authorities automatically. I have had a rock 
in my hands that was 3 billion years old. Andromeda :p.

You may not call that security but it sure works by obscurity ;-).

> But it does *not* tell that. HTTP is multi-hop. All the error message
> states is that 127.0.0.1 *somewhere* is unavailable. Good luck defining
> "somewhere".

> All it tells with certaintly is that there is at least one proxy 
> operating. Attackers have zero information about whether that server is 
> the origin server or just another proxy. They also dont have any info 
> from that page about how many proxies down the chain the error was 
> generated.
> Either way if erver X was the target they have to get to 
> server X through the proxy they are already contacting and/or profiling, 
> and cannot do so due to it being down and unavailable.

> It could as easily be one of these scenarios:

> <snip>

> Good luck if its the third one. Which is actually quite popular with all
> the major CDN.

Alright thanks for the information. I would still not be unhappy about 
this information if I was interested in something.

> In a system relying on obscurity the effort required is near zero.
> System profiling takes under 10 HTTP messages and attack scan to find
> the obscured vulnerability takes however many needed to scan through
> CVE/0-day the attacker or their tools knows about for that combo of
> software.

Back in the day (I've read that book that Assange cooperated on) 'hackers' 
were dealing with systems they knew nothing about, including how to 
program for it. You can say that did not make it terribly secure as 
eventually "they" found out. It did take a lot of effort though. When they 
learned, the first wrote a port/host scanner on that system, and after a 
time he came across a banking system that spewed out credit card 
information on first connect. Nothing required. Completely and only 
"protected" by obscurity, as it seems, as it was. Of course you can't rely 
on that.

But know this.

In a world where all important software is open source (as per "obscurity 
is not security) and if you stick to that, and your system is identified, 
then this attacker needs only use his toolkit or whatever advanced toolkit 
he can get / write. I'm thinking there are a bunch or a lot of female 
hackers these days too though. Maybe even 10%. Women understand this 
better.

What if, instead, you had a customized version of that software unique to 
your system. Many a hacker needs to study the exact version or environment 
of the software he wants to attack?.

Just by simple adjustment you can render any and all toolkits worthless.

This toolkit is actually a mass-targetting feature. You can perfectly 
devise defenses that counter it. It requires obscurity. Then your attacker 
suddenly needs to gain access to your binaries or source code. You now 
have an information advantage that is momentous. The toolkit is a feminine 
attack on a masculine defense, and hence successful. Obscurity, real 
obscurity ;-) always requires dedication on a single target to defeat it. 
Passwords are a form of obscurity, by the way. They are called secrets :p.

> And what does published documentation have to do with proxy A failing to 
> connect to some upstream server? This thread is not about documentation, 
> its about whether or not some admin has secured their CDN proxy.

Well actually I thought it was about whether this user considered the 
error message a vulnerability to his own system. I guess your response is 
"secure so it won't matter". Whenever someone gives me that sort of 
reassurance, I know I'm in danger ;-). But I think you already know the 
answer to your own question.

>> The same applies to attacking a system. If the cost becomes too high 
>> for the benefit that can be gained, people will just leave a system 
>> alone.
>> 
>> It is important.

> Now that is dogma. Probably true most of the time, but still experience
> (and a bit of thinking) uncovers cases where it is false.

Not sure what you mean. In general that would mean the cost is not too 
high? Or maybe there just is no cost to doing something. Or someone 
believes there is something to be gained anyway?

The world of proxies :D.

> It is a negative statement about availability with the property that
> when its *not* happening one still cannot be certain about up/down
> status on the particular server application.

Alright, good information. But I think this user was primarily concerned 
with the basic unpleasantness of any such information being shown? 
Practical impossiblity does not equate to fundamental uselessness. Just 
because something ... I guess I'm a bit stupid at this point, but just 
because you can be /pretty sure/ that nothing bad can happen doesn't mean 
it then becomes good practice to just go and flaunting it. You *are* 
depending on your perfect or sufficient knowledge of the entire system. 
That level of dependence might cause butt-hurt. If you happen to have 
missed something your negligence of precaution may suddenly cause a bite 
mark somewhere ;-). Just saying, you know.

> * If the attacker did manage to trigger this event they have no certinty
> about whether it was their action or a combination of their action and
> some other parallel traffic they are unaware of.

You are way cool you know. Seriously, mean it. Like I'm talking to the 
expert of experts ;D. :).

> How does that strike you for height on the bar?

I'd take on the challenge lol. In due time, maybe in a previous life. ;-).

LOL!

> The one attack where this is useful is a DoS attack, where it is a sign
> that DoS is happening, but since ther is clearly a proxy involved the
> Dos success/fail state is still uncertain. Proxies have this ability to
> limit DoS to particular clients and/or present different variants of
> response to different requests. So the attack request may be getting
> this error while regular traffic does not even notice.

Above my head at this point.. Even if you could use this feedback as an 
indication of DDoS (DoS) success ... well I guess it could warrant 
conditions for a further stage that would be impossible or too dangerous 
without it. Dunno. I know that this level of knowledge or feedback can be 
quite vital though. Sometimes if a feedback mechanism fails you are just 
left without recourse to proceed. I really hate that. I hate those people 
:P. LOL.

I'm no hacker, just pretending to be one to some people at some times 
;-P.

Seriously. I'm just a stupid person ;p.

> This is slightly incorrect. Protecting the server requires protecting 
> it, not hiding. Proxy offers a higher bar to DoS attacks, and added 
> complexity of software HTTP interptetations and ACLs that need to be 
> broken or bypassed before any attack is successful.

I mostly meant "hiding" as a form of encapsulating. I mean, the webserver 
could be on an internal network, right? It is not anymore hidden than your 
couch in your living room is hidden. Not everything has to be global IP 
exposed.

Personally I like this idea of connecting with e.g. VPN or whatever and 
then using a service that only answers to e.g. 127.0.0.1.

You can see how beginner I am :P.

> You still have to place the protections in both proxy and server to gain
> those proxy benefits. At which point the server location starts to mean
> almost nothing in terms of protection.

Do you mean that given adequate firewall settings a connect will be 
impossible anyway from a host that is not supposed to?

And at that point given this adequate or perfect design, any other 
precaution is rendered redundant?

> Having the server on the same host as the proxy adds the localhost 
> hardware protections. That is all. Then exposes it to side effects of 
> CPU consumption attacks made against the proxy which would not be an 
> issue if it were separate.

Right. I believe separate-host service or provision is important anyway. 
Such as not storing audit logs on the same machine.

> That high CPU consumption is just one normal peak traffic case for a
> proxy. So it can occur almost on demand if an attacker chooses. Squid is
> designed to continue operating under those conditions. Servers and in
> particular "application layer" stuff is much more fragile.

Not sure what you mean. You're saying that although Squid might be 
vulnerable to this scheme, it is not really of greatest interest given 
other vectors?

I guess application layer flaws account for the vast majority of 
break-ins?

Not sure, it seems that way. The higher you get in your coding (such as 
PHP) the easier it is to make mistakes or leave things open. Apart from C 
and its inherent weaknesses. PHP is like designed to have a nest of vipers 
that will bite you if you stick your hand in. Whatever.

> I mean it is mandatory for each proxy along the chain to send headers
> detailing the hosts and protocols the message has passed over. One
> ambiguous detail in a rarely occuring error message is the hard way to
> get any info which is spewed forth on every request. And on diagnostic
> requests is presented "on a silver platter" as the saying goes.

Right. I'm sorry, I was not very cognisant of that. I never studied these 
header responses from e.g. CDNs. It also seems weird that this information 
is mandatory.

>> Aye, it is prettier as well if this information is not shown/leaked.

> Then dont use that error template. Eliezer pointed out how. When it does
> occur it presents the minimum of vital debugging information necessary
> to resolve the outage.

Aye, I'm not necessarily saying it would be a flaw or mistake or error of 
the Squid proxy, just that perfection is not guaranteed if this is the 
default :p.

> This error message is specifically designed to reveal those two details
> of URL and which server was down. Such that it can be identified and
> fixed. So this does not qualify as a vulnerability leak.

That doesn't follow, but whatever ;-) :).

> In reverse-proxy cases the end user receiving it is perhapse not the
> best target, the log would be better. Patches making the distinction are
> welcome.

Aye, I guess that was the point. It would not matter to me if this was a 
forward proxy. It is pretty damn helpful that Squid tells me what host was 
down in a nice graphical page. For a regular forward proxy being informed 
like that is supremely helpful.

I'm not really in the position to do any coding at this point though in 
other systems, no matter how much I would love to get more familiar with 
code of projects that interest me. But thanks for the offer.

> I usually have no need, for instance, for detailed database connection
> failure reports. It is ugly and exposes a lot of internals to your user.
> A common user will also be thinking "what is this shit?". It is not
> tailored for the presentation that was created for the website proper.
> If anything, an error message like that would need to get a message page
> that is in line with the semantics/display/appearance of the page/site
> itself. Just to be consistent and keep the encapsulation intact.

This is what errorpage.css config file is for. Reverse-proxy
installations should use it to brand their proxy generated responses.

> :-) and welcome. If you would like to help improve the error messages,
> patches to update the templates are welcome.

> But be aware that proxies have a very wide set of use-cases to cater
> for, so eliminating details is not always the best choice. The case in
> point being that the detials being a worry to some now are critical for
> ISP installations to report, but not reverse-proxy. We ride the fine
> line between relevance and leaks.

;-). I guess that was the point. It is merely for reverse proxies that it 
is worrisome, I guess.

I barely even know what a reverse proxy is. Okay, I do. Slightly.

:p.

Thanks man.

Oh I'm gonna stop responding like this in detail because for some reason 
my email client (Alpine) has stopped prepending > to the 
replying-to-message for some reason and I don't know why or how or how to 
stop it :p.

Makes it rather tiresome to quote stuff.

Regards...