[squid-users] "NF getsockopt(SO_ORIGINAL_DST)" filling cache.log due to AWS ELB healthchecks

[John Smith]

The outbound traffic from the L1proxy instance in question connects to a
public IP / DNS name of an ELB in another AWS region.
We need to send some traffic to a different AWS region, thus the mess below:

AWS instances (clients) ->
AWS internal ELB for L1 proxies -> AWS L1 proxy instances ->
a different AWS internal ELB for  L1 proxy cluster -> a different AWS L1
proxy instance (this is where we have the problem is with 'intercept or
transparent) ->
*One AWS region above, a different AWS region below*
AWS external (publicly addressable) ELB for L2 proxies in a different AWS
region -> AWS L2 proxy instances -> the Internet

These AWS instances have both internal IPs and public IPs, and they don't
really know about their own public IPs.  That may be part or all of the
confusion.

AWS ELBs are published as DNS names, they have multiple IPs, and we are
using DNS to connect to them.

I'm not exactly certain how the ELB functions, at least I don't know enough
to answer your question.
The healthcheck and listeners are are TCP, not HTTP.


On Thu, Oct 29, 2015 at 1:19 PM, Amos Jeffries <squid3 at treenet.co.nz> wrote:

> On 30/10/2015 8:39 a.m., John Smith wrote:
> > Hi Eliezer,
> >
> > It is entirely possible that haproxy is a better solution than squid for
> > what we are doing.
> > I have never used either solution, and inherited this 'working' squid
> > configuration with the task of cleaning things up and stabilizing it.
> > Regarding your question of 'How do the first layer of proxies send their
> > request to the second layer of proxies?', all I can tell you is that all
> > the work is done in the squid.conf, and I've posted the entire contents
> > with a few replacements for security reasons.
>
> The LB is the first layer.
> Squid is the second layer.
> The cache_peer Squid is relaying on to are a third.
>
>
> The hierarchy is thus:
>
>   clients --(?1?)--> LB --(?2?)--> Squid --(HTTP)--> peer(s)
>
> The "?1?" and "?2" parts are what we need to figure out.
>
> How are the clients messages getting to the LB?
>
> a) is it published in DNS as the host of some domain and clients making
> requests directly from it that need to be serviced by the proxies you
> are wrangling with?
>
> b) are the clients outboung traffic being intercepted/diverted in their
> travels to some other server and sent through the LB instead?
>
>
> How is the load blancer then relying the TCP connections and HTTP
> messages inside them to Squid?
>
> A) Is it using NAT to change both dst-IP and dst-port on the TCP packets?
>
> B) Is it terminating the TCP connection from clients, opening one  to
> Squid and relaying the HTTP inside based on URL hostnames etc. ?
>
> C) Is it opening a L2 tunnel and relaying the TCP packets to the Squid
> machine ?
>
>
>
> > As I've said, I've removed the word 'intercept' several times and the
> > requests to secondary proxies no longer work.
> > I just confirmed this behaviour again.
> > If this is as 'quiet' as I can make the logs then it is what it is.
>
>
> "getsockopt(SO_ORIGINAL_DST) failed ...: (92) Protocol not available"
>
> Should never happen in a working proxy. Ever.
>
> It can appear because a) the NAT tables in the Squid machine kernel are
> overflowing, or b) the external network is configured in a broken way.
> Both are really bad scenarios. Your are having the (b) problem.
>
> Amos
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20151029/bf770501/attachment-0001.html>