[squid-users] [Feature request] add IP version to logformat format codes
Scott
3m9n51s2ewut at thismonkey.com
Sun Feb 16 12:59:57 UTC 2020
> Date: Sun, 16 Feb 2020 01:50:19 +1300
> From: Amos Jeffries <squid3 at treenet.co.nz>
> To: squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] squid-users Digest, Vol 66, Issue 17
>
> On 16/02/20 12:42 am, Scott wrote:
> >> Date: Fri, 14 Feb 2020 11:03:50 -0500
> >> From: Alex Rousskov
> >>
> >> On 2/14/20 10:36 AM, Scott wrote:
> >>
> >>> I know it's derivable by other means, but it would be nice to have a
> >>> logformat format code that provided the client and server IP version numbers.
> >>
> >>> eg: >v for Client IP version (4 or 6) and <v for Server
> >>
> >>
> >> Other than client and server, Squid can log a few other IP addresses,
> >> including:
> >>
> >> >a Client source IP address
> >> >la Local IP address the client connected to
> >> la Local listening IP address the client connection was...
> >> <a Server IP address of the last server or peer connection
> >> <la Local IP address of the last server or peer connection
> >> icap::<A ICAP server IP address. Similar to <A.
> >>
> >>
> >> If we add support for automated IP version extraction, it should be
> >> supported as a single new parameter for all existing %codes that log IP
> >> addresses rather than new %codes (one %code for each of the existing
> >> %codes that log IP addresses). For example:
> >>
> >> %>a{version}
> >>
> >> FWIW, personally, I am not sure we should add such a %code option
> >> because, I presume, the same information can be obtained simply by
> >> checking the first character of the logged IP address for being '['.
> >> Said that, I am open to hearing arguments why it should be added.
> >>
> >>
> >> Cheers,
> >>
> >> Alex.
> >>
> >
> > Thanks Alex,
> >
> > bear in mind that normally Squid handles but two connections (c->squid,
> > squid->peer/origin), despite the fact that there are normally four addresses
> > (client, squid-inside, squid-outside, peer/origin). If it were agreed to
> > support such a logging function, why would one bother having >a{version} and
> >> la{version} when both MUST be the same? Same goes for <a and <la.
> >
>
> If you are using an IPv6 enabled Squid on a Hybrid-stack machine you may
> notice that it does not have IPv4 listeners at all. Squid talks to IPv4
> clients through IPv6 :: or a v4-mapping address.
>
>
>
> > That's the whole point of "<" and ">". These two qualifiers are linked to
> > the inside and outside IP versions, not the "l" in ">la" and "<la". That's
> > why I suggested a new variable "v" with two sides/directions (>/<).
> >
> > As to the suggestion that one differentiate IP versions by the signifier '[',
> > from my experience "%>a" in logformat does NOT provide surrounding square
> > brackets.
>
> For Squid %<a / %>a codes the more correct sign is when the IP contains
> a ':' it is IPv6 or later.
>
>
> >
> > The argument I would make (and I do appreciate you hearing it) is that
> > programmatically (think grep/awk or pcre filtering) it's much easier to
> > determine how much traffic (client/server) is either v4 or v6 is by using a
> > fixed field rather than positive/negative lookaheads in the address codes
> > (given the lack of []).
>
> IMO it would be better to implement the long outstanding request for
> SNMP counters providing that information. No need to parse the logs then.
>
> Amos
>
I don't believe SNMP is the best place for per-connection metrics/telemetry.
SNMP is better suited to aggregated numerical data.
If I'm already logging proxy connections and gathering per-connection data
such as IP endpoints, URIs, HTTP methods, etc. then I think it also makes
sense to log the IP version. In my case I am very interested in
client->proxy and proxy->peer/origin sessions and whether they are IPv4 or 6.
I would lose all context were the protocol version be stored in some SNMP
gauge. Also having to 'grep' a field to get the version seems a bit b-grade.
It's just a feature request that I thought some other people would also
appreciate. I'll just knock up a patch for myself.
Hey while you're here Amos, did you see a post from me a few days ago
regarding ssl::server_name matching the Host: header in non-TLS connections?
I didn't get any response and was wondering if the documentation should
mention it.
Cheers
Scott
> Date: Sat, 15 Feb 2020 09:58:42 -0400
> From: Felipe Polanco <felipeapolanco at gmail.com>
> To: Amos Jeffries <squid3 at treenet.co.nz>
> CC: Squid Users <squid-users at lists.squid-cache.org>
> Subject: Re: [squid-users] Question regarding TPROXY and sslBump
>
> Thanks for the reply,
> Speaking strictly about TPROXY, are there any limitations compared to
> regular transparent intercept?
> We have full control of the network and TCP routing.
> We have done regular https intercept in the past and is working fine,
> but now we would like to try TPROXY in bridging mode instead of routing
> mode.
> Thanks,
>
> On Sat, Feb 15, 2020 at 3:17 AM Amos Jeffries <[1]squid3 at treenet.co.nz>
> wrote:
>
> On 15/02/20 10:28 am, Felipe Polanco wrote:
> > Hi,
> >
> > Can squid running in TPROXY mode intercept and decrypt HTTPS
> payload
> > with sslBump?
> >
> Maybe. It can do so about as well as NAT intercept mode can.
> Wherther TPROXY works depends on what level of access you have to
> control the TCP packet routing.
> Whether SSL-Bump can decrypt depends on what TLS features are being
> used
> by the HTTPS traffic - and whether it is HTTPS at all.
> These things are only loosely related.
> Amos
> _______________________________________________
> squid-users mailing list
> [2]squid-users at lists.squid-cache.org
> [3]http://lists.squid-cache.org/listinfo/squid-users
>
> References
>
> 1. mailto:squid3 at treenet.co.nz
> 2. mailto:squid-users at lists.squid-cache.org
> 3. http://lists.squid-cache.org/listinfo/squid-users
> Date: Sun, 16 Feb 2020 04:40:55 +1300
> From: Amos Jeffries <squid3 at treenet.co.nz>
> To: Squid Users <squid-users at lists.squid-cache.org>
> Subject: Re: [squid-users] Question regarding TPROXY and sslBump
>
> On 16/02/20 2:58 am, Felipe Polanco wrote:
> > Thanks for the reply,
> >
> > Speaking strictly about TPROXY, are there any limitations compared to
> > regular transparent intercept?
>
> I assume that by "regular transparent intercept" you mean NAT intercept.
>
> The primary difference between TPROXY and NAT ... is that NAT is *not*
> "transparent". All the differences derive from that.
>
> To use TPROXY the machine running it must have the ability to spoof IPs
> on packets outgoing from Squid and to properly deliver them afterwards.
> This primarily affects Squid hosted in cloud services where that
> low-level control is not permitted or quite difficult.
>
> The problems NAT introduces by having a different IP address on traffic
> arriving at servers largely disappear. But all other issues related to
> middleware touching the messages in transit remain the same.
>
> Amos
>
> Date: Sat, 15 Feb 2020 13:15:58 -0500
> From: Alex Rousskov <rousskov at measurement-factory.com>
> To: squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] squid-users Digest, Vol 66, Issue 17
>
> On 2/15/20 6:42 AM, Scott wrote:
> >> From: Alex Rousskov
> >> >a Client source IP address
> >> >la Local IP address the client connected to
> >> la Local listening IP address the client connection was...
> >> <a Server IP address of the last server or peer connection
> >> <la Local IP address of the last server or peer connection
> >> icap::<A ICAP server IP address. Similar to <A.
> >>
> >>
> >> If we add support for automated IP version extraction, it should be
> >> supported as a single new parameter for all existing %codes that log IP
> >> addresses rather than new %codes (one %code for each of the existing
> >> %codes that log IP addresses). For example:
> >>
> >> %>a{version}
>
> > bear in mind that normally Squid handles but two connections (c->squid,
> > squid->peer/origin), despite the fact that there are normally four addresses
> > (client, squid-inside, squid-outside, peer/origin). If it were agreed to
> > support such a logging function, why would one bother having >a{version} and
> > >la{version} when both MUST be the same? Same goes for <a and <la.
>
> Sorry, I do not understand the relevance of your rhetorical(?) question
> because my suggestion does not require anybody to log duplicate
> information: If an admin wants the IP version of the client-to-Squid
> connection, they can use either >a{version} or >la{version}, whichever
> works best for them.
>
> My point was that Squid can already log more than four IP addresses
> (corresponding to more than two connections), and that number can only
> go up. If additional version logging support is indeed warranted, then
> %address_x{ip_version} is a better interface than %address_x_ip_version
> or %connection_y_ip_version because the former needs to be documented
> and implemented once, while the latter needs to be documented and
> implemented once for every [two] address-logging options.
>
> In other words, the "each connection has two addresses" observation
> reduces the current pain of adding new %codes by 50%, but my suggestion
> reduces that pain by 83% (and approaching 100% long-term).
>
>
> > As to the suggestion that one differentiate IP versions by the signifier '[',
> > from my experience "%>a" in logformat does NOT provide surrounding square
> > brackets.
>
> Yes, according to Amos, the admin should be using ':' instead. The
> presence of a '.' can be used as well AFAICT. I am sorry for throwing a
> bracketing red herring into the debate!
>
>
> > The argument I would make (and I do appreciate you hearing it) is that
> > programmatically (think grep/awk or pcre filtering) it's much easier to
> > determine how much traffic (client/server) is either v4 or v6 is by using a
> > fixed field rather than positive/negative lookaheads in the address codes
> > (given the lack of []).
>
> For grep/pcre, the complexity of using ':' instead of '6' (or '[.]'
> instead of '4') seems about the same to me as far as "ease" of filtering
> is concerned. You could argue that records with a fixed version field
> can be filtered _faster_, but not "much easier" AFAICT.
>
> Not sure about awk.
>
>
> Please note that I am _not_ arguing against adding IP version logging
> yet, just making sure that we use the right logging interface (and that
> the arguments used in the debate are valid). I am not convinced either
> way yet.
>
> Convincing others that an already logged information should also be
> available for logging in a different format would be difficult but
> probably not impossible. After all, Squid does support %code .min_width
> formatting option that may delete some of the field information and even
> %code options that extract partial information (%>h{field} being one of
> the best examples); perhaps "log just the IP version" could be seen as a
> similar option.
>
>
> HTH,
>
> Alex.
>
> Date: Sat, 15 Feb 2020 14:08:21 -0600
> From: Rafael Silva Daniel <rafaelsilvadaniel at gmail.com>
> To: squid-users at lists.squid-cache.org
> Subject: [squid-users] please, can someone help me with the negotiate
> kerberos?
>
> Helo! i think i did almost everything right, firstly i made it in a test
> enviroment with debian stretch running squid 3.5 and a windows server 2008
> based domain controller, and it worked!
>
> but when i tried to deploy it in the production enviroment running debian
> stretch, squid 3.5 and windows server 2012 as the domain controller the
> authentication never works, the file /var/log/squid/cache.log shows this:
>
> 2020/02/14 15:40:21 kid1| ERROR: Negotiate Authentication validating user.
> Result: {result=BH, notes={message: gss_acquire_cred() failed: Unspecified
> GSS failure. Minor code may provide more information. No principal in
> keytab matches desired name; }}
> negotiate_kerberos_auth.cc(610): pid=13887 :2020/02/14 15:40:22|
> negotiate_kerberos_auth: DEBUG: Got 'YR (LETTERS AND NUMBERS)' from squid
> (length: 2439).
> negotiate_kerberos_auth.cc(663): pid=13887 :2020/02/14 15:40:22|
> negotiate_kerberos_auth: DEBUG: Decode '(LETTERS AND NUMBERS)' (decoded
> length: 1826).
>
> Obs1:I replaced a big string with letters and numbers by "(LETTERS AND
> NUMBERS)"
> Obs2: i posted more of the file in this link https://pastebin.com/Z2fe98dB
>
> well, the results of running: kinit -kt /etc/squid/HTTP.keytab
> HTTP/squid2.domain.local at DOMAIN.LOCAL:
> root at SERVER:~# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/squid2.domain.local at DOMAIN.LOCAL
>
> Valid starting Expires Service principal
> 02/15/2020 10:55:32 02/15/2020 20:55:32 krbtgt/DOMAIN.LOCAL at DOMAIN.LOCAL
> renew until 02/16/2020 09:55:32
>
>
>
> The results of running:klist -kte /etc/squid/HTTP.keytab
>
> Keytab name: FILE:/etc/squid/HTTP.keytab
> KVNO Timestamp Principal
> ---- -------------------
> ------------------------------------------------------
> 1 02/12/2020 17:33:15 squid2$@DOMAIN.LOCAL (arcfour-hmac)
> 1 02/12/2020 17:33:16 squid2$@DOMAIN.LOCAL (aes128-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 squid2$@DOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 SQUID2$@DOMAIN.LOCAL (arcfour-hmac)
> 1 02/12/2020 17:33:16 SQUID2$@DOMAIN.LOCAL (aes128-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 SQUID2$@DOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 HTTP/squid2.domain.local at DOMAIN.LOCAL
> (arcfour-hmac)
> 1 02/12/2020 17:33:16 HTTP/squid2.domain.local at DOMAIN.LOCAL
> (aes128-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 HTTP/squid2.domain.local at DOMAIN.LOCAL
> (aes256-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 host/squid2 at DOMAIN.LOCAL (arcfour-hmac)
> 1 02/12/2020 17:33:16 host/squid2 at DOMAIN.LOCAL (aes128-cts-hmac-sha1-96)
> 1 02/12/2020 17:33:16 host/squid2 at DOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 squid2$@DOMAIN.LOCAL (arcfour-hmac)
> 3 02/12/2020 17:36:59 squid2$@DOMAIN.LOCAL (aes128-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 squid2$@DOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 SQUID2$@DOMAIN.LOCAL (arcfour-hmac)
> 3 02/12/2020 17:36:59 SQUID2$@DOMAIN.LOCAL (aes128-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 SQUID2$@DOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 HTTP/squid2.domain.local at DOMAIN.LOCAL
> (arcfour-hmac)
> 3 02/12/2020 17:36:59 HTTP/squid2.domain.local at DOMAIN.LOCAL
> (aes128-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 HTTP/squid2.domain.local at DOMAIN.LOCAL
> (aes256-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 host/squid2 at DOMAIN.LOCAL (arcfour-hmac)
> 3 02/12/2020 17:36:59 host/squid2 at DOMAIN.LOCAL (aes128-cts-hmac-sha1-96)
> 3 02/12/2020 17:36:59 host/squid2 at DOMAIN.LOCAL (aes256-cts-hmac-sha1-96)
>
> And the results of running: root at SERVER:~#
> /usr/lib/squid/negotiate_kerberos_auth_test server.domain.local
> Token: (Alonglinewithnumbersandletters)
>
> the configs of the /etc/krb5.conf:
>
> [libdefaults]
> default_realm = DOMAIN.LOCAL
> dns_lookup_kdc = no
> dns_lookup_realm = no
> ticket_lifetime = 24h
> default_keytab_name = /etc/squid/HTTP.keytab
>
> default_tgs_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc
> des-cbc-md5
> default_tkt_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc
> des-cbc-md5
> permitted_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc
> des-cbc-md5
>
> [realms]
> DOMAIN.LOCAL = {
> kdc = dc01.domain.local
> admin_server = dc01.domain.local
> default_domain = domain.local
> }
>
> [domain_realm]
> .domain.local = DOMAIN.LOCAL
> domain.local = DOMAIN.LOCAL
>
> and the /etc/squid/squid.conf:
>
> http_port 3128
> dns_nameservers 200.198.5.4 200.198.5.5
> visible_hostname PROXY
> cache_dir ufs /var/spool/squid 100 16 256
> coredump_dir /var/spool/squid
>
> url_rewrite_program /usr/bin/squidGuard
>
> #auth parameter NEGOTIATE
> auth_param negotiate program /usr/lib/squid/negotiate_kerberos_auth -d -s
> HTTP/squid.domain.local -k /etc/squid/HTTP.keytab
> auth_param negotiate children 30
> auth_param negotiate keep_alive on
>
> acl Safe_ports port 80 # http
> acl Safe_ports port 443 # https
> acl Safe_ports port 90 # metodo
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl CONNECT method CONNECT
> acl auth proxy_auth REQUIRED
>
> http_access deny !Safe_ports
> http_access deny CONNECT !Safe_ports
> http_access allow localhost manager
> http_access deny manager
> http_access allow localhost
> http_access deny !auth
> http_access allow auth
>
>
>
> In the domain controller i created in the two zones the proper dns records,
> the host with squid can have his ip resolved to its right hostname, and its
> hostname resolved to its right ip, in the clients i setted the proxy as
> server.domain.local, and in the squid access.log the requests came but are
> all denied and a prompt for user and password are showed to the user
>
> Obs: the only data edited while posting was that i replaced our domain by
> domain.local, the name of the host by SERVER, and long strings of data in
> the cache log and negotiate kerberos test out, all the rest is what is
> really running in the files.
>
> please someone help me, i tried to read everything i could find but i am not
> finding how to understand what i am doing wrong, thanks in advance, D:
>
>
>
>
>
> --
> Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
More information about the squid-users
mailing list