[squid-users] Data usage reported in log files

[Antony Stone]

On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:

> Gentlemen, and it never occurred to you that there are other types of
> traffic besides HTTP / HTTPS, right?
> 
> DNS, ICMP, other protocols?

I'm assuming Yosi has been measuring only TCP traffic, but even if he's been 
measuring everything, I don't think DNS, ICMP and other protocols would add 
more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
totally-non-Squid traffic on the link being measured.


Antony.

> 11.03.2017 2:44, Yosi Greenfield пишет:
> > Aha! That could be it. I use sslbump, but not for all users. I'll
> > check that out, although I think that it's a problem even for bumped
> > users. Even for bumped users we don't bump all sites, so that really
> > could be it.
> > 
> > Thanks!
> > 
> > 
> > -----Original Message-----
> > From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
> > Behalf Of Marcus Kool
> > Sent: Friday, March 10, 2017 3:38 PM
> > To: squid-users at lists.squid-cache.org
> > Subject: Re: [squid-users] Data usage reported in log files
> > 
> > On 10/03/17 16:27, Yosi Greenfield wrote:
> >> Thanks!
> >> 
> >> Netflow is much larger.
> >> 
> >> I really want to know exactly what site is costing my users data. Many
> >> of our users are on metered connections and are paying for overage,
> >> but I can't tell where that overage is being used. Are they using
> >> youtube, webmail, wetransfer? I see only a fraction of their actual
> >> proxy usage in my squid logs.
> >> 
> >> Data compression would give the opposite result, so that's not what
> >> I'm seeing.
> >> 
> >> Any other ideas?
> > 
> > Is there any traffic that is not directed to Squid?
> > 
> > Do you use ssl-bump in bump mode ?
> > If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
> > 
> > Marcus
> > 
> >> -----Original Message-----
> >> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
> >> On Behalf Of Antony Stone
> >> Sent: Friday, March 10, 2017 2:21 PM
> >> To: squid-users at lists.squid-cache.org
> >> Subject: Re: [squid-users] Data usage reported in log files
> >> 
> >> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>> Hello all,
> >>> 
> >>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>> bytes reported as used by any particular user are often nowhere near
> >>> the bytes reported by netflow and tcpdump.
> >> 
> >> Which is larger?
> >> 
> >>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>> do so from the log files because of this.
> >> 
> >> Well, what is it you really want to know?
> >> 
> >> netflow / tcpdump will give you accurate numbers for the quantity of
> >> data on your Internet link - I assume this is what you're most
> >> interested in?
> > 
> >> Squid will show you what quantity of data goes to/from the clients,
> >> but is that really important?
> >> 
> >>> Can someone please explain to me what I might be missing? Why does
> >>> squid log report one thing and netflow and tcpdump show something
> >>> else?
> >> 
> >> Data compression?
> >> 
> >> HTTP responses are often gzipped, so if tcpdump is showing you smaller
> >> numbers of bytes than Squid reports, that's what I'd look at first.
> >> 
> >> 
> >> Antony.

-- 
Normal people think "If it ain't broke, don't fix it".
Engineers think "If it ain't broke, it doesn't have enough features yet".

                                                   Please reply to the list;
                                                         please *don't* CC me.