[squid-users] Data usage reported in log files

[Antony Stone]

On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP, no
> SIP, no torrents, no RDP, no other protocol. They simple does not exists
> and we're all believe that's all not above over 1% of overall traffic.
> Yes. Sure. Really.
> 
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from the 
IP of the Squid server, so as to ignore everything else he knows is happening 
on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid server, 
other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types of
> >>> traffic besides HTTP / HTTPS, right?
> >>> 
> >>> DNS, ICMP, other protocols?
> >> 
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
> >> been measuring everything, I don't think DNS, ICMP and other protocols
> >> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
> >> there is also totally-non-Squid traffic on the link being measured.
> > 
> > Come on, sure? Even in L7? Really? Cool story, bro!
> > 
> >> Antony.
> >> 
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll
> >>>> check that out, although I think that it's a problem even for bumped
> >>>> users. Even for bumped users we don't bump all sites, so that really
> >>>> could be it.
> >>>> 
> >>>> Thanks!
> >>>> 
> >>>> 
> >>>> -----Original Message-----
> >>>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: squid-users at lists.squid-cache.org
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>> 
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>> 
> >>>>> Netflow is much larger.
> >>>>> 
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for
> >>>>> overage, but I can't tell where that overage is being used. Are they
> >>>>> using youtube, webmail, wetransfer? I see only a fraction of their
> >>>>> actual proxy usage in my squid logs.
> >>>>> 
> >>>>> Data compression would give the opposite result, so that's not what
> >>>>> I'm seeing.
> >>>>> 
> >>>>> Any other ideas?
> >>>> 
> >>>> Is there any traffic that is not directed to Squid?
> >>>> 
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS)
> >>>> tunnels.
> >>>> 
> >>>> Marcus
> >>>> 
> >>>>> -----Original Message-----
> >>>>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: squid-users at lists.squid-cache.org
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>> 
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>> 
> >>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>>>>> bytes reported as used by any particular user are often nowhere near
> >>>>>> the bytes reported by netflow and tcpdump.
> >>>>> 
> >>>>> Which is larger?
> >>>>> 
> >>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>>>>> do so from the log files because of this.
> >>>>> 
> >>>>> Well, what is it you really want to know?
> >>>>> 
> >>>>> netflow / tcpdump will give you accurate numbers for the quantity of
> >>>>> data on your Internet link - I assume this is what you're most
> >>>>> interested in?
> >>>>> Squid will show you what quantity of data goes to/from the clients,
> >>>>> but is that really important?
> >>>>> 
> >>>>>> Can someone please explain to me what I might be missing? Why does
> >>>>>> squid log report one thing and netflow and tcpdump show something
> >>>>>> else?
> >>>>> 
> >>>>> Data compression?
> >>>>> 
> >>>>> HTTP responses are often gzipped, so if tcpdump is showing you
> >>>>> smaller numbers of bytes than Squid reports, that's what I'd look at
> >>>>> first.
> >>>>> 
> >>>>> 
> >>>>> Antony.

-- 
<flopsie> yes, but this is #lbw, we don't do normal

                                                   Please reply to the list;
                                                         please *don't* CC me.