[squid-users] Data usage reported in log files

[Yuri Voinov]

11.03.2017 3:47, Yosi Greenfield пишет:
> Gentlemen,
>
> Thanks Antony. Yes, we are accounting for everything else. I'm
> talking about port 3128 and 3129 only. 
>
> Any other traffic is being tracked both by netflow and tcpdump and
> they match. What does not match is 3128/9 and squid log.
It can be also because of tunneled traffic.
>
> I'll report back after the weekend if the discrepancy is all
> sslbump traffic.
>
> Thank you all,
> Yosi
>
>
> -----Original Message-----
> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
> Behalf Of Antony Stone
> Sent: Friday, March 10, 2017 4:31 PM
> To: squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] Data usage reported in log files
>
> On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:
>
>> Of course, there is no stream video from security cams, no voice IP, 
>> no SIP, no torrents, no RDP, no other protocol. They simple does not 
>> exists and we're all believe that's all not above over 1% of overall
> traffic.
>> Yes. Sure. Really.
>>
>> Only web-surfing :) Sure :)
> Thanks for the standard sarcasm.
>
> Has it occurred to you that Yosi might have been measuring traffic to & from
> the IP of the Squid server, so as to ignore everything else he knows is
> happening on his network, so he can compare like with like?
>
> My "not more than 1%" was for the additional traffic to/from the Squid
> server, other than HTTP/S.
>
>
> Antony.
>
>> 11.03.2017 3:19, Yuri Voinov пишет:
>>> 11.03.2017 2:57, Antony Stone пишет:
>>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>>> Gentlemen, and it never occurred to you that there are other types of
>>>>> traffic besides HTTP / HTTPS, right?
>>>>>
>>>>> DNS, ICMP, other protocols?
>>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
>>>> been measuring everything, I don't think DNS, ICMP and other protocols
>>>> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
>>>> there is also totally-non-Squid traffic on the link being measured.
>>> Come on, sure? Even in L7? Really? Cool story, bro!
>>>
>>>> Antony.
>>>>
>>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>>> check that out, although I think that it's a problem even for bumped
>>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>>> could be it.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
>>>>>> On Behalf Of Marcus Kool
>>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>>> To: squid-users at lists.squid-cache.org
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Netflow is much larger.
>>>>>>>
>>>>>>> I really want to know exactly what site is costing my users data.
>>>>>>> Many of our users are on metered connections and are paying for
>>>>>>> overage, but I can't tell where that overage is being used. Are they
>>>>>>> using youtube, webmail, wetransfer? I see only a fraction of their
>>>>>>> actual proxy usage in my squid logs.
>>>>>>>
>>>>>>> Data compression would give the opposite result, so that's not what
>>>>>>> I'm seeing.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>> Is there any traffic that is not directed to Squid?
>>>>>>
>>>>>> Do you use ssl-bump in bump mode ?
>>>>>> If not, Squid has no idea how many bytes go through the (HTTPS)
>>>>>> tunnels.
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
>>>>>>> On Behalf Of Antony Stone
>>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>>> To: squid-users at lists.squid-cache.org
>>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>>
>>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>>>> bytes reported as used by any particular user are often nowhere
> near
>>>>>>>> the bytes reported by netflow and tcpdump.
>>>>>>> Which is larger?
>>>>>>>
>>>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>>>> do so from the log files because of this.
>>>>>>> Well, what is it you really want to know?
>>>>>>>
>>>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>>>> data on your Internet link - I assume this is what you're most
>>>>>>> interested in?
>>>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>>>> but is that really important?
>>>>>>>
>>>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>>>> else?
>>>>>>> Data compression?
>>>>>>>
>>>>>>> HTTP responses are often gzipped, so if tcpdump is showing you
>>>>>>> smaller numbers of bytes than Squid reports, that's what I'd look at
>>>>>>> first.
>>>>>>>
>>>>>>>
>>>>>>> Antony.

-- 
Bugs to the Future
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x613DEC46.asc
Type: application/pgp-keys
Size: 2437 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170311/f69fef7f/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170311/f69fef7f/attachment.sig>