[squid-users] Data usage reported in log files

[Yuri Voinov]

According to the above, NetFlow will always show much more traffic than
the SQUID. This is obvious and there is nothing to discuss here. If this
is not clear to someone, put a collector that collects statistics at the
data link level and compare the counters. I'm not just talking about
TCP, Alex. There is also the UDP. And there are a lot of protocols that
squid can not see, including for the simple reason that these packets
are never routed to a SQUID.

We have not seen the network topology and the full configuration of
network devices - what are we arguing about and guessing about?


11.03.2017 3:27, Yuri Voinov пишет:
> Think of one simple thing. Squid does not see and can not see protocols
> that do not support. What do you expect from it? Does it work on L1/L2?
> No? Then what is the discussion about?
>
>
> 11.03.2017 3:22, Yuri Voinov пишет:
>> Of course, there is no stream video from security cams, no voice IP, no
>> SIP, no torrents, no RDP, no other protocol. They simple does not exists
>> and we're all believe that's all not above over 1% of overall traffic.
>> Yes. Sure. Really.
>>
>> Only web-surfing :) Sure :)
>>
>>
>> 11.03.2017 3:19, Yuri Voinov пишет:
>>> 11.03.2017 2:57, Antony Stone пишет:
>>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>>
>>>>> Gentlemen, and it never occurred to you that there are other types of
>>>>> traffic besides HTTP / HTTPS, right?
>>>>>
>>>>> DNS, ICMP, other protocols?
>>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been 
>>>> measuring everything, I don't think DNS, ICMP and other protocols would add 
>>>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also 
>>>> totally-non-Squid traffic on the link being measured.
>>> Come on, sure? Even in L7? Really? Cool story, bro!
>>>> Antony.
>>>>
>>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>>> check that out, although I think that it's a problem even for bumped
>>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>>> could be it.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
>>>>>> Behalf Of Marcus Kool
>>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>>> To: squid-users at lists.squid-cache.org
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Netflow is much larger.
>>>>>>>
>>>>>>> I really want to know exactly what site is costing my users data. Many
>>>>>>> of our users are on metered connections and are paying for overage,
>>>>>>> but I can't tell where that overage is being used. Are they using
>>>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>>>> proxy usage in my squid logs.
>>>>>>>
>>>>>>> Data compression would give the opposite result, so that's not what
>>>>>>> I'm seeing.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>> Is there any traffic that is not directed to Squid?
>>>>>>
>>>>>> Do you use ssl-bump in bump mode ?
>>>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
>>>>>>> On Behalf Of Antony Stone
>>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>>> To: squid-users at lists.squid-cache.org
>>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>>
>>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>>>> bytes reported as used by any particular user are often nowhere near
>>>>>>>> the bytes reported by netflow and tcpdump.
>>>>>>> Which is larger?
>>>>>>>
>>>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>>>> do so from the log files because of this.
>>>>>>> Well, what is it you really want to know?
>>>>>>>
>>>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>>>> data on your Internet link - I assume this is what you're most
>>>>>>> interested in?
>>>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>>>> but is that really important?
>>>>>>>
>>>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>>>> else?
>>>>>>> Data compression?
>>>>>>>
>>>>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>>>>
>>>>>>>
>>>>>>> Antony.

-- 
Bugs to the Future
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x613DEC46.asc
Type: application/pgp-keys
Size: 2437 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170311/ef3b8360/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170311/ef3b8360/attachment.sig>