[squid-users] Data usage reported in log files

Sat Mar 11 20:48:19 UTC 2017

Hey Yosi,

Can you see if the differences is on the incoming or outgoing traffic?
Squid will only account for incoming and if you are using some kind of caching with the quick_abort and other partial content prefetch it would make sense that the actual consumption of the bits from the Internet to squid will not match from squid to clients.

If you can send me or share with others your squid.conf we might be able to understand if something there might cause such an issue.

Thanks,
Eliezer

* Feel free to contact me directly by skype or phone
----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il

-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Yosi Greenfield
Sent: Friday, March 10, 2017 11:47 PM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

Gentlemen,

Thanks Antony. Yes, we are accounting for everything else. I'm talking about port 3128 and 3129 only. 

Any other traffic is being tracked both by netflow and tcpdump and they match. What does not match is 3128/9 and squid log.

I'll report back after the weekend if the discrepancy is all sslbump traffic.

Thank you all,
Yosi

-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Antony Stone
Sent: Friday, March 10, 2017 4:31 PM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP, 
> no SIP, no torrents, no RDP, no other protocol. They simple does not 
> exists and we're all believe that's all not above over 1% of overall
traffic.
> Yes. Sure. Really.
> 
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from the IP of the Squid server, so as to ignore everything else he knows is happening on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid server, other than HTTP/S.

Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types 
> >>> of traffic besides HTTP / HTTPS, right?
> >>> 
> >>> DNS, ICMP, other protocols?
> >> 
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if 
> >> he's been measuring everything, I don't think DNS, ICMP and other 
> >> protocols would add more than 1% on top of HTTP/S, unless (as 
> >> Marcus suggested) there is also totally-non-Squid traffic on the link being measured.
> > 
> > Come on, sure? Even in L7? Really? Cool story, bro!
> > 
> >> Antony.
> >> 
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll 
> >>>> check that out, although I think that it's a problem even for 
> >>>> bumped users. Even for bumped users we don't bump all sites, so 
> >>>> that really could be it.
> >>>> 
> >>>> Thanks!
> >>>> 
> >>>> 
> >>>> -----Original Message-----
> >>>> From: squid-users 
> >>>> [mailto:squid-users-bounces at lists.squid-cache.org]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: squid-users at lists.squid-cache.org
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>> 
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>> 
> >>>>> Netflow is much larger.
> >>>>> 
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for 
> >>>>> overage, but I can't tell where that overage is being used. Are 
> >>>>> they using youtube, webmail, wetransfer? I see only a fraction 
> >>>>> of their actual proxy usage in my squid logs.
> >>>>> 
> >>>>> Data compression would give the opposite result, so that's not 
> >>>>> what I'm seeing.
> >>>>> 
> >>>>> Any other ideas?
> >>>> 
> >>>> Is there any traffic that is not directed to Squid?
> >>>> 
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS) 
> >>>> tunnels.
> >>>> 
> >>>> Marcus
> >>>> 
> >>>>> -----Original Message-----
> >>>>> From: squid-users 
> >>>>> [mailto:squid-users-bounces at lists.squid-cache.org]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: squid-users at lists.squid-cache.org
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>> 
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>> 
> >>>>>> I'm analyzing my squid logs with sarg, and I see that the 
> >>>>>> number of bytes reported as used by any particular user are 
> >>>>>> often nowhere
near
> >>>>>> the bytes reported by netflow and tcpdump.
> >>>>> 
> >>>>> Which is larger?
> >>>>> 
> >>>>>> I'm trying to trace my users' data usage by site, but I'm 
> >>>>>> unable to do so from the log files because of this.
> >>>>> 
> >>>>> Well, what is it you really want to know?
> >>>>> 
> >>>>> netflow / tcpdump will give you accurate numbers for the 
> >>>>> quantity of data on your Internet link - I assume this is what 
> >>>>> you're most interested in?
> >>>>> Squid will show you what quantity of data goes to/from the 
> >>>>> clients, but is that really important?
> >>>>> 
> >>>>>> Can someone please explain to me what I might be missing? Why 
> >>>>>> does squid log report one thing and netflow and tcpdump show 
> >>>>>> something else?
> >>>>> 
> >>>>> Data compression?
> >>>>> 
> >>>>> HTTP responses are often gzipped, so if tcpdump is showing you 
> >>>>> smaller numbers of bytes than Squid reports, that's what I'd 
> >>>>> look at first.
> >>>>> 
> >>>>> 
> >>>>> Antony.

--
<flopsie> yes, but this is #lbw, we don't do normal

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users