[squid-users] log_referrer question
Bobby Matznick
bmatznick at pbandt.bank
Tue May 21 18:47:34 UTC 2024
To add and maybe clarify what my confusion is, the log entries below (hidden internal/external IP's, domain and username) don't seem to show what I expected, a line marked "referrer". Am I misunderstanding how that should show up in the log? Thanks
1716316179.294 0 ***.***.***.*** TCP_DENIED/407 4048 CONNECT cc-api-data.adobe.io:443 - HIER_NONE/- text/html
1716316179.297 0 ***.***.***.***TCP_DENIED/407 4048 CONNECT cc-api-data.adobe.io:443 - HIER_NONE/- text/html
1716316179.310 0 ***.***.***.***TCP_DENIED/407 4048 CONNECT cc-api-data.adobe.io:443 - HIER_NONE/- text/html
1716316179.313 0 ***.***.***.***TCP_DENIED/407 4112 CONNECT ib.adnxs.com:443 - HIER_NONE/- text/html
1716316179.316 0 ***.***.***.***TCP_DENIED/407 4144 CONNECT htlb.casalemedia.com:443 - HIER_NONE/- text/html
1716316179.316 0 ***.***.***.***TCP_DENIED/407 4048 CONNECT cc-api-data.adobe.io:443 - HIER_NONE/- text/html
1716316179.318 0 ***.***.***.***TCP_DENIED/407 4172 CONNECT fastlane.rubiconproject.com:443 - HIER_NONE/- text/html
1716316179.320 0 ***.***.***.***TCP_DENIED/407 4152 CONNECT hbopenbid.pubmatic.com:443 - HIER_NONE/- text/html
1716316179.322 20103 ***.***.***.***TCP_TUNNEL/200 3363 CONNECT th.bing.com:443 ***\\Username HIER_DIRECT/***.***.***.***-
1716316179.324 0 ***.***.***.***TCP_DENIED/407 4132 CONNECT bidder.criteo.com:443 - HIER_NONE/- text/html
1716316179.328 0 ***.***.***.***TCP_DENIED/407 4048 CONNECT cc-api-data.adobe.io:443 - HIER_NONE/- text/html
1716316179.331 0 ***.***.***.***TCP_DENIED/407 4048 CONNECT cc-api-data.adobe.io:443 - HIER_NONE/- text/html
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of squid-users-request at lists.squid-cache.org
Sent: Tuesday, May 21, 2024 11:51 AM
To: squid-users at lists.squid-cache.org
Subject: [External] squid-users Digest, Vol 117, Issue 23
Send squid-users mailing list submissions to
squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org>
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.squid-cache.org/listinfo/squid-users<https://lists.squid-cache.org/listinfo/squid-users>
or, via email, send a message with subject or body 'help' to
squid-users-request at lists.squid-cache.org<mailto:squid-users-request at lists.squid-cache.org>
You can reach the person managing the list at
squid-users-owner at lists.squid-cache.org<mailto:squid-users-owner at lists.squid-cache.org>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of squid-users digest..."
Today's Topics:
1. log_referrer question (Bobby Matznick)
----------------------------------------------------------------------
Message: 1
Date: Tue, 21 May 2024 17:50:49 +0000
From: Bobby Matznick <bmatznick at pbandt.bank<mailto:bmatznick at pbandt.bank>>
To: "squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org>"
<squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org>>
Subject: [squid-users] log_referrer question
Message-ID:
<MW5PR14MB52897188C2ED83596B406151B0EA2 at MW5PR14MB5289.namprd14.prod.outlook.com<mailto:MW5PR14MB52897188C2ED83596B406151B0EA2 at MW5PR14MB5289.namprd14.prod.outlook.com>>
Content-Type: text/plain; charset="utf-8"
I have been trying to use a combined log format for squid. The below line in the squid config is my current attempt.
logformat combined %>a %[ui %[un [%tl "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
It is working, as far as logging the normal stuff I would see before having tried to implement referrer. I noticed somewhere that you need to build squid with -enable-referrer-log, it was an older version, looked like 3.1 and lower, I am using 4.13. So, checked with squid -v and do not see "-enable-referrer_log" as one of the configure options used during install. Would I need to reinstall, or is that no longer necessary in version 4.13? Thanks!!
Bobby
From: squid-users <squid-users-bounces at lists.squid-cache.org<mailto:squid-users-bounces at lists.squid-cache.org>> On Behalf Of squid-users-request at lists.squid-cache.org<mailto:squid-users-request at lists.squid-cache.org>
Sent: Tuesday, April 23, 2024 6:00 AM
To: squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org>
Subject: [External] squid-users Digest, Vol 116, Issue 31
Caution: This is an external email and has a suspicious subject or content. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department
Send squid-users mailing list submissions to
squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org%3cmailto:squid-users at lists.squid-cache.org>>
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.squid-cache.org/listinfo/squid-users<https://lists.squid-cache.org/listinfo/squid-users><https://lists.squid-cache.org/listinfo/squid-users<https://lists.squid-cache.org/listinfo/squid-users>>
or, via email, send a message with subject or body 'help' to
squid-users-request at lists.squid-cache.org<mailto:squid-users-request at lists.squid-cache.org<mailto:squid-users-request at lists.squid-cache.org%3cmailto:squid-users-request at lists.squid-cache.org>>
You can reach the person managing the list at
squid-users-owner at lists.squid-cache.org<mailto:squid-users-owner at lists.squid-cache.org<mailto:squid-users-owner at lists.squid-cache.org%3cmailto:squid-users-owner at lists.squid-cache.org>>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of squid-users digest..."
Today's Topics:
1. Re: Warm cold times (Amos Jeffries)
2. Re: Container Based Issues Lock Down Password and Terminate
SSL (Amos Jeffries)
----------------------------------------------------------------------
Message: 1
Date: Tue, 23 Apr 2024 19:41:37 +1200
From: Amos Jeffries <squid3 at treenet.co.nz<mailto:squid3 at treenet.co.nz<mailto:squid3 at treenet.co.nz%3cmailto:squid3 at treenet.co.nz>>>
To: squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org%3cmailto:squid-users at lists.squid-cache.org>>
Subject: Re: [squid-users] Warm cold times
Message-ID: <9d8f4de6-c797-4e70-aaf5-c073f45c3390 at treenet.co.nz<mailto:9d8f4de6-c797-4e70-aaf5-c073f45c3390 at treenet.co.nz<mailto:9d8f4de6-c797-4e70-aaf5-c073f45c3390 at treenet.co.nz%3cmailto:9d8f4de6-c797-4e70-aaf5-c073f45c3390 at treenet.co.nz>>>
Content-Type: text/plain; charset=UTF-8; format=flowed
On 22/04/24 17:42, Jonathan Lee wrote:
> Has anyone else taken up the fun challenge of doing windows update caching. It is amazing when it works right. It is a complex configuration, but it is worth it to see a warm download come down that originally took 30 mins instantly to a second client. I didn?t know how much of the updates are the same across different vendor laptops.
>
There have been several people over the years.
The collected information is being gathered at
<https://wiki.squid-cache.org/ConfigExamples/Caching/WindowsUpdates<https://wiki.squid-cache.org/ConfigExamples/Caching/WindowsUpdates><https://wiki.squid-cache.org/ConfigExamples/Caching/WindowsUpdates<https://wiki.squid-cache.org/ConfigExamples/Caching/WindowsUpdates>>>
If you would like to check and update the information for the current
Windows 11 and Squid 6, etc. that would be useful.
Wiki updates are now made using github PRs against the repository at
<https://github.com/squid-cache/squid-cache.github.io<https://github.com/squid-cache/squid-cache.github.io><https://github.com/squid-cache/squid-cache.github.io<https://github.com/squid-cache/squid-cache.github.io>>>.
> Amazing stuff Squid team.
> I wish I could get some of the Roblox Xbox stuff to cache but it?s a night to get running with squid in the first place, I had to splice a bunch of stuff and also wpad the Xbox system.
FWIW, what I have seen from routing perspective is that Roblox likes to
use custom ports and P2P connections for a lot of things. So no high
expectations there, but anything cacheable is great news.
>> On Apr 18, 2024, at 23:55, Jonathan Lee wrote:
>>
>> ?Does anyone know the current warm cold download times for dynamic cache of windows updates?
>>
>> I can say my experience was a massive increase in the warm download it was delivered in under a couple mins versus 30 or so to download it cold. The warm download was almost instant on the second device. Very green energy efficient.
>>
>>
>> Does squid 5.8 or 6 work better on warm delivery?
There is no significant differences AFAIK. They both come down to what
you have configured. That said, the ongoing improvements may make v6
some amount of "better" - even if only trivial.
>> Is there a way to make 100 percent sure a docker container can?t get inside the cache?
For Windows I would expect the only "100% sure" way is to completely
forbid access to the disk where the cache is stored.
The rest of your questions are about container management and Windows
configuration. Which are kind of off-topic.
Cheers
Amos
------------------------------
Message: 2
Date: Tue, 23 Apr 2024 20:03:42 +1200
From: Amos Jeffries <squid3 at treenet.co.nz<mailto:squid3 at treenet.co.nz<mailto:squid3 at treenet.co.nz%3cmailto:squid3 at treenet.co.nz>>>
To: squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org%3cmailto:squid-users at lists.squid-cache.org>>
Subject: Re: [squid-users] Container Based Issues Lock Down Password
and Terminate SSL
Message-ID: <58c41ee7-b88c-4d5a-bd12-220d44465067 at treenet.co.nz<mailto:58c41ee7-b88c-4d5a-bd12-220d44465067 at treenet.co.nz<mailto:58c41ee7-b88c-4d5a-bd12-220d44465067 at treenet.co.nz%3cmailto:58c41ee7-b88c-4d5a-bd12-220d44465067 at treenet.co.nz>>>
Content-Type: text/plain; charset=UTF-8; format=flowed
On 23/04/24 11:52, Jonathan Lee wrote:
> Hello fellow Squid Accelerator/Dynamic Cache/Web Cache Users/PfSense users
>
> I think this might resolve any container based issues/fears if they
> happened to get into the cache. Ie a Docker Proxy got installed and
> tried to data marshal the network card inside of a freeBSD jail or
> something like that. Biggest fear with my cache it is a big cache now
>
> Please yet me know what you think or if it is wrong.
>
> Here is my configuration. I wanted to share it as it might help to
> secure some of this.
FTR, this config was auto-generated by pfsense. A number of things which
that tool forces into the config could be done much better in the latest
Squid, but the tool does not do due to needing to support older Squid
version.
>
> Keep in mine I use cachemgr.cgi within Squidlight so I had to set the
> password and I have to also adapt the php status file to include the
> password and also the sqlight php file.
>
> After that the status and gui pages work still with the new password.
> Only issues area that it shows up in clear text when it goes over the
> proxy I can see my password clear as day again that was an issue listed
> inside the Squid O?REILLY book also.
Please ensure you are using the latest Squid v6 release. That release
has both a number of security fixes, and working https:// URL access to
the manager reports.
The cachemgr.cgi tool is deprecated fro a number of issues including
that style of embedding passwords in the URLs.
Francesco and I have created a tool that can be found at
<https://github.com/yadij/cachemgr.js/blob/master/README.md<https://github.com/yadij/cachemgr.js/blob/master/README.md><https://github.com/yadij/cachemgr.js/blob/master/README.md<https://github.com/yadij/cachemgr.js/blob/master/README.md>>> for basic
access to the reports directly from Browser.
That tool uses HTTP authentication configured via the well-documented
proxy_auth ACLs and http_access for more secure access than the old URL
based mechanism (which still exists, just deprecated).
Cheers
Amos
------------------------------
Subject: Digest Footer
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org%3cmailto:squid-users at lists.squid-cache.org>>
https://lists.squid-cache.org/listinfo/squid-users<https://lists.squid-cache.org/listinfo/squid-users><https://lists.squid-cache.org/listinfo/squid-users<https://lists.squid-cache.org/listinfo/squid-users>>
------------------------------
End of squid-users Digest, Vol 116, Issue 31
********************************************
CONFIDENTIALITY NOTICE: The information contained in and attached to this email is intended only for the confidential use of the person or entity to which the email is addressed. This email and any attachments may contain privileged and confidential information. If you are not the intended recipient, you are notified that you received this email in error and that any reading, retention, use or distribution of this email and attachments is strictly prohibited. If you received this email in error, you are requested to immediately notify us by calling 888-728-3550 or by return email and immediately and permanently delete the email and any attachments. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20240521/c5379930/attachment.htm<http://lists.squid-cache.org/pipermail/squid-users/attachments/20240521/c5379930/attachment.htm>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0.jpg
Type: image/jpeg
Size: 6398 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20240521/c5379930/attachment.jpg<http://lists.squid-cache.org/pipermail/squid-users/attachments/20240521/c5379930/attachment.jpg>>
------------------------------
Subject: Digest Footer
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org<mailto:squid-users at lists.squid-cache.org>
https://lists.squid-cache.org/listinfo/squid-users<https://lists.squid-cache.org/listinfo/squid-users>
------------------------------
End of squid-users Digest, Vol 117, Issue 23
********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20240521/b7a4ba7d/attachment-0001.htm>
More information about the squid-users
mailing list