<font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><div>Hi AMos;</div><div>Thanks for clarification, It is working as expected now... Appreciate your support....</div><div><br></div><div>Thanks again.</div><div><br><font size="2">Thanks & Regards<br>
</font><font size="2">Nilesh Suresh Gavali<br></font><font size="2"><br></font></div><br><font color="#990099">-----Forwarded by Nilesh Gavali/MUM/TCS on 10/07/2016 04:52PM -----</font><div class="iNotesHistory iNotesForward" style="padding-left:5px;"><div style="padding-right:0px;padding-left:5px;border-left:solid black 2px;">To: squid-users@lists.squid-cache.org<br>From: squid-users-request@lists.squid-cache.org<br>Sent by: "squid-users" <squid-users-bounces@lists.squid-cache.org><br>Date: 10/06/2016 12:44AM<br>Subject: squid-users Digest, Vol 26, Issue 25<br><br><div><font face="Courier New,Courier,monospace" size="3">Send squid-users mailing list submissions to<br> squid-users@lists.squid-cache.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> <a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><br>or, via email, send a message with subject or body 'help' to<br> squid-users-request@lists.squid-cache.org<br><br>You can reach the person managing the list at<br> squid-users-owner@lists.squid-cache.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of squid-users digest..."<br><br><br>Today's Topics:<br><br> 1. Re: Squid - AD kerberos auth and Linux Server proxy access<br> not working (Amos Jeffries)<br> 2. Re: Caching http google deb files (Hardik Dangar)<br> 3. Re: intercept + IPv6 + IPFilter 5.1 (Egerváry Gergely)<br> 4. Re: Caching http google deb files (Antony Stone)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 6 Oct 2016 06:03:09 +1300<br>From: Amos Jeffries <squid3@treenet.co.nz><br>To: squid-users@lists.squid-cache.org<br>Subject: Re: [squid-users] Squid - AD kerberos auth and Linux Server<br> proxy access not working<br>Message-ID: <4e608076-5ca4-cdac-a5e4-6d0af5106f1d@treenet.co.nz><br>Content-Type: text/plain; charset=utf-8<br><br>On 6/10/2016 5:31 a.m., Nilesh Gavali wrote:<br>> <<NILESH>> here is the compete squid.conf for your reference-<br>> <br>> #<br>> # Recommended minimum configuration:<br>> #### AD SSO Integration #####<br>> #auth_param negotiate program /usr/lib64/squid/squid_kerb_auth -d -s <br>> GSS_C_NO_NAME<br>> auth_param negotiate program /usr/lib64/squid/squid_kerb_auth -s <br>> HTTP/proxy02.CUST.IN@CUST.IN<br>> auth_param negotiate children 20<br>> auth_param negotiate keep_alive on<br>> <br>> acl ad_auth proxy_auth REQUIRED<br>> <br>> #### AD Group membership ####<br>> <br>> <br>> external_acl_type AD_Group ttl=300 negative_ttl=0 children=10 %LOGIN <br>> /usr/lib64/squid/squid_ldap_group -P -R -b "DC=CUST,DC=IN" -D svcproxy -W <br>> /etc/squid/pswd/pswd -f <br>> "(&(objectclass=person)(userPrincipalName=%v)(memberof=cn=%a,ou=InternetAccess,ou=Groups,dc=cust,dc=in))" <br>> -h CUST.IN -s sub -v 3<br>> <br>> acl AVWSUS external AD_Group lgOnlineUpdate<br>> acl windowsupdate dstdomain "/etc/squid/sitelist/infra_update_site"<br>> <br>> acl custUSER external AD_Group lgInternetAccess_custUsers<br>> acl custallowedsite dstdomain "/etc/squid/sitelist/cust_allowed_site"<br>> <br>> #acl SHAVLIK external AD_Group lgShavlikUpdate<br>> acl shavlikupdate dstdomain "/etc/squid/sitelist/shavlik_update_site"<br>> <br><snip defaults><br><br>> # Example rule allowing access from your local networks.<br>> # Adapt to list your (internal) IP networks from where browsing<br>> # should be allowed<br>> acl AVSRVR src 10.50.2.107 # Cloud SEPM Servr<br>> acl SHAVLIK_SRVR src 10.50.2.112 # Shavlik Server(NTLM Only Access)<br>> acl IWCCP01 src 10.55.15.103 # Application access to Worldpay/bottomline <br>> Payment test site.<br><br><snip defaults><br>> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS<br>> #<br>> # Example rule allowing access from your local networks.<br>> # Adapt localnet in the ACL section to list your (internal) IP networks<br>> # from where browsing should be allowed<br>> <br>> #http_access allow test shavlikupdate<br>> http_access allow SHAVLIK_SRVR shavlikupdate<br>> http_access allow AVSRVR windowsupdate<br><br>The "AVWSUS" ACL below requires authentication in order to check the<br>group membership. That is what triggers the 407 response to happen.<br><br>Move the IWCCP01 line up to here and it should stop.<br><br>To make your configuration clearer about which lines need auth and which<br>lines do not you could place the following line right here:<br><br> http_access deny !ad_auth<br><br>All things that do need auth or group names should always go below it.<br>Things that need to avoid auth should always go above it.<br><br><br>> http_access allow AVWSUS windowsupdate<br>> http_access allow IWCCP01<br>> #http_access allow IWCCP01 custallowedsite<br>> http_access allow custUSER custallowedsite<br>> http_access allow ad_auth<br>> # And finally deny all other access to this proxy<br>> http_access deny all<br>> <br><br>Amos<br><br><br><br>------------------------------<br><br>Message: 2<br>Date: Thu, 6 Oct 2016 00:10:46 +0530<br>From: Hardik Dangar <hardikdangar+squid@gmail.com><br>To: Jok Thuau <jok@spikes.com><br>Cc: Squid Users <squid-users@lists.squid-cache.org><br>Subject: Re: [squid-users] Caching http google deb files<br>Message-ID:<br> <CA+sSnVZ+6csWqt60nwwSM0QDSmx+DutqQeXgL-bGtYbFC6WRuw@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Hey Jok,<br><br>Thanks for the suggetion but the big issue with that is i have to download<br>whole repository about ( 80-120 GB ) first and then each week i need to<br>download 20 to 25 GB. We hardly use any of that except few popular repos.<br>big issue i always have with most of them is third party repo's.<br>squid-deb-proxy is quite reliable but again its squid with custom config<br>nothing else and it fails to cache google debs.<br><br>Squid is perfect for me because it can cache things which is requested<br>first time. So next time anybody requests it it's ready. The problem lies<br>when big companies like google and github does not wants us to cache their<br>content and puts various tricks so we can't do that. My issue is same<br>google deb files are downloaded 50 times in same day as apt updates happen<br>and i waste 100s of gb into same content. Country where i live bandwidth is<br>very very costly matter and fast connections are very costly. So this is<br>important for me.<br><br>@Amos,<br><br>I think it's about time Squid needs update of code which can cache use<br>cases like difficult to handle google and github. I am interested to create<br>proposal and will soon share at squid dev and ask for ideas and will try to<br>get official approval so i can build this according to squid standards.<br><br>but before that can you help me with few things.essentially i don't have<br>much experience with C code. as i have worked most of my life with<br>php,python and javascript side. I do know how to write C code but i am not<br>an expert at it. So i want to know if there is any pattern squid follows<br>except the oop pattern. I also want to know workflow of squid i.e. what<br>happens when it receives request and how acls are applied programmatically<br>and how refresh patterns are applied. is there a way i can debug and check<br>if refresh patterns are applied for given url. as well as<br>reply_header_replace has replaced header if i can see those lines in debug<br>it will help me with this. i know debug options can help me but if i turn<br>it with level 9 it is very difficult to go past so many debug entries.<br><br>My idea is to develop a module which will not change any of the squid code<br>but will be loaded only if its called explicitly within squid config. So i<br>want to know is there any piece of code available within squid which<br>behaves similarly just like your archive mode.<br><br><br><br><br>On Wed, Oct 5, 2016 at 9:49 PM, Jok Thuau <jok@spikes.com> wrote:<br><br>> This is sort of off-topic, but have you considered using a deb repo<br>> mirroring software?<br>> (it would mean that you need to update your clients to point to that<br>> rather than google, but that's not really difficult).<br>> software like aptly (aptly.info) are really good about this (though a<br>> little hard to get going in the first place). or a deb-caching proxy<br>> (apt-cacher-ng? squid-deb-proxy?)<br>><br>><br>> On Tue, Oct 4, 2016 at 7:30 AM, Hardik Dangar <<br>> hardikdangar+squid@gmail.com> wrote:<br>><br>>> Wow, i couldn't think about that. google might need tracking data that<br>>> could be the reason they have blindly put vary * header. oh Irony, company<br>>> which talks to all of us on how to deliver content is trying to do such<br>>> thing.<br>>><br>>> I have looked at your patch but how do i enable that ? do i need to write<br>>> custom ACL ? i know i need to compile and reinstall after applying patch<br>>> but what do i need to do exactly in squid.conf file as looking at your<br>>> patch i am guessing i need to write archive acl or i am too naive to<br>>> understand C code :)<br>>><br>>> Also<br>>><br>>> reply_header_replace is any good for this ?<br>>><br>>><br>>> On Tue, Oct 4, 2016 at 7:47 PM, Amos Jeffries <squid3@treenet.co.nz><br>>> wrote:<br>>><br>>>> On 5/10/2016 2:34 a.m., Hardik Dangar wrote:<br>>>> > Hey Amos,<br>>>> ><br>>>> > We have about 50 clients which downloads same google chrome update<br>>>> every 2<br>>>> > or 3 days means 2.4 gb. although response says vary but requested file<br>>>> is<br>>>> > same and all is downloaded via apt update.<br>>>> ><br>>>> > Is there any option just like ignore-no-store? I know i am asking for<br>>>> too<br>>>> > much but it seems very silly on google's part that they are sending<br>>>> very<br>>>> > header at a place where they shouldn't as no matter how you access<br>>>> those<br>>>> > url's you are only going to get those deb files.<br>>>><br>>>><br>>>> Some things G does only make sense whan you ignore all the PR about<br>>>> wanting to make the web more efficient and consider it's a company whose<br>>>> income is derived by recording data about peoples habits and activities.<br>>>> Caching can hide that info from them.<br>>>><br>>>> ><br>>>> > can i hack squid source code to ignore very header ?<br>>>> ><br>>>><br>>>> Google are explicitly saying the response changes. I suspect there is<br>>>> something involving Google account data being embeded in some of the<br>>>> downloads. For tracking, etc.<br>>>><br>>>><br>>>> If you are wanting to test it I have added a patch to<br>>>> <<a href="http://bugs.squid-cache.org/show_bug.cgi?id=4604">http://bugs.squid-cache.org/show_bug.cgi?id=4604</a>> that should implement<br>>>> archival of responses where the ACLs match. It is completely untested by<br>>>> me beyond building, so YMMV.<br>>>><br>>>> Amos<br>>>><br>>>><br>>><br>>> _______________________________________________<br>>> squid-users mailing list<br>>> squid-users@lists.squid-cache.org<br>>> <a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><br>>><br>>><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="http://lists.squid-cache.org/pipermail/squid-users/attachments/20161006/98c8f171/attachment-0001.html">http://lists.squid-cache.org/pipermail/squid-users/attachments/20161006/98c8f171/attachment-0001.html</a>><br><br>------------------------------<br><br>Message: 3<br>Date: Wed, 5 Oct 2016 20:49:54 +0200<br>From: Egerváry Gergely <gergely@egervary.hu><br>To: squid-users@lists.squid-cache.org<br>Subject: Re: [squid-users] intercept + IPv6 + IPFilter 5.1<br>Message-ID: <57F54B52.4000803@egervary.hu><br>Content-Type: text/plain; charset=utf-8<br><br>>> Should "intercept" work with IPv6 on NetBSD 7-STABLE and IPFilter 5.1?<br><br>Okay, we have "fixed" Squid interception, and IPFilter in the kernel,<br>and now it's working good. But did we do it in the right way?<br><br>While reading ip_nat.c in IPFilter, I found that SIOCGNATL - and its<br>function called ipf_nat_lookupredir() - is a frontend to two functions:<br>ipf_nat_inlookup() and ipf_nat_outlookup().<br><br>We are now calling SIOCGNATL to use ipf_nat_outlookup(). But should not<br>we call it to use ipf_nat_inlookup() instead?<br><br>In Squid, we are working with 3 different addresses:<br>- source IP:port of the connection (browser client)<br>- real destination IP:port (the target web server)<br>- interception destination IP:port (Squid itself)<br><br>In IPFilter, the terminology is different: "real" refers to the<br>original source, not the original destination.<br><br>In my understanding, on redirect (RDR) rules, where we know the<br>original source address and the rewrited destination address, we should<br>use ipf_nat_inlookup() to get the original destination address.<br><br>ipf_nat_outlookup() should be used on source-NAT (MAP) scenarios,<br>what we don't need for Squid.<br><br>If that's true, IPFilter was correct - we have to revert our IPFilter<br>patches - and modify Intercept.cc instead.<br><br>See IPFilter source code comments below:<br><br>========<br>Function: ipf_nat_inlookup<br>Returns: nat_t* - NULL == no match, else pointer to matching NAT entry<br>Parameters:<br>fin(I) - pointer to packet information<br>flags(I) - NAT flags for this packet<br>p(I) - protocol for this packet<br>src(I) - source IP address<br>mapdst(I) - destination IP address<br><br>Lookup a nat entry based on the mapped destination ip address/port<br>and real source address/port. We use this lookup when receiving a<br>packet, we're looking for a table entry, based on the destination<br>address.<br><br>========<br>Function: ipf_nat_outlookup<br>Returns: nat_t* - NULL == no match, else pointer to matching NAT entry<br>Parameters:<br>fin(I) - pointer to packet information<br>flags(I) - NAT flags for this packet<br>p(I) - protocol for this packet<br>src(I) - source IP address<br>dst(I) - destination IP address<br>rw(I) - 1 == write lock on held, 0 == read lock.<br><br>Lookup a nat entry based on the source 'real' ip address/port<br>and destination address/port. We use this lookup when sending a packet<br>out, we're looking for a table entry, based on the source address.<br><br>========<br><br>See full ip_nat.c source code here:<br><br><a href="http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/external/bsd/ipf/netinet/ip_nat.c?rev=1.16&content-type=text/x-cvsweb-markup">http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/external/bsd/ipf/netinet/ip_nat.c?rev=1.16&content-type=text/x-cvsweb-markup</a><br><br>Thank you,<br>-- <br>Gergely EGERVARY<br><br><br><br>------------------------------<br><br>Message: 4<br>Date: Wed, 5 Oct 2016 21:13:21 +0200<br>From: Antony Stone <Antony.Stone@squid.open.source.it><br>To: squid-users@lists.squid-cache.org<br>Subject: Re: [squid-users] Caching http google deb files<br>Message-ID: <201610052113.21686.Antony.Stone@squid.open.source.it><br>Content-Type: Text/Plain; charset="iso-8859-15"<br><br>On Wednesday 05 October 2016 at 20:40:46, Hardik Dangar wrote:<br><br>> Hey Jok,<br>> <br>> Thanks for the suggetion but the big issue with that is i have to download<br>> whole repository about ( 80-120 GB ) first and then each week i need to<br>> download 20 to 25 GB.<br><br>This is not true for apt-cacher-ng. You install it and it does nothing. You <br>point your Debian (or Ubuntu, maybe other Debian-derived distros as well, I <br>haven't tested) machines at it as their APT proxy, and it then caches content <br>as it gets requested and downloaded. Each machine which requests a new <br>package causes that package to get cached. Each machine which requests a <br>cached package gets the local copy (unless it's been updated, in which case <br>the cache gets updated).<br><br>> We hardly use any of that except few popular repos.<br>> big issue i always have with most of them is third party repo's.<br>> squid-deb-proxy is quite reliable but again its squid with custom config<br>> nothing else and it fails to cache google debs.<br>> <br>> Squid is perfect for me because it can cache things which is requested<br>> first time. So next time anybody requests it it's ready.<br><br>This is exactly how apt-cacher-ng works. I use it myself and I would <br>recommend you investigate it further for this purpose.<br><br>> The problem lies when big companies like google and github does not wants us<br>> to cache their content and puts various tricks so we can't do that.<br><br>That's a strange concept for a Debian repository (even third-party).<br><br>Are you sure you're talking about repositories and not just isolated .deb <br>files?<br><br><br>Antony.<br><br>-- <br>A user interface is like a joke.<br>If you have to explain it, it didn't work.<br><br> Please reply to the list;<br> please *don't* CC me.<br><br><br>------------------------------<br><br>Subject: Digest Footer<br><br>_______________________________________________<br>squid-users mailing list<br>squid-users@lists.squid-cache.org<br><a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><br><br><br>------------------------------<br><br>End of squid-users Digest, Vol 26, Issue 25<br>*******************************************<br></font></div></squid-users-bounces@lists.squid-cache.org></div></div></font><p>=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you</p>
<p></p>