[squid-users] What does this error mean?
Patrick Flaherty
vze2k3sa at verizon.net
Tue Nov 10 12:43:20 UTC 2015
-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
Behalf Of squid-users-request at lists.squid-cache.org
Sent: Tuesday, November 10, 2015 5:09 AM
To: squid-users at lists.squid-cache.org
Subject: squid-users Digest, Vol 15, Issue 26
Send squid-users mailing list submissions to
squid-users at lists.squid-cache.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.squid-cache.org/listinfo/squid-users
or, via email, send a message with subject or body 'help' to
squid-users-request at lists.squid-cache.org
You can reach the person managing the list at
squid-users-owner at lists.squid-cache.org
When replying, please edit your Subject line so it is more specific than
"Re: Contents of squid-users digest..."
Today's Topics:
1. Re: What does this error mean? (Amos Jeffries)
2. Re: Subject: Re: authentication of every GET request from
part of URL? (Sreenath BH)
3. Re: Subject: Re: authentication of every GET request from
part of URL? (Amos Jeffries)
4. Assert, followed by shm_open() fail. (Steve Hill)
5. Help, long response time(2 seconds) in squid! (=?GBK?B?0OzTwL2h?=)
6. cache peer problem with Https only !! (Ahmad Alzaeem)
----------------------------------------------------------------------
Message: 1
Date: Tue, 10 Nov 2015 05:38:42 +1300
From: Amos Jeffries <squid3 at treenet.co.nz>
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] What does this error mean?
Message-ID: <5640CC12.7080907 at treenet.co.nz>
Content-Type: text/plain; charset=utf-8
On 10/11/2015 1:54 a.m., Yuri Voinov wrote:
>
> This mean that client sent RST packet. You can ignore this error.
>
Well, its not always the client sending it. Could be a NAT device somewhere
hitting some timeout or connnection limit and aborting idle connections.
If it is occuring a lot then it might be worth investigating. Having long
live TCP connections just sitting around unused for very long periods is not
very good for resource utilization. Hardware device tables overflowing in a
NAT or router device is also pretty bad.
Amos
Hello,
Thank you Yuri and Amos for your responses. The log entry points to the
Squid IP and one of my client IP addresses. They are connect through an
'Internal' network using Oracle's Virtual Box. So there is nothing in
between meaning it's the client connecting to Squid directly. So it must be
out client software creating a RST or the Windows Stack itself.
Thanks
Patrick
------------------------------
Message: 2
Date: Mon, 9 Nov 2015 22:42:08 +0530
From: Sreenath BH <bhsreenath at gmail.com>
To: Alex Rousskov <rousskov at measurement-factory.com>
Cc: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Subject: Re: authentication of every GET
request from part of URL?
Message-ID:
<CALgKBSkUH9JUSB5-ZetZitnUYhW0Keik3QQj2rk-wsjC=mM+Bg at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Hi Alex,
thanks for your detailed asnwers.
Here are more details.
1. If the URL does not have any token, we would like to send an error
message back to the browser/client, without doing a cache lookup, or going
to backend apache server.
2. If the token is invalid (that is we can't find it in a database), that
means we can not serve data. In this case we would like to send back a HTTP
error (something like a 401 or 404, along with a more descriptive message)
3. If the token is valid(found), remove the token from the URL, and use
remaining part of URL as the key to look in Squid cache.
4. If found return that data, along with proper HTTP status code.
5. If cache lookup fails(not cached), send HTTP request to back-end apache
server (removing the token), get returned result, store in cache, and return
to client/browser.
I read about ACL helper programs, and it appears I can do arbitrary
validations in it, so should work.
Is it correct to assume that the external ACL code runs before url
rewriting?,
Does the URL rewriter run before a cache lookup?
thanks,
Sreenath
On 11/8/15, Alex Rousskov <rousskov at measurement-factory.com> wrote:
> On 11/08/2015 06:34 AM, Sreenath BH wrote:
>
>> Is there a way for me to invoke some custom code for every request
>> that Squid receives?
>
> Yes, there are several interfaces, including a built-in ACL, an
> external ACL helper, a URL rewriter, an eCAP/ICAP service. Roughly
> speaking, the former ones are easier to use and the latter ones are more
powerful.
>
>
>> That script would do the following:
>>
>> 1. Extract part of the URL(the token) and look up in a database to
>> see if it is valid.
>> If valid, proceed to lookup cached object, other wise go to
>> back-end fetch, etc.
>> 2. If the token is not found in database, return with an error, so
>> that Squid can send back a not found type (some HTTP error) of
>> response.
>
> If the above are your requirements, avoid the word "authentication"
> might help. It confuses people into thinking you want something far
> more complex.
>
>
> The validation in step #1 can be done by an external ACL. However, you
> probably forgot to mention that the found token should be removed from
> the URL. To edit the URL, you need to use a URL rewriter or an
> eCAP/ICAP service.
>
> Everything else can be done by built-in ACLs unless you need to serve
> very custom error messages. In the latter case, you will need an eCAP
> or ICAP service.
>
> However, if "go to back-end fetch" means loading response from some
> storage external to Squid without using HTTP, then you need an eCAP or
> ICAP service to do that fetching.
>
> I recommend that you clarify these parts of your specs:
>
> What do you want to do when the token is not found in the URL?
>
> What do you want to do when an invalid token is found in the URL?
>
> Will sending a response using a simple template filled with some basic
> request details suffice when a valid token is not found in the database?
>
>
> HTH,
>
> Alex.
>
>
>
>> On 7/11/2015 1:33 a.m., Sreenath BH wrote:
>>> Hi
>>> I am very new to Squid, and think have a strange requirement.
>>> We want to serve cached content only if the client has been
>>> authenticated before.
>>> Since we don't expect the client software to send any information in
>>> headers, we embed a token in the URL that we present to the user.
>>>
>>
>> Um, you know how sending username and password in plain-text Basic
>> auth headers is supposed to be the worst form of security around?
>>
>> It's not quite. Sending credentials in the URL is worse. Even if its
>> just an encoded token.
>>
>> Why are you avoiding actual HTTP authentication?
>>
>> Why be so actively hostile to every other cache in existence?
>>
>>
>>> So when the client s/w uses this URL, we want to extract the token
>>> from URL and do a small database query to ensure that the token is
>>> valid.
>>>
>>> This is in accelerator mode.
>>> Is it possible to use something similar to basic_fake_auth and put
>>> my code there that does some database query?
>>
>> The "basic_..._auth" parts of that helpers name mean that it performs
>> HTTP Basic authentication.
>>
>> The "fake" part means that it does not perform any kind of validation.
>>
>> All of the text above has been describing how you want to perform
>> actions which are the direct opposite of everything basic_fake_auth does.
>>
>>> If the query fails, we don't return the cached content?
>>
>> What do you want to be delivered instead?
>>
>> Amos
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>
>
------------------------------
Message: 3
Date: Tue, 10 Nov 2015 06:42:27 +1300
From: Amos Jeffries <squid3 at treenet.co.nz>
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Subject: Re: authentication of every GET
request from part of URL?
Message-ID: <5640DB03.8030200 at treenet.co.nz>
Content-Type: text/plain; charset=utf-8
On 10/11/2015 6:12 a.m., Sreenath BH wrote:
> Hi Alex,
>
> thanks for your detailed asnwers.
>
> Here are more details.
> 1. If the URL does not have any token, we would like to send an error
> message back to the browser/client, without doing a cache lookup, or
> going to backend apache server.
>
> 2. If the token is invalid (that is we can't find it in a database),
> that means we can not serve data. In this case we would like to send
> back a HTTP error (something like a 401 or 404, along with a more
> descriptive message)
>
All of the above is external_acl_type helper operations.
> 3. If the token is valid(found), remove the token from the URL, and
> use remaining part of URL as the key to look in Squid cache.
>
> 4. If found return that data, along with proper HTTP status code.
The above is url_rewrite_program operations.
> 5. If cache lookup fails(not cached), send HTTP request to back-end
> apache server (removing the token), get returned result, store in
> cache, and return to client/browser.
And that part is normal caching. Squid will do it by default.
Except the "removing the token" part. Which was done at step #4 already, so
has no relevance here at step #5.
>
> I read about ACL helper programs, and it appears I can do arbitrary
> validations in it, so should work.
> Is it correct to assume that the external ACL code runs before url
> rewriting?,
The http_access tests are run before re-writing.
If the external ACL is one of those http_access tests the answer is yes.
>
> Does the URL rewriter run before a cache lookup?
Yes.
Although, please note that despite this workaround for your cache. It really
is *only* your proxy which will work nicely. Every other cache on the planet
will see your applications URLs are being unique and needing different
caching slots.
This not only wastes cache space for them, but also forces them to pass
extra traffic in the form of full-object fetches at your proxy. Which raises
the bandwidth costs for both them and you far beyond what proper header
based authentication or authorization would.
As the other sysadmin around the world notice this unnecessarily raised cost
they will start to hack their configs to force-cache the responses from your
application. Which will bypass your protection system entirely since your
proxy may not not even see many of the requests.
The earlier you can get the application re-design underway to remove the
credentials token from URL, the earlier the external problems and costs will
start to dsappear.
Amos
------------------------------
Message: 4
Date: Mon, 9 Nov 2015 17:58:59 +0000
From: Steve Hill <steve at opendium.com>
To: squid-users at lists.squid-cache.org
Subject: [squid-users] Assert, followed by shm_open() fail.
Message-ID: <5640DEE3.1030003 at opendium.com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On Squid 3.5.11 I'm seeing occasional asserts:
2015/11/09 13:45:21 kid1| assertion failed: DestinationIp.cc:41:
"checklist->conn() && checklist->conn()->clientConnection != NULL"
More concerning though, is that usually when a Squid process crashes, it is
automatically restarted, but following these asserts I'm often seeing:
FATAL: Ipc::Mem::Segment::open failed to
shm_open(/squidnocache-squidnocache-cf__metadata.shm): (2) No such file or
directory
After this, Squid is still running, but won't service requests and requires
a manual restart.
Has anyone seen this before?
Cheers.
--
- Steve Hill
Technical Director
Opendium Limited http://www.opendium.com
Direct contacts:
Instant messager: xmpp:steve at opendium.com
Email: steve at opendium.com
Phone: sip:steve at opendium.com
Sales / enquiries contacts:
Email: sales at opendium.com
Phone: +44-1792-824568 / sip:sales at opendium.com
Support contacts:
Email: support at opendium.com
Phone: +44-1792-825748 / sip:support at opendium.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: steve.vcf
Type: text/x-vcard
Size: 283 bytes
Desc: not available
URL:
<http://lists.squid-cache.org/pipermail/squid-users/attachments/20151109/392
307ce/attachment-0001.vcf>
------------------------------
Message: 5
Date: Tue, 10 Nov 2015 16:49:27 +0800
From: "=?GBK?B?0OzTwL2h?=" <yongjianchn at sina.com>
To: "squid-users" <squid-users at lists.squid-cache.org>
Subject: [squid-users] Help, long response time(2 seconds) in squid!
Message-ID: <20151110084927.3A98EDD8001 at webmail.sinamail.sina.com.cn>
Content-Type: text/plain; charset=GBK
Hi, All:
I tried to use squid as a web cache server today, but when I test it with
http_load, I found squid may have a latency of 2 seconds in some cases.
Someone help me? Thanks!
The test is
-------
http_load -parallel 1 -seconds 20 url.txt # the content in url.txt is
`http://10.210.136.51:3128/xyj/1`
------
config for squid is
-----
http_port 3128 accel vhost vport
cache_peer 10.210.136.51 parent 8888 0
# use mem only
cache_mem 1000 MB
-----
The access log
-----
1447142491.264 39 10.210.136.54 TCP_MEM_HIT_ABORTED/200 4165 GET
http://10.210.136.51:3128/xyj/1 - HIER_NONE/- application/octet-stream
1447142491.283 37 10.210.136.54 TCP_MEM_HIT_ABORTED/200 4165 GET
http://10.210.136.51:3128/xyj/1 - HIER_NONE/- application/octet-stream
1447142493.288 2023 10.210.136.54 TCP_MEM_HIT_ABORTED/200 4165 GET
http://10.210.136.51:3128/xyj/1 - HIER_NONE/- application/octet-stream #
2023 ms! why?
1447142493.307 2023 10.210.136.54 TCP_MEM_HIT_ABORTED/200 4165 GET
http://10.210.136.51:3128/xyj/1 - HIER_NONE/- application/octet-stream #
2023 ms! why?
1447142493.326 38 10.210.136.54 TCP_MEM_HIT_ABORTED/200 4165 GET
http://10.210.136.51:3128/xyj/1 - HIER_NONE/- application/octet-stream
1447142493.348 40 10.210.136.54 TCP_MEM_HIT_ABORTED/200 4165 GET
http://10.210.136.51:3128/xyj/1 - HIER_NONE/- application/octet-stream
-----
------------------------------
Message: 6
Date: Tue, 10 Nov 2015 12:08:50 +0300
From: "Ahmad Alzaeem" <ahmed.zaeem at netstream.ps>
To: <squid-users at lists.squid-cache.org>
Subject: [squid-users] cache peer problem with Https only !!
Message-ID: <000b01d11b97$6a894e10$3f9bea30$@netstream.ps>
Content-Type: text/plain; charset="utf-8"
Hi im using pfsense with cache peer
Squid version is 3.4.10
I have peer proxy on port 80 and I can use it with http and https
Now if I use pfsense in the middle and let pfsense go to remote proxy
(10.12.0.32 port 80 )
And I get internt from the pfsense proxy
I only have http websites working !!!
But https websites don't work
Any help ?
Here is my pfsnese config :
# This file is automatically generated by pfSense
# Do not edit manually !
http_port 172.23.101.253:3128
icp_port 0
dns_v4_first on
pid_filename /var/run/squid/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/pbi/squid-amd64/local/etc/squid/icons
visible_hostname mne
cache_mgr azaeem at mne.ps
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable off
pinger_program /usr/pbi/squid-amd64/local/libexec/squid/pinger
logfile_rotate 2
debug_options rotate=2
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src 172.23.101.0/24
forwarded_for off
via off
httpd_suppress_version_string on
uri_whitespace strip
acl dynamic urlpath_regex cgi-bin ?
cache deny dynamic
cache_mem 64 MB
maximum_object_size_in_memory 256 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 4 MB
cache_dir ufs /var/squid/cache 100 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320
#Remote proxies
# Setup some default acls
# From 3.2 further configuration cleanups have been done to make things
easier and safer. The manager, localhost, and to_localhost ACL definitions
are now built-in.
# acl localhost src 127.0.0.1/32
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 3128 3127
1025-65535
acl sslports port 443 563
# From 3.2 further configuration cleanups have been done to make things
easier and safer. The manager, localhost, and to_localhost ACL definitions
are now built-in.
#acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports
# Always allow localhost connections
# From 3.2 further configuration cleanups have been done to make things
easier and safer.
# The manager, localhost, and to_localhost ACL definitions are now built-in.
# http_access allow localhost
request_body_max_size 0 KB
delay_access 1 allow allsrc
# Reverse Proxy settings
# Custom options before auth
dns_nameservers 8.8.8.8 10.12.0.33
cache_peer 10.12.0.32 parent 80 0 no-query no-digest no-tproxy proxy-only
# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc
cheers
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.squid-cache.org/pipermail/squid-users/attachments/20151110/9d3
a01a2/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
------------------------------
End of squid-users Digest, Vol 15, Issue 26
*******************************************
More information about the squid-users
mailing list