[squid-users] Squid 3.5.22 Bug when using Mimetype Detection? rep_mime_type

Mon Jan 9 14:31:04 UTC 2017

Hi Amos,

sry that my reply took that long.

I've tested with Squid 3.5.23 on Debian Stretch and the issue is still 
present. Also I was able to create the same issue with the Online OTRS 
Demo website as I had with our internal one.

I did run it with the debug options you gave me. Since you requested 
more info about my config, I stripped a lot out and made sure the issue 
is still the same. XXXXXXXXXXXX indicates that I replaced the whole line 
with XXX.. to ensure no sensitive data is leaked.

So I found out when allowing an IP without authentication and without 
group membership before the real auth is required for everything else, 
then the issue is triggered when Mimetype detection is used. I could'nt 
find a way to avoid the issue. unless I remove the http_access line for 
the target that should be accessible without authentication and without 
group membership. Or I remove the Mimetype Detection lines or better the 
exception for my group.

I hope you can confirm this as a bug or tell me what I made wrong.

This archiv includes: a small squid.conf, cache.log and access.log
https://data-core.org/debug_issue.zip

Hope you can bring light into the dark :)

Best regards,
Enrico

Am 2017-01-03 09:07, schrieb Amos Jeffries:
> On 2017-01-03 07:33, Flashdown wrote:
>> Hello together,
>> 
>> with Squid 3.5.22 I have switched from using a url-regex to Mime Type
>> Detection, which seemed to work nicely until now... :/
>> 
> 
> The thing to be very wary of with this change is that when reply
> blocking the request does still make it through to the server and any
> processing done there still happens.
> 
> With things like ticketing systems that means the tickets and any auth
> related token creating has actually happened, even if the client is
> prevented from being told what the created token was.
> 
> That is a major difference from URL based blocking, which would reject
> before any server contact.
> 
> 
>> OS: Debian Stretch 4.8.0-1-amd64 #1 SMP Debian 4.8.7-1 (2016-11-13)
>> x86_64 GNU/Linux
>> 
> 
> 
> I really advise going to 3.5.23. Several major issues about sending
> wrong replies on certain occasions were fixed there. New .deb should
> be available by now to fix that.
> 
> 
> 
>> I faced the following Situation:
>> 
>> When I globally deny specific mimetypes using a blockfile, then it
>> performs as it should, so only mime types I defined in the block file
>> are getting blocked, so far so good.
>> 
>> When I do an exception for a group I belong to like unblocking
>> application/octet-stream, then I can download files, so the exception
>> works in the first place.
>> acl mime_IT rep_mime_type application/octet-stream
>> http_reply_access allow IT mime_IT
>> 
>> Normally internal targets are excluded from the Proxy using Proxy
>> Exception lists. But I do not get these settings automatically, so my
>> browser did not contain this exception so I was able to discover the
>> following behavior:
>> 
>> The Issue is occuring when browsing to an internal OTRS Web Server via
>> FQDN (It's a web ticket system) through the proxy I get "Access
>> Denied" from the Proxy on all requests. But when browsing to an online
>> OTRS Demo site with the same OTRS version like this one:
>> http://itsm-demo.otrs.com/otrs/index.pl then it works. When I now try
>> again to access the internal OTRS Server through the proxy it works.
>> That's strange, when I now force reload (CTRL+F5 in Firefox) the
>> internal OTRS Ticketsystems webpage, I get the "access denied" again.
>> 
> 
> How is the auth happening between the users and the proxy to find out
> what the groups are?
>  The reply checks may be interferring with the auth replies.
> 
> 
>> When I remove the exception from the global block list for the group I
>> belong to,- here it's IT- then this issue does not occur and the
>> website is accessible like it should.
>> So I just need to comment out these lines:
>> #acl mime_IT rep_mime_type application/octet-stream
>> #http_reply_access allow IT mime_IT
>> 
>> When I add text/html & application/xml to the global block exception,
>> then this error does not occur anymore.
>> acl mime_IT rep_mime_type application/octet-stream text/html 
>> application/xml
>> http_reply_access allow IT mime_IT
>> 
>> So currently I can workaround the issue in 3 different ways:
>> 1. Do not create a global mimetype block exception for groups I belong 
>> to
>> 2. Browse to the start page of an Online Demo OTRS Site and then
>> reload the internal Website
>> 3. Add text/html & application/xml to my exception even if these
>> Mimetypes are not part of the global block list, so they are not
>> supposed to be blocked. (I just looked at the internal website and it
>> just uses text/html and application/xml on the start page (Login Page)
>> so I added them to the exception list for my group and it worked)
>> 
>> Conclusion: When having a global mime type block and unblocking a
>> specific mime type for a specific group, then this group will most
>> propably face issues with mime types that are not supposed to be
>> blocked. So in case of errors, I need to unblock not blocked mimetypes
>> ,too.
>> 
>> 
>> My Squid config for mime type blocking:
>> ---------------------------------------
>> ## Define Default MIMETYPE ERROR Message and global block access list
>> acl block_mimetypes rep_mime_type "/etc/squid/mimetype_blacklist.acl"
>> deny_info ERR_BLOCKED_FILES block_mimetypes
>> 
>> # Configure Execptions
>> acl mime_IT rep_mime_type application/octet-stream
>> http_reply_access allow IT mime_IT
>> 
>> acl mime_SpecialGroup rep_mime_type application/octet-stream
>> http_reply_access allow SpecialGroup mime_SpecialGroup
>> 
>> 
>> #Applying Global MimeType Block
>> http_reply_access deny block_mimetypes
> 
> Any other http_reply_access rules? the implicit-default behaviour may
> be having an effect if there are.
> 
> 
>> ---------------------------------------
>> Squid main config:
>> ---------------------------------------
>> http_port 8080 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=16MB cert=/*********************
>> ssl_bump splice localhost
>> ssl_bump splice SSL_Exclude
> 
> These splice rules may be related to the issue. Any connection that
> gets spliced may be used by other requests without squid being aware
> of them.
> 
>> ssl_bump bump all
>> sslproxy_cert_error allow SSL_TrustedSites
>> sslproxy_cert_error deny all
>> ---------------------------------------
>> 
>> 
>> 
>> Contents of mimetype_blacklist.acl:
>> ---------------------------------------
> 
> That looks okay to me, just remember that it is a regex pattern. So if
> a type entered there matches a sub-string it can block unexpectedly.
> That includes sub-strings in the type parameters.
> 
> 
>> 1483381150.095    186 10.3.101.23 TCP_DENIED_REPLY/403 4493 GET
>> http://otrs-server.**.**.com/otrs/index.pl - HIER_DIRECT/10.2.1.107
>> text/html
>> 1483381150.118      3 10.3.101.23 TCP_DENIED_REPLY/403 4451 GET
>> http://*****-proxy.**.**.com:8080/squid-internal-static/icons/SN.png -
>> HIER_NONE/- text/html
> 
> FWIW: the text/html is the type of the HTML error page being delivered
> for the 403 response. The content-type for the original response
> payload which was blocked is not logged.
> 
> You can debug with "debug_options 11,2" to get a cache.log trace of
> what messages are happening and what the original server response
> headers were.
> 
> Amos
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users