<div dir="ltr">HI Amos,<div><br></div><div>Thank you for responding.</div><div><br></div><div>To clarify, when I referred to HTTPS requests, I was referring to CONNECT requests - I should have been more clear, my apologies. No authentication is being performed by either server, so I'm not sure what you're seeing in the logs that relates to that.</div><div><br></div><div>CONNECT requests are logged correctly on both squid servers and appear to operate correctly for every request.</div><div><br></div><div>Interestingly, I was mistaken before. It's not the host header that's missing - that's still present. It's the full URI within the GET request.</div><div><br></div><div>As requested, here is all the information:</div><div><br></div><div><b>Squid1 version and build information:</b></div><div><div>Squid Cache: Version 3.5.12</div><div>Service Name: squid</div><div>Ubuntu linux</div><div>configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' 'BUILDCXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-build-info=Ubuntu linux' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'</div></div><div><br></div><div><b>Squid2 version and build information:</b></div><div><div>Squid Cache: Version 3.5.27</div><div>Service Name: squid</div><div>Ubuntu linux</div><div><br></div><div>This binary uses OpenSSL 1.0.2g 1 Mar 2016. For legal restrictions on distribution see <a href="https://www.openssl.org/source/license.html" target="_blank">https://www.openssl.org/source/license.html</a></div><div><br></div><div>configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' 'BUILDCXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed' 'CXX=g++' 'CC=gcc' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--with-openssl' '--enable-ssl' '--enable-ssl-crtd' '--enable-build-info=Ubuntu linux' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'</div></div><div><br></div><div><br></div><div><br></div><div>Here is a verbatim copy of both squid.conf files, with sensitive information replaced:</div><div><br></div><div><div><b>Squid1:</b></div><div>http_port 3128 name=port_3128</div><div>http_access allow all</div><div>nonhierarchical_direct off</div><div><br></div><div>acl port_3128_acl myportname port_3128</div><div>always_direct deny port_3128_acl</div><div>never_direct allow port_3128_acl</div><div><br></div><div># 3128</div><div>cache_peer 2.2.2.2 parent 3128 0 no-query proxy-only default name=proxy3128</div><div>cache_peer_access proxy3128 allow port_3128_acl</div><div>cache_peer_access proxy3128 deny all</div><div>debug_options 11,2</div><div><br></div><div><br></div><div><b>Squid2:</b></div><div>http_access allow all</div><div>http_port 3128</div><div>debug_options 11,2</div><div><br></div><div><br></div><div>And here is a copy of the cache.log for a failed request:</div><div><br></div><div><b>Squid1:</b></div><div>----------</div><div>2018/09/03 13:36:43| pinger: Initialising ICMP pinger ...</div><div>2018/09/03 13:36:45.088 kid1| 11,2| client_side.cc(2346) parseHttpRequest: HTTP Client local=<a href="http://1.1.1.1:3128/" target="_blank">1.1.1.1:3128</a> remote=<a href="http://3.3.3.3:52250/" target="_blank">3.3.3.3:52250</a> FD 8 flags=1</div><div>2018/09/03 13:36:45.088 kid1| 11,2| client_side.cc(2347) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/" target="_blank">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Proxy-Connection: keep-alive</div><div>Cache-Control: max-age=0</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div><br></div><div><br></div><div>----------</div><div>2018/09/03 13:36:45.088 kid1| 11,2| http.cc(2234) sendRequest: HTTP Server local=<a href="http://1.1.1.1:55718/" target="_blank">1.1.1.1:55718</a> remote=<a href="http://2.2.2.2:3128/" target="_blank">2.2.2.2:3128</a> FD 14 flags=1</div><div>2018/09/03 13:36:45.089 kid1| 11,2| http.cc(2235) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET /messages/391/ HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Via: 1.1 Squid1 (squid/3.5.12)</div><div>X-Forwarded-For: 3.3.3.3</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div><br></div><div><b>Squid2:</b></div><div>2018/09/03 13:36:45.089 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=<a href="http://2.2.2.2:3128/" target="_blank">2.2.2.2:3128</a> remote=<a href="http://1.1.1.1:55718/" target="_blank">1.1.1.1:55718</a> FD 15 flags=1</div><div>2018/09/03 13:36:45.089 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET /messages/391/ HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Via: 1.1 Squid1 (squid/3.5.12)</div><div>X-Forwarded-For: 3.3.3.3</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div><div>2018/09/03 13:36:45.089 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=<a href="http://2.2.2.2:3128/" target="_blank">2.2.2.2:3128</a> remote=<a href="http://1.1.1.1:55718/" target="_blank">1.1.1.1:55718</a> FD 15 flags=1</div><div>2018/09/03 13:36:45.089 kid1| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY:</div><div>---------</div><div>HTTP/1.1 400 Bad Request</div><div>Server: squid/3.5.27</div><div>Mime-Version: 1.0</div><div>Date: Mon, 03 Sep 2018 13:36:45 GMT</div><div>Content-Type: text/html;charset=utf-8</div><div>Content-Length: 3556</div><div>X-Squid-Error: ERR_INVALID_URL 0</div><div>Vary: Accept-Language</div><div>Content-Language: en</div><div>X-Cache: MISS from Squid2</div><div>X-Cache-Lookup: NONE from Squid2:3128</div><div>Via: 1.1 Squid2 (squid/3.5.27)</div><div>Connection: close</div><div><br></div><div><br></div><div><br></div><div>And the same again for a successful request:</div><div><b><br></b></div><div><b>Squid1:</b></div><div>2018/09/03 13:46:22.850 kid1| 11,2| client_side.cc(2346) parseHttpRequest: HTTP Client local=<a href="http://1.1.1.1:3128/" target="_blank">1.1.1.1:3128</a> remote=<a href="http://3.3.3.3:53182/" target="_blank">3.3.3.3:53182</a> FD 8 flags=1</div><div>2018/09/03 13:46:22.850 kid1| 11,2| client_side.cc(2347) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/" target="_blank">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Proxy-Connection: keep-alive</div><div>Cache-Control: max-age=0</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div><br></div><div><br></div><div>----------</div><div>2018/09/03 13:46:22.852 kid1| 11,2| http.cc(2234) sendRequest: HTTP Server local=<a href="http://1.1.1.1:55798/" target="_blank">1.1.1.1:55798</a> remote=<a href="http://2.2.2.2:3128/" target="_blank">2.2.2.2:3128</a> FD 12 flags=1</div><div>2018/09/03 13:46:22.852 kid1| 11,2| http.cc(2235) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/" target="_blank">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Via: 1.1 Squid1 (squid/3.5.12)</div><div>X-Forwarded-For: 3.3.3.3</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div><b>Squid2:</b></div><div>2018/09/03 13:46:22.853 kid1| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=<a href="http://2.2.2.2:3128/" target="_blank">2.2.2.2:3128</a> remote=<a href="http://1.1.1.1:55798/" target="_blank">1.1.1.1:55798</a> FD 12 flags=1</div><div>2018/09/03 13:46:22.853 kid1| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/" target="_blank">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Via: 1.1 Squid1 (squid/3.5.12)</div><div>X-Forwarded-For: 3.3.3.3</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div><div>2018/09/03 13:46:22.859 kid1| 11,2| http.cc(2229) sendRequest: HTTP Server local=<a href="http://2.2.2.2:58376/" target="_blank">2.2.2.2:58376</a> remote=<a href="http://4.4.4.4/" target="_blank">4.4.4.4:80</a> FD 10 flags=1</div><div>2018/09/03 13:46:22.859 kid1| 11,2| http.cc(2230) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET /messages/391/ HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com/" target="_blank">redacted.com</a></div><div>Via: 1.1 Squid1 (squid/3.5.12), 1.1 Squid2 (squid/3.5.27)</div><div>X-Forwarded-For: 3.3.3.3, 1.1.1.1</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Sep 3, 2018 at 5:22 PM Amos Jeffries <<a href="mailto:squid3@treenet.co.nz">squid3@treenet.co.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 3/09/18 9:32 PM, Michael Thomas wrote:<br>
> I'm trying to figure out this weird intermittent issue.<br>
> <br>
> I have two squid servers running, non-caching, non-transparent.<br>
> <br>
> Client -> Squid1 -> Squid2 -> Internet<br>
> <br>
> All HTTPS requests work as expected, but randomly, about 50% of the<br>
> time, HTTP requests fail.<br>
> <br>
> The reason for the failure is that the first squid server (Squid1) is<br>
> not correctly forwarding the request to the second. It is stripping away<br>
> the hostname, and obviously the second squid server has no clue what to<br>
> do with it then!<br>
<br>
Can you please provide:<br>
<br>
* details of your Squid version(s), the output of "squid -v" contains that.<br>
<br>
* a copy of the HTTP message actually received by Squid2.<br>
<br>
* a copy of the same HTTP message as it is leaving Squid1.<br>
<br>
"debug_options 11,2" can provide that in your cache.log.<br>
<br>
<br>
Also, is there any sign of CONNECT messages requesting tunnels to Squid2<br>
being received at Squid1 ?<br>
<br>
<br>
Your logs contain signs of authentication being performed, and you<br>
mention HTTPS being handled by these proxies. Yet the config showed no<br>
signs of either having been configured. Can you please provide the<br>
actual config where you are seeing this behaviour and generating the<br>
logs from.<br>
A minimal is fine, but it does need to be generating the logs, etc for<br>
the info to correlate correctly.<br>
<br>
<br>
Amos<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div>