<div dir="ltr"><div dir="ltr"><div dir="ltr">Thanks Amos,<div><br></div><div>I updated both servers to Squid 4.2 and the issue persisted. I understand what you're saying about the configuration and lack of security - in production, this will be in place. I was removed to try and resolve the issue we encountered.</div><div><br></div><div>After adding the additional configuration to debug_options, I noticed that the issue occurs when the same socket is re-used for a second request.</div><div><br></div><div>I believe what's happening is that Squid is 'forgetting' that it's not connecting to the origin when a persistent connection is in use. Setting server_persistent_connections off resolves the issue, albeit in a way that will likely harm performance.</div><div><br></div><div>Here is an excerpt from cache.log on Squid1</div><div><br></div><div><div>Successful Request:</div><div>2018/09/05 08:20:19.401 kid1| 11,2| client_side.cc(1274) parseHttpRequest: HTTP Client local=<a href="http://1.1.1.1:3128">1.1.1.1:3128</a> remote=<a href="http://3.3.3.3:52210">3.3.3.3:52210</a> FD 14 flags=1</div><div>2018/09/05 08:20:19.401 kid1| 11,2| client_side.cc(1278) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Host: <a href="http://redacted.com">redacted.com</a></div><div>Proxy-Connection: keep-alive</div><div>Cache-Control: max-age=0</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div><br></div><div><br></div><div>----------</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(161) peerSelect: e:=IV/0x21f7e50*2 <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(458) peerSelectFoo: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(463) peerSelectFoo: peerSelectFoo: direct = DIRECT_UNKNOWN (always_direct to be checked)</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(218) peerCheckAlwaysDirectDone: peerCheckAlwaysDirectDone: DENIED</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(458) peerSelectFoo: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(471) peerSelectFoo: peerSelectFoo: direct = DIRECT_UNKNOWN (never_direct to be checked)</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(195) peerCheckNeverDirectDone: peerCheckNeverDirectDone: ALLOWED</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(201) peerCheckNeverDirectDone: direct = DIRECT_NO (never_direct allow)</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(458) peerSelectFoo: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(134) peerSelectIcpPing: peerSelectIcpPing: <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(145) peerSelectIcpPing: peerSelectIcpPing: counted 0 neighbors</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(698) peerGetSomeParent: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(964) peerAddFwdServer: adding FIRSTUP_PARENT/<a href="http://2.2.2.2">2.2.2.2</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(958) peerAddFwdServer: skipping ANY_OLD_PARENT/<a href="http://2.2.2.2">2.2.2.2</a>; have FIRSTUP_PARENT/<a href="http://2.2.2.2">2.2.2.2</a></div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(958) peerAddFwdServer: skipping DEFAULT_PARENT/<a href="http://2.2.2.2">2.2.2.2</a>; have FIRSTUP_PARENT/<a href="http://2.2.2.2">2.2.2.2</a></div><div>2018/09/05 08:20:19.401 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths: Find IP destination for: <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a>' via 2.2.2.2</div><div>2018/09/05 08:20:19.401 kid1| 44,2| peer_select.cc(302) peerSelectDnsPaths: Found sources for '<a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a>'</div><div>2018/09/05 08:20:19.401 kid1| 44,2| peer_select.cc(303) peerSelectDnsPaths:   always_direct = DENIED</div><div>2018/09/05 08:20:19.401 kid1| 44,2| peer_select.cc(304) peerSelectDnsPaths:    never_direct = ALLOWED</div><div>2018/09/05 08:20:19.401 kid1| 44,2| peer_select.cc(314) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=<a href="http://2.2.2.2:3128">2.2.2.2:3128</a> flags=1</div><div>2018/09/05 08:20:19.401 kid1| 44,2| peer_select.cc(317) peerSelectDnsPaths:        timedout = 0</div><div>2018/09/05 08:20:19.401 kid1| 44,3| peer_select.cc(103) ~ps_state: <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a></div><div>2018/09/05 08:20:19.401 kid1| 51,3| fd.cc(198) fd_open: fd_open() FD 9</div><div>2018/09/05 08:20:19.402 kid1| 11,2| http.cc(2260) sendRequest: HTTP Server local=<a href="http://1.1.1.1:45688">1.1.1.1:45688</a> remote=<a href="http://2.2.2.2:3128">2.2.2.2:3128</a> FD 9 flags=1</div><div>2018/09/05 08:20:19.402 kid1| 11,2| http.cc(2261) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com">redacted.com</a></div><div>Via: 1.1 smtp01 (squid/4.2)</div><div>X-Forwarded-For: 3.3.3.3</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>Failed Request, note: FD 9 is not closed/opened between these requests:</div><div>2018/09/05 08:20:22.124 kid1| 11,2| client_side.cc(1274) parseHttpRequest: HTTP Client local=<a href="http://1.1.1.1:3128">1.1.1.1:3128</a> remote=<a href="http://3.3.3.3:52219">3.3.3.3:52219</a> FD 15 flags=1</div><div>2018/09/05 08:20:22.124 kid1| 11,2| client_side.cc(1278) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a> HTTP/1.1</div><div>Host: <a href="http://redacted.com">redacted.com</a></div><div>Proxy-Connection: keep-alive</div><div>Cache-Control: max-age=0</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div><br></div><div><br></div><div>----------</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(161) peerSelect: e:=IV/0x21f7e50*2 <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(458) peerSelectFoo: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(463) peerSelectFoo: peerSelectFoo: direct = DIRECT_UNKNOWN (always_direct to be checked)</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(218) peerCheckAlwaysDirectDone: peerCheckAlwaysDirectDone: DENIED</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(458) peerSelectFoo: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(471) peerSelectFoo: peerSelectFoo: direct = DIRECT_UNKNOWN (never_direct to be checked)</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(195) peerCheckNeverDirectDone: peerCheckNeverDirectDone: ALLOWED</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(201) peerCheckNeverDirectDone: direct = DIRECT_NO (never_direct allow)</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(458) peerSelectFoo: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(134) peerSelectIcpPing: peerSelectIcpPing: <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(145) peerSelectIcpPing: peerSelectIcpPing: counted 0 neighbors</div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(698) peerGetSomeParent: GET <a href="http://redacted.com">redacted.com</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(964) peerAddFwdServer: adding FIRSTUP_PARENT/<a href="http://2.2.2.2">2.2.2.2</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(958) peerAddFwdServer: skipping ANY_OLD_PARENT/<a href="http://2.2.2.2">2.2.2.2</a>; have FIRSTUP_PARENT/<a href="http://2.2.2.2">2.2.2.2</a></div><div>2018/09/05 08:20:22.124 kid1| 44,3| peer_select.cc(958) peerAddFwdServer: skipping DEFAULT_PARENT/<a href="http://2.2.2.2">2.2.2.2</a>; have FIRSTUP_PARENT/<a href="http://2.2.2.2">2.2.2.2</a></div><div>2018/09/05 08:20:22.124 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths: Find IP destination for: <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a>' via 2.2.2.2</div><div>2018/09/05 08:20:22.124 kid1| 44,2| peer_select.cc(302) peerSelectDnsPaths: Found sources for '<a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a>'</div><div>2018/09/05 08:20:22.124 kid1| 44,2| peer_select.cc(303) peerSelectDnsPaths:   always_direct = DENIED</div><div>2018/09/05 08:20:22.124 kid1| 44,2| peer_select.cc(304) peerSelectDnsPaths:    never_direct = ALLOWED</div><div>2018/09/05 08:20:22.124 kid1| 44,2| peer_select.cc(314) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=<a href="http://2.2.2.2:3128">2.2.2.2:3128</a> flags=1</div><div>2018/09/05 08:20:22.124 kid1| 44,2| peer_select.cc(317) peerSelectDnsPaths:        timedout = 0</div><div>2018/09/05 08:20:22.125 kid1| 44,3| peer_select.cc(103) ~ps_state: <a href="http://redacted.com/messages/391/">http://redacted.com/messages/391/</a></div><div>2018/09/05 08:20:22.125 kid1| 11,2| http.cc(2260) sendRequest: HTTP Server local=<a href="http://1.1.1.1:45688">1.1.1.1:45688</a> remote=<a href="http://2.2.2.2:3128">2.2.2.2:3128</a> FD 9 flags=1</div><div>2018/09/05 08:20:22.125 kid1| 11,2| http.cc(2261) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET /messages/391/ HTTP/1.1</div><div>Upgrade-Insecure-Requests: 1</div><div>User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8</div><div>Accept-Encoding: gzip, deflate</div><div>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8</div><div>Cookie: __cfduid=redacted; csrftoken=redacted; sessionid=redacted; _ga=redacted</div><div>AlexaToolbar-ALX_NS_PH: AlexaToolbar/alx-4.0.3</div><div>Host: <a href="http://redacted.com">redacted.com</a></div><div>Via: 1.1 smtp01 (squid/4.2)</div><div>X-Forwarded-For: 3.3.3.3</div><div>Cache-Control: max-age=0</div><div>Connection: keep-alive</div></div><div><br></div></div></div></div>