[squid-users] forwarded_for problems log client ip apache 2.4

Fri Jan 29 00:37:55 UTC 2016

Hey,

It is off-topic but I do have a setup that works with this and it 
depends on couple things.
The first thing is that if it's not clear to me how you use the squid 
and the apache services together.
You squid.conf shows two ports that both are in forward mode rather then 
reverse mode which the setup would be pretty different by the proxy 
functionality.

The basic scenario that the proxy provides a Forwarded-For header is 
when it is when it has someone to inform about it such as internal 
service or a reverse proxy.
When it's a parent or sibling proxy then the forwarded_for option should 
be in "on" mode. Just notice that if you have some WAN connection in the 
middle then without an HTTPS secured connection it would probably be 
meaningless for a service unless it has a specific set of IP addresses 
that it trusts.(unless the service has a reverse or forward dns 
resolution mechanism that will "automatically" add\identify origin 
sources by the domain name A\AAAA\CNAME records)

Currently squid doesn't have the option to use some ACLs in order to 
decide to who\what he will send the forwarded-for headers which might be 
important in use cases like I think yours is.
Basically based on the assumption that this proxy doesn't have any child 
proxy services the right way to implement the forwarded-for is using the 
"truncate" and not the "on" option to avoid any sort of ip 
impersonations.(since any client can add "X-Forwarded-For: X.Y.Z.I" to 
the request).

As for the apache remote_ip module and squid it is very simple to test, 
a simple tcpdump on the proxy or the apache server with some filters 
will show you what is on the wire and what the apache server receives.

The main question is what you do see in your apache logs and what you 
expect to appear in them?
I can lend you my working remoteip modules settings:
RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 192.168.10.10

 From the docs at:
- 
https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteiptrustedproxy
- 
https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteipinternalproxy

I assume that you are wrongly using the
"RemoteIPTrustedProxy" directive to trust this proxy about internal 10/8 
192.168/16 etc addresses spaces which it cannot(as documented).

So my suggestion is to try the "RemoteIPInternalProxy" instead of 
"RemoteIPTrustedProxy".

Notice that remote_ip is IP related module and will not result in 
reporting any sort of domain name in the access logs, resulting in such 
log format will be an apache log related subject which I have never used.

Currently the log format I am using in apache is:
LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" 
%v" combined_vhost

Which will show the remote_ip module resolved IP address and will report 
the target vhost in the end of the log line so it won't break some log 
parsing tools.

All The Bests,
Eliezer

* I wrote this long email partially as documentation of the subject for 
later use in searches.

On 28/01/2016 15:38, L.P.H. van Belle wrote:
> Hai,
>
> I having some troubles to get my client ip (and/or hostname) logged in
> my apache webserver.
>
> I do think this is something in my squid setup, but i can find it..
>
> So if anyone can help me out a bit, would be great.
>
> I’ve tested with the forwarded_for options tried all options here.
>
> http://www.squid-cache.org/Versions/v3/3.5/cfgman/forwarded_for.html
>
> im using Debian Jessie, Apache 2.4 with mod_remoteip
>
> http://httpd.apache.org/docs/current/mod/mod_remoteip.html#remoteipheader
>
> My settings for remoteip   ( and yes the modules is enabled )
>
> a2query -m | grep remote
>
> remoteip (enabled by site administrator)
>
> <IfModule mod_remoteip>
>
>      # for remote proxy setup
>
>      RemoteIPHeader X-Forwarded-For
>
>      # for cluster setup
>
>      #RemoteIPHeader X-Real-IP
>
>      RemoteIPTrustedProxy 127.0.0.1/8
>
>      RemoteIPTrustedProxy 192.168.x.x/24
>
>      RemoteIPTrustedProxy 192.168.x.x/24
>
>      RemoteIPTrustedProxy prxy1.internal.domain.tld
>
>      RemoteIPTrustedProxy prxy2.internal.domain.tld
>
> #original : LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\"
> \"%{User-Agent}i\"" combined
>
> LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\""
> combined
>
> </IfModule>
>
> any tips on howto debug this, i did find lots of things with google, but
> none worked for me.
>
> This is my (sanitized)  squid config, default values are not shown.
>
> Any improvement tips are welkom  ;-) but my bigest problem now is
> getting the ip of the client in my webserver logs.
>
> Greetz,
>
> Louis
>
> # squid 3.5.12 config
>
> auth_param negotiate program /usr/lib/squid/negotiate_wrapper_auth -d \
>
>      --kerberos /usr/lib/squid/negotiate_kerberos_auth -s
> HTTP/prxy1.internal.domain.tld at REALM \
>
>      --ntlm /usr/bin/ntlm_auth --helper-protocol=gss-spnego
> --domain=NTDOMAIN
>
> auth_param negotiate children 50 startup=10 idle=1
>
> auth_param negotiate keep_alive on
>
> auth_param basic program /usr/lib/squid/basic_ldap_auth -R \
>
>      -b "ou=domain,dc=internal,dc=domain,dc=tld" \
>
>      -D changed_to_protect_myself at internal.domain.tld -W
> /etc/squid/private/ldap-bind \
>
>      -f (sAMAccountName=%s) \
>
>      -h dc2.internal.domain.tld \
>
>      -h dc1.internal.domain.tld
>
> auth_param basic children 5 startup=5 idle=1
>
> auth_param basic realm Internet Proxy Autorisation
>
> auth_param basic credentialsttl 2 hours
>
> authenticate_cache_garbage_interval 2 hour
>
> authenticate_ttl 2 hour
>
> authenticate_ip_ttl 2 hour
>
> # ACCESS CONTROLS
>
> #
> -----------------------------------------------------------------------------
>
> acl localnet src fc00::/7       # RFC 4193 local private network range
>
> acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged)
> machines
>
> ## PC Networks
>
> acl localnet src 192.168.XXX.0/24
>
> acl localnet src 10.XXX.0.0/24
>
> acl localnet src 10.XXX.1.0/24
>
> acl localnet src 10.XXX.2.0/24
>
> acl localnet src 10.XXX.3.0/24
>
> acl localnet src 10.XXX.4.0/24
>
> ## Per location/function networks
>
> acl localnet-funct1 src 192.168.XXX.0/24
>
> acl localnet-funct2 src 10.XXX.0.0/24
>
> acl localnet-funct3 src 10.XXX.1.0/24
>
> acl localnet-funct4 src 10.XXX.2.0/24
>
> acl localnet-funct5 src 10.XXX.3.0/24
>
> acl localnet-funct6 src 10.XXX.4.0/24
>
> acl localnet-funct7 src 10.XXX.210.0/24
>
> acl localnet-funct8 src 172.20.XXX.0/24
>
> acl localnet-funct1-server-range src 192.168.XXX.XXX-192.168.XXX.XXX
>
> acl localnet-funct1-mailhopper src 192.168.XXX.XXX
>
> acl localnet-funct1-antivirus src 192.168.XXX.XXX
>
> acl localnet-funct1-xen1 src 192.168.XXX.XXX
>
> acl localnet-funct1-gateway src 192.168.XXX.XXX
>
> acl localnet-funct1-mail1 src 192.168.XXX.XXX
>
> acl localnet-funct1-lin-228 src 192.168.XXX.XXX
>
> acl localnet-funct1-lin-009 src 192.168.XXX.XXX
>
> acl localnet-funct1-monitoring src 192.168.XXX.XXX
>
> acl localnet-funct1-lin-003 src 192.168.XXX.XXX
>
> ## acl time frames.
>
> acl work-ochtend time MTWHF 08:15-11:59
>
> acl work-pauze time MTWHF 12:00-13:30
>
> acl work-middag time MTWHF 13:31-17:00
>
> acl after-work-hours time MTWHF 17:01-23:59
>
> acl before-work-hours time MTWHF 00:00-08:14
>
> ######Block Video Streaming##############
>
> acl media rep_mime_type video/flv video/x-flv
>
> acl media rep_mime_type -i ^video/
>
> acl media rep_mime_type -i ^video\/
>
> acl media rep_mime_type ^application/x-shockwave-flash
>
> acl media rep_mime_type ^application/vnd.ms.wms-hdr.asfv1
>
> acl media rep_mime_type ^application/x-fcs
>
> acl media rep_mime_type ^application/x-mms-framed
>
> acl media rep_mime_type ^video/x-ms-asf
>
> acl media rep_mime_type ^audio/mpeg
>
> acl media rep_mime_type ^audio/x-scpls
>
> acl media rep_mime_type ^video/x-flv
>
> acl media rep_mime_type ^video/mp2t
>
> acl media rep_mime_type ^video/mpeg4
>
> acl media rep_mime_type ms-hdr
>
> acl media rep_mime_type x-fcs
>
> acl mediapr urlpath_regex \.flv(\?.*)?$
>
> acl mediapr urlpath_regex -i \.(avi|mp4|mov|m4v|mkv|flv)(\?.*)?$
>
> acl mediapr urlpath_regex -i
> \.(mpg|mpeg|avi|mov|flv|wmv|mkv|rmvb|ts|)(\?.*)?$
>
> acl whitelistsites url_regex -i "/etc/squid/acl/domain-customer-sites.txt"
>
> acl whitelistsites url_regex -i "/etc/squid/acl/allowed-sites.txt"
>
> acl whitelistdirect url_regex -i "/etc/squid/acl/allowed-direct-sites.txt"
>
> acl ads dstdom_regex "/etc/squid/acl/blocked-ads-company.txt"
>
> acl blockedsites dstdom_regex -i "/etc/squid/acl/blocked-sites.txt"
>
> acl allow_client_mac arp "/etc/squid/acl/allow-arp-client.txt"
>
> acl downloaders rep_mime_type -i ^application/x-nzb$
>
> acl lan-domainname dstdomain .internal.domain.tld
>
> acl lan-domainname dstdomain .internal2.domain.tld
>
> acl lan-domainname dstdomain .internal3.domain.tld
>
> acl lan-domainname dstdomain .internal4.domain.tld
>
> acl lan-domainname dstdomain .internal5.domain.tld
>
> acl lan-domainname dstdomain .internal6.domain.tld
>
> acl wan-domainname dstdomain .domain.tld
>
> acl windowsupdate dstdomain windowsupdate.microsoft.com
>
> acl windowsupdate dstdomain .update.microsoft.com
>
> acl windowsupdate dstdomain download.windowsupdate.com
>
> acl windowsupdate dstdomain redir.metaservices.microsoft.com
>
> acl windowsupdate dstdomain images.metaservices.microsoft.com
>
> acl windowsupdate dstdomain c.microsoft.com
>
> acl windowsupdate dstdomain www.download.windowsupdate.com
>
> acl windowsupdate dstdomain wustat.windows.com
>
> acl windowsupdate dstdomain crl.microsoft.com
>
> acl windowsupdate dstdomain sls.microsoft.com
>
> acl windowsupdate dstdomain productactivation.one.microsoft.com
>
> acl windowsupdate dstdomain ntservicepack.microsoft.com
>
> acl windowsupdate dstdomain au.download.windowsupdate.com
>
> acl windowsupdate dstdomain ds.download.windowsupdate.com
>
> acl windowsupdate dstdomain ctldl.windowsupdate.com
>
> acl windowsupdate dstdomain .data.microsoft.com
>
> acl antivirusupdate dstdomain .trendmicro.com
>
> acl antivirusupdate dstdomain safebrowsing.google.com
>
> acl antivirusupdate dstdomain safebrowsing-cache.google.com
>
> acl wuCONNECT dstdomain www.update.microsoft.com
>
> acl wuCONNECT dstdomain sls.microsoft.com
>
> ## SSL PORTS ( you need to define ssl ports also at Safe_ports )
>
> acl SSL_ports port 443          # https
>
> acl SSL_ports port 631          # cups
>
> acl SSL_ports port 888          # 3dm raid manager
>
> acl SSL_ports port 2812         # Monit
>
> acl SSL_ports port 5225         # HP Toolbox
>
> acl SSL_ports port 8000         # ?
>
> acl SSL_ports port 8080         # ?
>
> acl SSL_ports port 16384-16403  # iChat AV (Audio-RTP, RTCP; Video-RTP,
> RTCP)
>
> acl Safe_ports port 21          # ftp
>
> acl Safe_ports port 80          # http
>
> acl Safe_ports port 70          # gopher
>
> acl Safe_ports port 443         # https
>
> acl Safe_ports port 210         # wais
>
> acl Safe_ports port 280         # http-mgmt
>
> acl Safe_ports port 488         # gss-http
>
> acl Safe_ports port 591         # filemaker
>
> acl Safe_ports port 631         # cups
>
> acl Safe_ports port 667         # darkstat
>
> acl Safe_ports port 777         # multiling http
>
> acl Safe_ports port 888         # 3dm raid manager
>
> acl Safe_ports port 8000        # ?
>
> acl Safe_ports port 8080        # ?
>
> acl Safe_ports port 16384-16403 # iChat AV (Audio-RTP, RTCP; Video-RTP,
> RTCP)
>
> #acl Safe_ports port 1025-65535  # unregistered ports
>
> acl CONNECT method CONNECT
>
> http_access deny !Safe_ports
>
> http_access deny CONNECT !SSL_ports
>
> # Only allow cachemgr access from localhost
>
> http_access allow localhost manager
>
> http_access deny manager
>
> http_access deny to_localhost
>
> #
>
> # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
>
> #
>
> ## BEFORE AUTH : bypass autorisation ( windows updates/antivirus )
>
> http_access allow CONNECT wuCONNECT localnet
>
> http_access allow windowsupdate localnet
>
> http_access allow antivirusupdate localnet
>
> ## Deny blocked sites first.
>
> http_access deny blockedsites
>
> ## Deny Ads servers
>
> http_access deny ads
>
> deny_info TCP_RESET ads
>
> #### Override rules for internal use
>
> http_access allow localnet-funct1-server-range
>
> http_access allow localnet-funct2
>
> http_access allow lan-domainname localnet
>
> http_access allow wan-domainname localnet
>
> http_access allow whitelistdirect localnet
>
> ###############################################################################
>
> ## AUTH HERE
>
> http_access allow authenticated
>
> ###############################################################################
>
> ##########Access Lists VIDEO STREAMS #########
>
> http_access allow mediapr allow_client_mac
>
> http_reply_access allow media allow_client_mac
>
> http_access deny mediapr
>
> http_reply_access deny media
>
> ################################## other rules.
>
> # whitelisted sites
>
> http_access allow whitelistsites
>
> # Example rule allowing access from your local networks.
>
> # Adapt localnet in the ACL section to list your (internal) IP networks
>
> # from where browsing should be allowed
>
> http_access allow localnet
>
> # And finally deny all other access to this proxy
>
> http_access deny all
>
> ## iptables port 80 redirect to 3128
>
> http_port 192.168.XXX.XXX:3128 intercept connection-auth=off
>
> ## company default port set by GPO (must use
> hostname.internal.domain.tld for kerberos auth )
>
> http_port 192.168.XXX.XXX:8080
>
> cache_mem 65536 MB
>
> maximum_object_size_in_memory 5 MB
>
> coredump_dir /var/spool/squid
>
> # disable cache_log
>
> cache_log /dev/null
>
> ## obligated setting for disableing cache_log
>
> logfile_rotate 0
>
> ftp_user anonymousftp at domain.tld
>
> pinger_enable off
>
> # OPTIONS FOR TUNING THE CACHE
>
> #
> -----------------------------------------------------------------------------
>
> #cache deny localnet-funct3
>
> #cache deny localnet-funct2
>
> ## order is important, first one hit is used.
>
> ## windows cache
>
> refresh_pattern -i
> windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320
> 80% 129600 reload-into-ims
>
> refresh_pattern -i
> microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80%
> 129600 reload-into-ims
>
> refresh_pattern -i
> windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80%
> 129600 reload-into-ims
>
> # debian cache
>
> refresh_pattern ^(ht|f)tp://.*debian.*/Packages\.(bz2|gz|diff/Index)$
> 0       0%      0
>
> refresh_pattern ^(ht|f)tp://.*debian.*/Release(\.gpg)?$
> 0       0%      0
>
> refresh_pattern ^(ht|f)tp://.*debian.*/Sources\.(bz2|gz|diff/Index)$
> 0       0%      0
>
> refresh_pattern ^(ht|f)tp://.*debian.*/Translation-en_GB\.bz2)$
> 0       0%      0
>
> # Add any of your own refresh_pattern entries above these.
>
> refresh_pattern ^ftp:           1440    20%     10080
>
> refresh_pattern ^gopher:        1440    0%      1440
>
> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
>
> refresh_pattern .               0       20%     4320
>
> # range-offset
>
> range_offset_limit 800 MB windowsupdate
>
> range_offset_limit 100 MB antivirusupdate
>
> quick_abort_min -1
>
> forward_timeout 1 minutes
>
> connect_timeout 5 seconds
>
> cache_mgr webmaster at domain.tld
>
> mail_from prxy1 at internal.domain.tld
>
> visible_hostname prxy1.internal.domain.tld
>
> hostname_aliases prxy1.internal.domain.tld
>
> httpd_suppress_version_string on
>
> snmp_port 3401
>
> snmp_access allow localnet-funct1-monitoring
>
> snmp_access deny all
>
> snmp_incoming_address 192.168.XXX.XXX
>
> icp_port 3130
>
> htcp_port 4827
>
> udp_incoming_address 192.168.XXX.XXX
>
> error_default_language nl
>
> err_page_stylesheet /etc/squid/errorpage.css
>
> always_direct allow CONNECT
>
> # ICAP OPTIONS
>
> #
> -----------------------------------------------------------------------------
>
> ## Tested with Squid 3.5.10/3.5.12 squidclamav 6.14
>
> icap_enable on
>
> icap_send_client_ip on
>
> icap_send_client_username on
>
> icap_client_username_header X-Authenticated-User
>
> icap_persistent_connections on
>
> icap_preview_enable on
>
> icap_preview_size 1024
>
> icap_service service_req reqmod_precache bypass=1
> icap://127.0.0.1:1344/squidclamav
>
> adaptation_access service_req allow all
>
> icap_service service_resp respmod_precache bypass=1
> icap://127.0.0.1:1344/squidclamav
>
> adaptation_access service_resp allow all
>
> dns_v4_first on
>
> fqdncache_size 2048
>
> memory_pools on
>
> memory_pools_limit 512 MB
>
> forwarded_for on
>
> refresh_all_ims on
>
> reload_into_ims on
>
> workers 8
>
>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>