[squid-users] logfileHandleWrite: daemon:/var/log/squid/access.log: error writing ((32) Broken pipe)
NgTech LTD
ngtech1ltd at gmail.com
Wed Sep 7 16:26:40 UTC 2022
Good one, Alex.
For this specific use case you need a special rotate script which will know
the confs file and will loop over them.
Later on I will try to see if yave one of these on my servers.
Basically you will need an array of config files and loop on them.
The pid shouldn't be relevevant for a rotate operation but it depends on
the nature of the system.(on a 24/7 system you should know about a service
that is down way before the logrotate happpens)
If you have a set of config files you can generate a set of postrotate
commands compared to a special script.
Let me know if this solution might fit for your use case.
Eliezer
בתאריך יום ד׳, 7 בספט׳ 2022, 3:53, מאת Alex Rousskov <
rousskov at measurement-factory.com>:
> > pid_filename /var/run/squid2.pid
>
> > postrotate
> > test ! -e /var/run/squid.pid || ... /usr/sbin/squid -k rotate
> > endscript
>
> I spotted one more (potentially critical) problem: Your Squid
> configuration sets pid_filename to /var/run/squid2.pid but your
> logrotate configuration assumes Squid uses /var/run/squid.pid.
>
> IMHO, in general, it is best not to guess where Squid has its PID if you
> are using "squid -k ...". If you want to test whether Squid is currently
> running, try using "squid -k check" instead.
>
>
> HTH,
>
> Alex.
>
>
>
> On 9/6/22 20:45, Alex Rousskov wrote:
> > On 9/6/22 18:02, roee klinger wrote:
> >> it seems that the logs has filled over 100GB of log data, since I made
> >> a configuration mistake (I think?) by setting this:
> >>
> >> logfile_rotate 0
> >
> > This is correct setting when using an external log rotation tool like
> > the logrotate daemon. More on that below.
> >
> >
> >> If I remember and read correctly, this means that the rotation of the
> >> files is disabled and they will just keeping increasing
> >> in size if left unchecked.
> >
> > To be more precise, this means that you are relying on an external tool
> > to rename the log files. With this setting, Squid rotate command closes
> > the access log and opens a new one (under the same name). While that
> > might sound useless, it is the right (and necessary) thing for Squid to
> > do when combined with the correct external log rotation setup.
> >
> >
> >> I have now gone ahead and changed all the configuration file to this
> >> setting:
> >>
> >> logfile_rotate 1
> >>
> >> So now it should rotate once daily, and on the next rotation it should
> >> be deleted, and this is all handled by logrotate on Debian-based
> >> machines?
> >
> > AFAIK, if you are using an external (to Squid) tool like logrotate, you
> > should be setting logfile_rotate to zero.
> >
> >
> >> This is my / cat /etc/logrotate.d/squid:
> >> ➜ / cat /etc/logrotate.d/squid
> >> #
> >> # Logrotate fragment for squid.
> >> #
> >> /var/log/squid/*.log {
> >> daily
> >> compress
> >> delaycompress
> >> rotate 2
> >> missingok
> >> nocreate
> >> sharedscripts
> >> prerotate
> >> test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports daily
> >> endscript
> >> postrotate
> >> test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid ||
> >> /usr/sbin/squid -k rotate
> >> endscript
> >> }
> >
> > This is not my area of expertise, but the above configuration does not
> > look 100% correct to me: sarg-reports execution failures should have no
> > effect on log rotation but does (AFAICT). There may be other problems
> > (e.g., I do not know whether your /usr/sbin/squid finds the right Squid
> > configuration file). I hope sysadmin experts on this mailing list will
> > help you polish this.
> >
> > You should be able to test whether the above is working (e.g., by asking
> > logrotate to rotate). Testing is critical even if you do end up getting
> > expert log rotation help on this list (this email is not it!).
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> >> Is there a way for me to set it so it just get deleted every 24 or 12
> >> hours without the archive first?
> >>
> >> Thanks,
> >> Roee
> >> On 6 Sep 2022, 16:28 +0300, Alex Rousskov
> >> <rousskov at measurement-factory.com>, wrote:
> >>> On 9/6/22 07:41, roee klinger wrote:
> >>>
> >>>> It is also important to know that I am running multiple Squid
> instances
> >>>> on the same machine, they are all getting the error at the same time
> >>>
> >>> What external event(s) happen at that time? Something is probably
> >>> sending a signal to the logging daemon process. It would be good to
> know
> >>> what that something (and that signal) is. Your syslog or cache.log
> might
> >>> contain more info. Analyzing the timing/schedule of these problems may
> >>> also be helpful in identifying the trigger.
> >>>
> >>>
> >>>> Is a possible workaround that might be just replacing the line with
> >>>> this?
> >>>
> >>>> access_log /var/log/squid/access2.log
> >>>
> >>> As you know, this configuration (in this deprecated spelling or with
> and
> >>> explicit "stdio:" prefix) will result in Squid workers writing to the
> >>> log file directly instead of asking the logging daemon. This will,
> >>> naturally, get rid of the pipe between workers and their daemons, and
> >>> the associated broken pipe error.
> >>>
> >>>> or will this cause a problem?
> >>>
> >>> Impossible to say for sure without knowing whether your workers benefit
> >>> from the anticipated performance advantages of avoiding blocking file
> >>> I/O _and_ whether those advantages are real (in your environment). Too
> >>> many variables and too many unknowns. I would treat this as an
> important
> >>> (and potentially disruptive) configuration change and carefully test
> the
> >>> outcome.
> >>>
> >>>
> >>> HTH,
> >>>
> >>> Alex.
> >>>
> >>>
> >>>> INFO -
> >>>> Versions:
> >>>>
> >>>> Squid Cache: Version 4.10
> >>>> Ubuntu 20.04.4 LTS
> >>>>
> >>>>
> >>>> Example squid.conf:
> >>>>
> >>>> visible_hostname squid2
> >>>>
> >>>> access_log daemon:/var/log/squid/access2.log squid
> >>>>
> >>>> cache_log /var/log/squid/cache2.log
> >>>>
> >>>> pid_filename /var/run/squid2.pid
> >>>>
> >>>>
> >>>> acl localnet src 0.0.0.1-0.255.255.255# RFC 1122 "this" network (LAN)
> >>>>
> >>>> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8> # RFC 1918 local
> >>>> private network (LAN)
> >>>>
> >>>> acl localnet src 100.64.0.0/10 <http://100.64.0.0/10># RFC 6598
> >>>> shared address space (CGN)
> >>>>
> >>>> acl localnet src 169.254.0.0/16 <http://169.254.0.0/16> # RFC 3927
> >>>> link-local (directly plugged) machines
> >>>>
> >>>> acl localnet src 172.16.0.0/12 <http://172.16.0.0/12># RFC 1918
> >>>> local private network (LAN)
> >>>>
> >>>> acl localnet src 192.168.0.0/16 <http://192.168.0.0/16> # RFC 1918
> >>>> local private network (LAN)
> >>>>
> >>>> acl localnet src fc00::/7 # RFC 4193 local private network range
> >>>>
> >>>> acl localnet src fe80::/10# RFC 4291 link-local (directly plugged)
> >>>> machines
> >>>>
> >>>> acl SSL_ports port 443
> >>>>
> >>>> acl Safe_ports port 80# http
> >>>>
> >>>> acl Safe_ports port 21# ftp
> >>>>
> >>>> acl Safe_ports port 443 # https
> >>>>
> >>>> acl Safe_ports port 70# gopher
> >>>>
> >>>> acl Safe_ports port 210 # wais
> >>>>
> >>>> acl Safe_ports port 1025-65535# unregistered ports
> >>>>
> >>>> acl Safe_ports port 280 # http-mgmt
> >>>>
> >>>> acl Safe_ports port 488 # gss-http
> >>>>
> >>>> acl Safe_ports port 591 # filemaker
> >>>>
> >>>> acl Safe_ports port 777 # multiling http
> >>>>
> >>>> acl CONNECT method CONNECT
> >>>>
> >>>> http_access deny !Safe_ports
> >>>>
> >>>> http_access deny CONNECT !SSL_ports
> >>>>
> >>>> http_access allow localhost manager
> >>>>
> >>>> http_access deny manager
> >>>>
> >>>> # include /etc/squid/conf.d/*
> >>>>
> >>>> http_access allow localhost
> >>>>
> >>>> acl aws src *censored*
> >>>>
> >>>> http_access allow aws
> >>>>
> >>>> # http_access deny all
> >>>>
> >>>> tcp_outgoing_address *censored*
> >>>>
> >>>> http_port 10002
> >>>>
> >>>> coredump_dir /var/spool/squid
> >>>>
> >>>> refresh_pattern ^ftp: 144020% 10080
> >>>>
> >>>> refresh_pattern ^gopher:14400%1440
> >>>>
> >>>> refresh_pattern -i (/cgi-bin/|\?) 0 0%0
> >>>>
> >>>> refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0
> >>>> refresh-ims
> >>>>
> >>>> refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims
> >>>>
> >>>> refresh_pattern \/InRelease$ 0 0% 0 refresh-ims
> >>>>
> >>>> refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0
> >>>> refresh-ims
> >>>>
> >>>> refresh_pattern . 0 20% 4320
> >>>>
> >>>>
> >>>> shutdown_lifetime 1 seconds
> >>>>
> >>>> logfile_rotate 0
> >>>>
> >>>> max_filedescriptors 16384
> >>>>
> >>>> dns_nameservers 8.8.8.8 8.8.4.4 1.1.1.1
> >>>>
> >>>> cache deny all
> >>>>
> >>>> cache_dir null /tmp
> >>>>
> >>>> via off
> >>>>
> >>>> forwarded_for off
> >>>>
> >>>> request_header_access From deny all
> >>>>
> >>>> request_header_access Server deny all
> >>>>
> >>>> request_header_access WWW-Authenticate deny all
> >>>>
> >>>> request_header_access Link deny all
> >>>>
> >>>> request_header_access Cache-Control deny all
> >>>>
> >>>> request_header_access Proxy-Connection deny all
> >>>>
> >>>> request_header_access X-Cache deny all
> >>>>
> >>>> request_header_access X-Cache-Lookup deny all
> >>>>
> >>>> request_header_access Via deny all
> >>>>
> >>>> request_header_access X-Forwarded-For deny all
> >>>>
> >>>> request_header_access Pragma deny all
> >>>>
> >>>> request_header_access Keep-Alive deny all
> >>>>
> >>>> dns_v4_first on
> >>>>
> >>>>
> >>>> Example service file:
> >>>>
> >>>> ## Copyright (C) 1996-2020 The Squid Software Foundation and
> >>>> contributors
> >>>>
> >>>> ##
> >>>>
> >>>> ## Squid software is distributed under GPLv2+ license and includes
> >>>>
> >>>> ## contributions from numerous individuals and organizations.
> >>>>
> >>>> ## Please see the COPYING and CONTRIBUTORS files for details.
> >>>>
> >>>> ##
> >>>>
> >>>>
> >>>> [Unit]
> >>>>
> >>>> Description=Squid Web Proxy Server
> >>>>
> >>>> Documentation=man:squid(8)
> >>>>
> >>>> After=network.target network-online.target nss-lookup.target
> >>>>
> >>>>
> >>>> [Service]
> >>>>
> >>>> Type=forking
> >>>>
> >>>> PIDFile=/var/run/squid2.pid
> >>>>
> >>>> ExecStartPre=/usr/sbin/squid --foreground -z -f /etc/squid/squid2.conf
> >>>>
> >>>> ExecStart=/usr/sbin/squid -sYC -f /etc/squid/squid2.conf
> >>>>
> >>>> ExecReload=/bin/kill -HUP $MAINPID
> >>>>
> >>>> KillMode=mixed
> >>>>
> >>>>
> >>>> [Install]
> >>>>
> >>>> WantedBy=multi-user.target
> >>>>
> >>>>
> >>>>
> >>>> Permissions:
> >>>>
> >>>> ➜ ls -alt /etc/squid/
> >>>> total 128
> >>>> drwxr-xr-x 2 root root 4096 Sep 6 11:33 .
> >>>> -rw-r--r-- 1 root root 2831 Sep 6 11:33 squid7.conf
> >>>> drwxr-xr-x 116 root root 4096 Sep 6 11:33 ..
> >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:33 squid2.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:33 squid13.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid23.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid19.conf
> >>>> -rw-r--r-- 1 root root 2832 Sep 6 11:32 squid1.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid17.conf
> >>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid4.conf
> >>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid21.conf
> >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:31 squid25.conf
> >>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid12.conf
> >>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid3.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:30 squid10.conf
> >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:30 squid11.conf
> >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid18.conf
> >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid8.conf
> >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid6.conf
> >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid28.conf
> >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:25 squid9.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid24.conf
> >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid22.conf
> >>>> -rw-r--r-- 1 root root 2837 Sep 6 11:25 squid20.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid16.conf
> >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid15.conf
> >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid14.conf
> >>>> -rw-r--r-- 1 root root 2831 Sep 6 11:25 squid5.conf
> >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:25 squid27.conf
> >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid26.conf
> >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid30.conf
> >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid29.conf
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> squid-users mailing list
> >>>> squid-users at lists.squid-cache.org
> >>>> http://lists.squid-cache.org/listinfo/squid-users
> >>>
> >>> _______________________________________________
> >>> squid-users mailing list
> >>> squid-users at lists.squid-cache.org
> >>> http://lists.squid-cache.org/listinfo/squid-users
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220907/bcae7f38/attachment-0001.htm>
More information about the squid-users
mailing list