[squid-users] logfileHandleWrite: daemon:/var/log/squid/access.log: error writing ((32) Broken pipe)
roee klinger
roeeklinger60 at gmail.com
Fri Sep 9 12:40:32 UTC 2022
Thank you, Alex and Eli,
Just wanted to update that I was able to solve the problem in case someone runs into it in the future,
based on Eliezers suggestion, I disabled logrotate (although it can be tweaked to work), and just wrote
a custom cron job to run daily and rotate the logs for each squid instance using something like this:
> quote_type
> squid -f /etc/squid/squid12.conf -k rotate
> ..
> ..
> ..
>
Now everything seems to be working properly, logfile_rotate is set to 1, so the files move over once
and then they get deleted, just as I wanted.
On 7 Sep 2022, 19:27 +0300, NgTech LTD <ngtech1ltd at gmail.com>, wrote:
> Good one, Alex.
>
> For this specific use case you need a special rotate script which will know the confs file and will loop over them.
> Later on I will try to see if yave one of these on my servers.
> Basically you will need an array of config files and loop on them.
>
> The pid shouldn't be relevevant for a rotate operation but it depends on the nature of the system.(on a 24/7 system you should know about a service that is down way before the logrotate happpens)
> If you have a set of config files you can generate a set of postrotate commands compared to a special script.
>
> Let me know if this solution might fit for your use case.
>
> Eliezer
>
> > בתאריך יום ד׳, 7 בספט׳ 2022, 3:53, מאת Alex Rousskov <rousskov at measurement-factory.com>:
> > > > pid_filename /var/run/squid2.pid
> > >
> > > > postrotate
> > > > test ! -e /var/run/squid.pid || ... /usr/sbin/squid -k rotate
> > > > endscript
> > >
> > > I spotted one more (potentially critical) problem: Your Squid
> > > configuration sets pid_filename to /var/run/squid2.pid but your
> > > logrotate configuration assumes Squid uses /var/run/squid.pid.
> > >
> > > IMHO, in general, it is best not to guess where Squid has its PID if you
> > > are using "squid -k ...". If you want to test whether Squid is currently
> > > running, try using "squid -k check" instead.
> > >
> > >
> > > HTH,
> > >
> > > Alex.
> > >
> > >
> > >
> > > On 9/6/22 20:45, Alex Rousskov wrote:
> > > > On 9/6/22 18:02, roee klinger wrote:
> > > >> it seems that the logs has filled over 100GB of log data, since I made
> > > >> a configuration mistake (I think?) by setting this:
> > > >>
> > > >> logfile_rotate 0
> > > >
> > > > This is correct setting when using an external log rotation tool like
> > > > the logrotate daemon. More on that below.
> > > >
> > > >
> > > >> If I remember and read correctly, this means that the rotation of the
> > > >> files is disabled and they will just keeping increasing
> > > >> in size if left unchecked.
> > > >
> > > > To be more precise, this means that you are relying on an external tool
> > > > to rename the log files. With this setting, Squid rotate command closes
> > > > the access log and opens a new one (under the same name). While that
> > > > might sound useless, it is the right (and necessary) thing for Squid to
> > > > do when combined with the correct external log rotation setup.
> > > >
> > > >
> > > >> I have now gone ahead and changed all the configuration file to this
> > > >> setting:
> > > >>
> > > >> logfile_rotate 1
> > > >>
> > > >> So now it should rotate once daily, and on the next rotation it should
> > > >> be deleted, and this is all handled by logrotate on Debian-based
> > > >> machines?
> > > >
> > > > AFAIK, if you are using an external (to Squid) tool like logrotate, you
> > > > should be setting logfile_rotate to zero.
> > > >
> > > >
> > > >> This is my / cat /etc/logrotate.d/squid:
> > > >> ➜ / cat /etc/logrotate.d/squid
> > > >> #
> > > >> # Logrotate fragment for squid.
> > > >> #
> > > >> /var/log/squid/*.log {
> > > >> daily
> > > >> compress
> > > >> delaycompress
> > > >> rotate 2
> > > >> missingok
> > > >> nocreate
> > > >> sharedscripts
> > > >> prerotate
> > > >> test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports daily
> > > >> endscript
> > > >> postrotate
> > > >> test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid ||
> > > >> /usr/sbin/squid -k rotate
> > > >> endscript
> > > >> }
> > > >
> > > > This is not my area of expertise, but the above configuration does not
> > > > look 100% correct to me: sarg-reports execution failures should have no
> > > > effect on log rotation but does (AFAICT). There may be other problems
> > > > (e.g., I do not know whether your /usr/sbin/squid finds the right Squid
> > > > configuration file). I hope sysadmin experts on this mailing list will
> > > > help you polish this.
> > > >
> > > > You should be able to test whether the above is working (e.g., by asking
> > > > logrotate to rotate). Testing is critical even if you do end up getting
> > > > expert log rotation help on this list (this email is not it!).
> > > >
> > > >
> > > > HTH,
> > > >
> > > > Alex.
> > > >
> > > >
> > > >> Is there a way for me to set it so it just get deleted every 24 or 12
> > > >> hours without the archive first?
> > > >>
> > > >> Thanks,
> > > >> Roee
> > > >> On 6 Sep 2022, 16:28 +0300, Alex Rousskov
> > > >> <rousskov at measurement-factory.com>, wrote:
> > > >>> On 9/6/22 07:41, roee klinger wrote:
> > > >>>
> > > >>>> It is also important to know that I am running multiple Squid instances
> > > >>>> on the same machine, they are all getting the error at the same time
> > > >>>
> > > >>> What external event(s) happen at that time? Something is probably
> > > >>> sending a signal to the logging daemon process. It would be good to know
> > > >>> what that something (and that signal) is. Your syslog or cache.log might
> > > >>> contain more info. Analyzing the timing/schedule of these problems may
> > > >>> also be helpful in identifying the trigger.
> > > >>>
> > > >>>
> > > >>>> Is a possible workaround that might be just replacing the line with
> > > >>>> this?
> > > >>>
> > > >>>> access_log /var/log/squid/access2.log
> > > >>>
> > > >>> As you know, this configuration (in this deprecated spelling or with and
> > > >>> explicit "stdio:" prefix) will result in Squid workers writing to the
> > > >>> log file directly instead of asking the logging daemon. This will,
> > > >>> naturally, get rid of the pipe between workers and their daemons, and
> > > >>> the associated broken pipe error.
> > > >>>
> > > >>>> or will this cause a problem?
> > > >>>
> > > >>> Impossible to say for sure without knowing whether your workers benefit
> > > >>> from the anticipated performance advantages of avoiding blocking file
> > > >>> I/O _and_ whether those advantages are real (in your environment). Too
> > > >>> many variables and too many unknowns. I would treat this as an important
> > > >>> (and potentially disruptive) configuration change and carefully test the
> > > >>> outcome.
> > > >>>
> > > >>>
> > > >>> HTH,
> > > >>>
> > > >>> Alex.
> > > >>>
> > > >>>
> > > >>>> INFO -
> > > >>>> Versions:
> > > >>>>
> > > >>>> Squid Cache: Version 4.10
> > > >>>> Ubuntu 20.04.4 LTS
> > > >>>>
> > > >>>>
> > > >>>> Example squid.conf:
> > > >>>>
> > > >>>> visible_hostname squid2
> > > >>>>
> > > >>>> access_log daemon:/var/log/squid/access2.log squid
> > > >>>>
> > > >>>> cache_log /var/log/squid/cache2.log
> > > >>>>
> > > >>>> pid_filename /var/run/squid2.pid
> > > >>>>
> > > >>>>
> > > >>>> acl localnet src 0.0.0.1-0.255.255.255# RFC 1122 "this" network (LAN)
> > > >>>>
> > > >>>> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8> # RFC 1918 local
> > > >>>> private network (LAN)
> > > >>>>
> > > >>>> acl localnet src 100.64.0.0/10 <http://100.64.0.0/10># RFC 6598
> > > >>>> shared address space (CGN)
> > > >>>>
> > > >>>> acl localnet src 169.254.0.0/16 <http://169.254.0.0/16> # RFC 3927
> > > >>>> link-local (directly plugged) machines
> > > >>>>
> > > >>>> acl localnet src 172.16.0.0/12 <http://172.16.0.0/12># RFC 1918
> > > >>>> local private network (LAN)
> > > >>>>
> > > >>>> acl localnet src 192.168.0.0/16 <http://192.168.0.0/16> # RFC 1918
> > > >>>> local private network (LAN)
> > > >>>>
> > > >>>> acl localnet src fc00::/7 # RFC 4193 local private network range
> > > >>>>
> > > >>>> acl localnet src fe80::/10# RFC 4291 link-local (directly plugged)
> > > >>>> machines
> > > >>>>
> > > >>>> acl SSL_ports port 443
> > > >>>>
> > > >>>> acl Safe_ports port 80# http
> > > >>>>
> > > >>>> acl Safe_ports port 21# ftp
> > > >>>>
> > > >>>> acl Safe_ports port 443 # https
> > > >>>>
> > > >>>> acl Safe_ports port 70# gopher
> > > >>>>
> > > >>>> acl Safe_ports port 210 # wais
> > > >>>>
> > > >>>> acl Safe_ports port 1025-65535# unregistered ports
> > > >>>>
> > > >>>> acl Safe_ports port 280 # http-mgmt
> > > >>>>
> > > >>>> acl Safe_ports port 488 # gss-http
> > > >>>>
> > > >>>> acl Safe_ports port 591 # filemaker
> > > >>>>
> > > >>>> acl Safe_ports port 777 # multiling http
> > > >>>>
> > > >>>> acl CONNECT method CONNECT
> > > >>>>
> > > >>>> http_access deny !Safe_ports
> > > >>>>
> > > >>>> http_access deny CONNECT !SSL_ports
> > > >>>>
> > > >>>> http_access allow localhost manager
> > > >>>>
> > > >>>> http_access deny manager
> > > >>>>
> > > >>>> # include /etc/squid/conf.d/*
> > > >>>>
> > > >>>> http_access allow localhost
> > > >>>>
> > > >>>> acl aws src *censored*
> > > >>>>
> > > >>>> http_access allow aws
> > > >>>>
> > > >>>> # http_access deny all
> > > >>>>
> > > >>>> tcp_outgoing_address *censored*
> > > >>>>
> > > >>>> http_port 10002
> > > >>>>
> > > >>>> coredump_dir /var/spool/squid
> > > >>>>
> > > >>>> refresh_pattern ^ftp: 144020% 10080
> > > >>>>
> > > >>>> refresh_pattern ^gopher:14400%1440
> > > >>>>
> > > >>>> refresh_pattern -i (/cgi-bin/|\?) 0 0%0
> > > >>>>
> > > >>>> refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0
> > > >>>> refresh-ims
> > > >>>>
> > > >>>> refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims
> > > >>>>
> > > >>>> refresh_pattern \/InRelease$ 0 0% 0 refresh-ims
> > > >>>>
> > > >>>> refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0
> > > >>>> refresh-ims
> > > >>>>
> > > >>>> refresh_pattern . 0 20% 4320
> > > >>>>
> > > >>>>
> > > >>>> shutdown_lifetime 1 seconds
> > > >>>>
> > > >>>> logfile_rotate 0
> > > >>>>
> > > >>>> max_filedescriptors 16384
> > > >>>>
> > > >>>> dns_nameservers 8.8.8.8 8.8.4.4 1.1.1.1
> > > >>>>
> > > >>>> cache deny all
> > > >>>>
> > > >>>> cache_dir null /tmp
> > > >>>>
> > > >>>> via off
> > > >>>>
> > > >>>> forwarded_for off
> > > >>>>
> > > >>>> request_header_access From deny all
> > > >>>>
> > > >>>> request_header_access Server deny all
> > > >>>>
> > > >>>> request_header_access WWW-Authenticate deny all
> > > >>>>
> > > >>>> request_header_access Link deny all
> > > >>>>
> > > >>>> request_header_access Cache-Control deny all
> > > >>>>
> > > >>>> request_header_access Proxy-Connection deny all
> > > >>>>
> > > >>>> request_header_access X-Cache deny all
> > > >>>>
> > > >>>> request_header_access X-Cache-Lookup deny all
> > > >>>>
> > > >>>> request_header_access Via deny all
> > > >>>>
> > > >>>> request_header_access X-Forwarded-For deny all
> > > >>>>
> > > >>>> request_header_access Pragma deny all
> > > >>>>
> > > >>>> request_header_access Keep-Alive deny all
> > > >>>>
> > > >>>> dns_v4_first on
> > > >>>>
> > > >>>>
> > > >>>> Example service file:
> > > >>>>
> > > >>>> ## Copyright (C) 1996-2020 The Squid Software Foundation and
> > > >>>> contributors
> > > >>>>
> > > >>>> ##
> > > >>>>
> > > >>>> ## Squid software is distributed under GPLv2+ license and includes
> > > >>>>
> > > >>>> ## contributions from numerous individuals and organizations.
> > > >>>>
> > > >>>> ## Please see the COPYING and CONTRIBUTORS files for details.
> > > >>>>
> > > >>>> ##
> > > >>>>
> > > >>>>
> > > >>>> [Unit]
> > > >>>>
> > > >>>> Description=Squid Web Proxy Server
> > > >>>>
> > > >>>> Documentation=man:squid(8)
> > > >>>>
> > > >>>> After=network.target network-online.target nss-lookup.target
> > > >>>>
> > > >>>>
> > > >>>> [Service]
> > > >>>>
> > > >>>> Type=forking
> > > >>>>
> > > >>>> PIDFile=/var/run/squid2.pid
> > > >>>>
> > > >>>> ExecStartPre=/usr/sbin/squid --foreground -z -f /etc/squid/squid2.conf
> > > >>>>
> > > >>>> ExecStart=/usr/sbin/squid -sYC -f /etc/squid/squid2.conf
> > > >>>>
> > > >>>> ExecReload=/bin/kill -HUP $MAINPID
> > > >>>>
> > > >>>> KillMode=mixed
> > > >>>>
> > > >>>>
> > > >>>> [Install]
> > > >>>>
> > > >>>> WantedBy=multi-user.target
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Permissions:
> > > >>>>
> > > >>>> ➜ ls -alt /etc/squid/
> > > >>>> total 128
> > > >>>> drwxr-xr-x 2 root root 4096 Sep 6 11:33 .
> > > >>>> -rw-r--r-- 1 root root 2831 Sep 6 11:33 squid7.conf
> > > >>>> drwxr-xr-x 116 root root 4096 Sep 6 11:33 ..
> > > >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:33 squid2.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:33 squid13.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid23.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid19.conf
> > > >>>> -rw-r--r-- 1 root root 2832 Sep 6 11:32 squid1.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid17.conf
> > > >>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid4.conf
> > > >>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid21.conf
> > > >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:31 squid25.conf
> > > >>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid12.conf
> > > >>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid3.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:30 squid10.conf
> > > >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:30 squid11.conf
> > > >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid18.conf
> > > >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid8.conf
> > > >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid6.conf
> > > >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid28.conf
> > > >>>> -rw-r--r-- 1 root root 2830 Sep 6 11:25 squid9.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid24.conf
> > > >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid22.conf
> > > >>>> -rw-r--r-- 1 root root 2837 Sep 6 11:25 squid20.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid16.conf
> > > >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid15.conf
> > > >>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid14.conf
> > > >>>> -rw-r--r-- 1 root root 2831 Sep 6 11:25 squid5.conf
> > > >>>> -rw-r--r-- 1 root root 2833 Sep 6 11:25 squid27.conf
> > > >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid26.conf
> > > >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid30.conf
> > > >>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid29.conf
> > > >>>>
> > > >>>>
> > > >>>> _______________________________________________
> > > >>>> squid-users mailing list
> > > >>>> squid-users at lists.squid-cache.org
> > > >>>> http://lists.squid-cache.org/listinfo/squid-users
> > > >>>
> > > >>> _______________________________________________
> > > >>> squid-users mailing list
> > > >>> squid-users at lists.squid-cache.org
> > > >>> http://lists.squid-cache.org/listinfo/squid-users
> > > >
> > > > _______________________________________________
> > > > squid-users mailing list
> > > > squid-users at lists.squid-cache.org
> > > > http://lists.squid-cache.org/listinfo/squid-users
> > >
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> > > http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220909/dcdba5fc/attachment-0001.htm>
More information about the squid-users
mailing list