<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection">
<div dir="auto">Thank you, Alex and Eli,<br />
<br />
Just wanted to update that I was able to solve the problem in case someone runs into it in the future,<br />
based on Eliezers suggestion, I disabled logrotate (although it can be tweaked to work), and just wrote<br />
a custom cron job to run daily and rotate the logs for each squid instance using something like this:<br /></div>
<blockquote style="border: none; margin: 0px 0px 0px 25px; padding: 0px;">
<div style="indent-level: 1">squid -f /etc/squid/squid12.conf -k rotate</div>
<div style="indent-level: 1">..</div>
<div style="indent-level: 1">..</div>
<div style="indent-level: 1">..</div>
<div style="indent-level: 1"><br /></div>
</blockquote>
<div dir="auto">Now everything seems to be working properly, logfile_rotate is set to 1, so the files move over once<br />
and then they get deleted, just as I wanted.</div>
</div>
<div name="messageReplySection">On 7 Sep 2022, 19:27 +0300, NgTech LTD <ngtech1ltd@gmail.com>, wrote:<br />
<blockquote type="cite" style="border-left-color: grey; border-left-width: thin; border-left-style: solid; margin: 5px 5px;padding-left: 10px;">
<div dir="auto">
<div>Good one, Alex.
<div dir="auto"><br /></div>
<div dir="auto">For this specific use case you need a special rotate script which will know the confs file and will loop over them.</div>
<div dir="auto">Later on I will try to see if yave one of these on my servers.</div>
<div dir="auto">Basically you will need an array of config files and loop on them.</div>
<div dir="auto"><br /></div>
<div dir="auto">The pid shouldn't be relevevant for a rotate operation but it depends on the nature of the system.(on a 24/7 system you should know about a service that is down way before the logrotate happpens)</div>
<div dir="auto">If you have a set of config files you can generate a set of postrotate commands compared to a special script.</div>
<div dir="auto"><br /></div>
Let me know if this solution might fit for your use case.</div>
<div dir="auto"><br /></div>
<div dir="auto">Eliezer<br />
<br />
<div class="gmail_quote" dir="auto">
<div dir="ltr" class="gmail_attr">בתאריך יום ד׳, 7 בספט׳ 2022, 3:53, מאת Alex Rousskov <<a href="mailto:rousskov@measurement-factory.com">rousskov@measurement-factory.com</a>>:<br /></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> > pid_filename /var/run/squid2.pid<br />
<br />
> postrotate<br />
> test ! -e /var/run/squid.pid || ... /usr/sbin/squid -k rotate<br />
> endscript<br />
<br />
I spotted one more (potentially critical) problem: Your Squid<br />
configuration sets pid_filename to /var/run/squid2.pid but your<br />
logrotate configuration assumes Squid uses /var/run/squid.pid.<br />
<br />
IMHO, in general, it is best not to guess where Squid has its PID if you<br />
are using "squid -k ...". If you want to test whether Squid is currently<br />
running, try using "squid -k check" instead.<br />
<br />
<br />
HTH,<br />
<br />
Alex.<br />
<br />
<br />
<br />
On 9/6/22 20:45, Alex Rousskov wrote:<br />
> On 9/6/22 18:02, roee klinger wrote:<br />
>> it seems that the logs has filled over 100GB of log data, since I made<br />
>> a configuration mistake (I think?) by setting this:<br />
>><br />
>> logfile_rotate 0<br />
><br />
> This is correct setting when using an external log rotation tool like<br />
> the logrotate daemon. More on that below.<br />
><br />
><br />
>> If I remember and read correctly, this means that the rotation of the<br />
>> files is disabled and they will just keeping increasing<br />
>> in size if left unchecked.<br />
><br />
> To be more precise, this means that you are relying on an external tool<br />
> to rename the log files. With this setting, Squid rotate command closes<br />
> the access log and opens a new one (under the same name). While that<br />
> might sound useless, it is the right (and necessary) thing for Squid to<br />
> do when combined with the correct external log rotation setup.<br />
><br />
><br />
>> I have now gone ahead and changed all the configuration file to this<br />
>> setting:<br />
>><br />
>> logfile_rotate 1<br />
>><br />
>> So now it should rotate once daily, and on the next rotation it should<br />
>> be deleted, and this is all handled by logrotate on Debian-based<br />
>> machines?<br />
><br />
> AFAIK, if you are using an external (to Squid) tool like logrotate, you<br />
> should be setting logfile_rotate to zero.<br />
><br />
><br />
>> This is my / cat /etc/logrotate.d/squid:<br />
>> ➜ / cat /etc/logrotate.d/squid<br />
>> #<br />
>> # Logrotate fragment for squid.<br />
>> #<br />
>> /var/log/squid/*.log {<br />
>> daily<br />
>> compress<br />
>> delaycompress<br />
>> rotate 2<br />
>> missingok<br />
>> nocreate<br />
>> sharedscripts<br />
>> prerotate<br />
>> test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports daily<br />
>> endscript<br />
>> postrotate<br />
>> test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid ||<br />
>> /usr/sbin/squid -k rotate<br />
>> endscript<br />
>> }<br />
><br />
> This is not my area of expertise, but the above configuration does not<br />
> look 100% correct to me: sarg-reports execution failures should have no<br />
> effect on log rotation but does (AFAICT). There may be other problems<br />
> (e.g., I do not know whether your /usr/sbin/squid finds the right Squid<br />
> configuration file). I hope sysadmin experts on this mailing list will<br />
> help you polish this.<br />
><br />
> You should be able to test whether the above is working (e.g., by asking<br />
> logrotate to rotate). Testing is critical even if you do end up getting<br />
> expert log rotation help on this list (this email is not it!).<br />
><br />
><br />
> HTH,<br />
><br />
> Alex.<br />
><br />
><br />
>> Is there a way for me to set it so it just get deleted every 24 or 12<br />
>> hours without the archive first?<br />
>><br />
>> Thanks,<br />
>> Roee<br />
>> On 6 Sep 2022, 16:28 +0300, Alex Rousskov<br />
>> <<a href="mailto:rousskov@measurement-factory.com" target="_blank" rel="noreferrer">rousskov@measurement-factory.com</a>>, wrote:<br />
>>> On 9/6/22 07:41, roee klinger wrote:<br />
>>><br />
>>>> It is also important to know that I am running multiple Squid instances<br />
>>>> on the same machine, they are all getting the error at the same time<br />
>>><br />
>>> What external event(s) happen at that time? Something is probably<br />
>>> sending a signal to the logging daemon process. It would be good to know<br />
>>> what that something (and that signal) is. Your syslog or cache.log might<br />
>>> contain more info. Analyzing the timing/schedule of these problems may<br />
>>> also be helpful in identifying the trigger.<br />
>>><br />
>>><br />
>>>> Is a possible workaround that might be just replacing the line with<br />
>>>> this?<br />
>>><br />
>>>> access_log /var/log/squid/access2.log<br />
>>><br />
>>> As you know, this configuration (in this deprecated spelling or with and<br />
>>> explicit "stdio:" prefix) will result in Squid workers writing to the<br />
>>> log file directly instead of asking the logging daemon. This will,<br />
>>> naturally, get rid of the pipe between workers and their daemons, and<br />
>>> the associated broken pipe error.<br />
>>><br />
>>>> or will this cause a problem?<br />
>>><br />
>>> Impossible to say for sure without knowing whether your workers benefit<br />
>>> from the anticipated performance advantages of avoiding blocking file<br />
>>> I/O _and_ whether those advantages are real (in your environment). Too<br />
>>> many variables and too many unknowns. I would treat this as an important<br />
>>> (and potentially disruptive) configuration change and carefully test the<br />
>>> outcome.<br />
>>><br />
>>><br />
>>> HTH,<br />
>>><br />
>>> Alex.<br />
>>><br />
>>><br />
>>>> INFO -<br />
>>>> Versions:<br />
>>>><br />
>>>> Squid Cache: Version 4.10<br />
>>>> Ubuntu 20.04.4 LTS<br />
>>>><br />
>>>><br />
>>>> Example squid.conf:<br />
>>>><br />
>>>> visible_hostname squid2<br />
>>>><br />
>>>> access_log daemon:/var/log/squid/access2.log squid<br />
>>>><br />
>>>> cache_log /var/log/squid/cache2.log<br />
>>>><br />
>>>> pid_filename /var/run/squid2.pid<br />
>>>><br />
>>>><br />
>>>> acl localnet src 0.0.0.1-0.255.255.255# RFC 1122 "this" network (LAN)<br />
>>>><br />
>>>> acl localnet src <a href="http://10.0.0.0/8" rel="noreferrer noreferrer" target="_blank">10.0.0.0/8</a> <<a href="http://10.0.0.0/8" rel="noreferrer noreferrer" target="_blank">http://10.0.0.0/8</a>> # RFC 1918 local<br />
>>>> private network (LAN)<br />
>>>><br />
>>>> acl localnet src <a href="http://100.64.0.0/10" rel="noreferrer noreferrer" target="_blank">100.64.0.0/10</a> <<a href="http://100.64.0.0/10" rel="noreferrer noreferrer" target="_blank">http://100.64.0.0/10</a>># RFC 6598<br />
>>>> shared address space (CGN)<br />
>>>><br />
>>>> acl localnet src <a href="http://169.254.0.0/16" rel="noreferrer noreferrer" target="_blank">169.254.0.0/16</a> <<a href="http://169.254.0.0/16" rel="noreferrer noreferrer" target="_blank">http://169.254.0.0/16</a>> # RFC 3927<br />
>>>> link-local (directly plugged) machines<br />
>>>><br />
>>>> acl localnet src <a href="http://172.16.0.0/12" rel="noreferrer noreferrer" target="_blank">172.16.0.0/12</a> <<a href="http://172.16.0.0/12" rel="noreferrer noreferrer" target="_blank">http://172.16.0.0/12</a>># RFC 1918<br />
>>>> local private network (LAN)<br />
>>>><br />
>>>> acl localnet src <a href="http://192.168.0.0/16" rel="noreferrer noreferrer" target="_blank">192.168.0.0/16</a> <<a href="http://192.168.0.0/16" rel="noreferrer noreferrer" target="_blank">http://192.168.0.0/16</a>> # RFC 1918<br />
>>>> local private network (LAN)<br />
>>>><br />
>>>> acl localnet src fc00::/7 # RFC 4193 local private network range<br />
>>>><br />
>>>> acl localnet src fe80::/10# RFC 4291 link-local (directly plugged)<br />
>>>> machines<br />
>>>><br />
>>>> acl SSL_ports port 443<br />
>>>><br />
>>>> acl Safe_ports port 80# http<br />
>>>><br />
>>>> acl Safe_ports port 21# ftp<br />
>>>><br />
>>>> acl Safe_ports port 443 # https<br />
>>>><br />
>>>> acl Safe_ports port 70# gopher<br />
>>>><br />
>>>> acl Safe_ports port 210 # wais<br />
>>>><br />
>>>> acl Safe_ports port 1025-65535# unregistered ports<br />
>>>><br />
>>>> acl Safe_ports port 280 # http-mgmt<br />
>>>><br />
>>>> acl Safe_ports port 488 # gss-http<br />
>>>><br />
>>>> acl Safe_ports port 591 # filemaker<br />
>>>><br />
>>>> acl Safe_ports port 777 # multiling http<br />
>>>><br />
>>>> acl CONNECT method CONNECT<br />
>>>><br />
>>>> http_access deny !Safe_ports<br />
>>>><br />
>>>> http_access deny CONNECT !SSL_ports<br />
>>>><br />
>>>> http_access allow localhost manager<br />
>>>><br />
>>>> http_access deny manager<br />
>>>><br />
>>>> # include /etc/squid/conf.d/*<br />
>>>><br />
>>>> http_access allow localhost<br />
>>>><br />
>>>> acl aws src *censored*<br />
>>>><br />
>>>> http_access allow aws<br />
>>>><br />
>>>> # http_access deny all<br />
>>>><br />
>>>> tcp_outgoing_address *censored*<br />
>>>><br />
>>>> http_port 10002<br />
>>>><br />
>>>> coredump_dir /var/spool/squid<br />
>>>><br />
>>>> refresh_pattern ^ftp: 144020% 10080<br />
>>>><br />
>>>> refresh_pattern ^gopher:14400%1440<br />
>>>><br />
>>>> refresh_pattern -i (/cgi-bin/|\?) 0 0%0<br />
>>>><br />
>>>> refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0<br />
>>>> refresh-ims<br />
>>>><br />
>>>> refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims<br />
>>>><br />
>>>> refresh_pattern \/InRelease$ 0 0% 0 refresh-ims<br />
>>>><br />
>>>> refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0<br />
>>>> refresh-ims<br />
>>>><br />
>>>> refresh_pattern . 0 20% 4320<br />
>>>><br />
>>>><br />
>>>> shutdown_lifetime 1 seconds<br />
>>>><br />
>>>> logfile_rotate 0<br />
>>>><br />
>>>> max_filedescriptors 16384<br />
>>>><br />
>>>> dns_nameservers 8.8.8.8 8.8.4.4 1.1.1.1<br />
>>>><br />
>>>> cache deny all<br />
>>>><br />
>>>> cache_dir null /tmp<br />
>>>><br />
>>>> via off<br />
>>>><br />
>>>> forwarded_for off<br />
>>>><br />
>>>> request_header_access From deny all<br />
>>>><br />
>>>> request_header_access Server deny all<br />
>>>><br />
>>>> request_header_access WWW-Authenticate deny all<br />
>>>><br />
>>>> request_header_access Link deny all<br />
>>>><br />
>>>> request_header_access Cache-Control deny all<br />
>>>><br />
>>>> request_header_access Proxy-Connection deny all<br />
>>>><br />
>>>> request_header_access X-Cache deny all<br />
>>>><br />
>>>> request_header_access X-Cache-Lookup deny all<br />
>>>><br />
>>>> request_header_access Via deny all<br />
>>>><br />
>>>> request_header_access X-Forwarded-For deny all<br />
>>>><br />
>>>> request_header_access Pragma deny all<br />
>>>><br />
>>>> request_header_access Keep-Alive deny all<br />
>>>><br />
>>>> dns_v4_first on<br />
>>>><br />
>>>><br />
>>>> Example service file:<br />
>>>><br />
>>>> ## Copyright (C) 1996-2020 The Squid Software Foundation and<br />
>>>> contributors<br />
>>>><br />
>>>> ##<br />
>>>><br />
>>>> ## Squid software is distributed under GPLv2+ license and includes<br />
>>>><br />
>>>> ## contributions from numerous individuals and organizations.<br />
>>>><br />
>>>> ## Please see the COPYING and CONTRIBUTORS files for details.<br />
>>>><br />
>>>> ##<br />
>>>><br />
>>>><br />
>>>> [Unit]<br />
>>>><br />
>>>> Description=Squid Web Proxy Server<br />
>>>><br />
>>>> Documentation=man:squid(8)<br />
>>>><br />
>>>> After=network.target network-online.target nss-lookup.target<br />
>>>><br />
>>>><br />
>>>> [Service]<br />
>>>><br />
>>>> Type=forking<br />
>>>><br />
>>>> PIDFile=/var/run/squid2.pid<br />
>>>><br />
>>>> ExecStartPre=/usr/sbin/squid --foreground -z -f /etc/squid/squid2.conf<br />
>>>><br />
>>>> ExecStart=/usr/sbin/squid -sYC -f /etc/squid/squid2.conf<br />
>>>><br />
>>>> ExecReload=/bin/kill -HUP $MAINPID<br />
>>>><br />
>>>> KillMode=mixed<br />
>>>><br />
>>>><br />
>>>> [Install]<br />
>>>><br />
>>>> WantedBy=multi-user.target<br />
>>>><br />
>>>><br />
>>>><br />
>>>> Permissions:<br />
>>>><br />
>>>> ➜ ls -alt /etc/squid/<br />
>>>> total 128<br />
>>>> drwxr-xr-x 2 root root 4096 Sep 6 11:33 .<br />
>>>> -rw-r--r-- 1 root root 2831 Sep 6 11:33 squid7.conf<br />
>>>> drwxr-xr-x 116 root root 4096 Sep 6 11:33 ..<br />
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:33 squid2.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:33 squid13.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid23.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid19.conf<br />
>>>> -rw-r--r-- 1 root root 2832 Sep 6 11:32 squid1.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid17.conf<br />
>>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid4.conf<br />
>>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid21.conf<br />
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:31 squid25.conf<br />
>>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid12.conf<br />
>>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid3.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:30 squid10.conf<br />
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:30 squid11.conf<br />
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid18.conf<br />
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid8.conf<br />
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid6.conf<br />
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid28.conf<br />
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:25 squid9.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid24.conf<br />
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid22.conf<br />
>>>> -rw-r--r-- 1 root root 2837 Sep 6 11:25 squid20.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid16.conf<br />
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid15.conf<br />
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid14.conf<br />
>>>> -rw-r--r-- 1 root root 2831 Sep 6 11:25 squid5.conf<br />
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:25 squid27.conf<br />
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid26.conf<br />
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid30.conf<br />
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid29.conf<br />
>>>><br />
>>>><br />
>>>> _______________________________________________<br />
>>>> squid-users mailing list<br />
>>>> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br />
>>>> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br />
>>><br />
>>> _______________________________________________<br />
>>> squid-users mailing list<br />
>>> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br />
>>> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br />
><br />
> _______________________________________________<br />
> squid-users mailing list<br />
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br />
> <a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br />
<br />
_______________________________________________<br />
squid-users mailing list<br />
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br />
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br /></blockquote>
</div>
</div>
</div>
_______________________________________________<br />
squid-users mailing list<br />
squid-users@lists.squid-cache.org<br />
http://lists.squid-cache.org/listinfo/squid-users<br /></blockquote>
</div>
</body>
</html>