[squid-users] Squid stopped working after cache.log and access.log rotation
Chee M Gui
cheemeng at function.com
Thu Apr 20 21:54:03 UTC 2017
Some more info:
cache.log is able to rotate.
Only access.log not rotating
root at paproxy:/# ls -alt /var/log/squid
total 15540
drwxrwxr-x 9 root syslog 4096 Apr 20 14:42 ..
drwxr-xr-x 2 proxy proxy 4096 Apr 20 06:25 .
-rw-r----- 1 proxy proxy 63 Apr 20 06:25 cache.log
-rw-r----- 1 proxy proxy 63 Apr 19 06:25 cache.log.1
-rw-r----- 1 proxy proxy 83 Apr 18 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy 63 Mar 19 06:25 cache.log.2
-rw-r----- 1 proxy proxy 15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy 117223 Mar 17 05:52 netdb.state
root at paproxy:/#
We installed squid on 3/15/2017/
root at paproxy:/etc/logrotate.d# date +%s
1492724638
root at paproxy:/etc/logrotate.d# more /var/log/squid/access.log.1
1489614186.653 6 192.168.5.103 TCP_MISS/503 3992 GET
http://ipv6.msftncsi.com/ncsi.txt - HIER_DIRECT/2001:5a8:100::b817:9fae
text/html
1489614186.668 21 192.168.5.103 TCP_MISS/200 280 GET
http://www.msftncsi.com/ncsi.txt - HIER_DIRECT/184.23.159.169 text/plain
1489614186.997 214 192.168.5.103 TCP_MISS/200 617 GET
http://login.live.com/ppcrlcheck.srf - HIER_DIRECT/131.253.61.66 text/htm
l
1489757088.048 10750 192.168.5.103 TCP_TUNNEL/200 5454 CONNECT
p.ebdr2.com:443 - HIER_DIRECT/74.217.250.5 -
1489757099.057 10783 192.168.5.103 TCP_TUNNEL/200 5454 CONNECT
p.ebdr2.com:443 - HIER_DIRECT/74.217.250.5 -
First entry in the access.log.1 file
https://www.epochconverter.com/
GMT: Wed, 15 Mar 2017 21:43:06 GMT
Your time zone: 3/15/2017, 2:43:06 PM GMT-7:00 DST
Last entry in the access.log.1 file
GMT: Fri, 17 Mar 2017 13:24:59 GMT
Your time zone: 3/17/2017, 6:24:59 AM GMT-7:00 DST
Squid was installed on 3/15/2017 on the server.
https://bugs.launchpad.net/ubuntu/+source/squid-deb-proxy/+bug/1544719
https://bugs.launchpad.net/ubuntu/+source/logrotate/+bug/1414754
On Thu, Apr 20, 2017 at 2:34 PM, Chee M Gui <cheemeng at function.com> wrote:
> Hi Eliezer
>
> Thank you for the fast reply.
> Squid is listening on 3128 on the server. See netstat output below.
> We would like to fix the logrotate script (if this is being used?) rather
> that wait for Ubuntu to fix the Squid package (which may take a while).
> There is no cron job under root or proxy or any other users on the server.
> The logrotate file in /etc/cron.daily is also a very old one.
>
> Thank you once again
>
> CMG
>
>
> root at paproxy:/etc/logrotate.d# netstat -an | more
>
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
> tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN
> tcp 13382 0 192.168.5.244:3128 192.168.5.103:49953
> CLOSE_WAIT
> tcp 90 0 198.27.136.41:52652 198.252.206.25:443
> CLOSE_WAIT
> tcp 232 0 192.168.5.244:3128 192.168.5.126:63442
> CLOSE_WAIT
> tcp 440 0 192.168.5.244:3128 192.168.5.126:63423
> CLOSE_WAIT
> tcp 221 0 192.168.5.244:3128 192.168.5.103:54521
> CLOSE_WAIT
>
>
> root at paproxy:/etc/logrotate.d# crontab -l
> no crontab for root
>
>
>
> root at paproxy:/etc/logrotate.d# crontab -u proxy -l
> no crontab for proxy
>
>
> root at paproxy:/etc/logrotate.d# vi squid
> #
> # Logrotate fragment for squid.
> #
> /var/log/squid/*.log {
> daily
> compress
> delaycompress
> rotate 2
> missingok
> nocreate
> sharedscripts
> prerotate
> test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports
> endscript
> postrotate
> test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid
> || /usr/sbin/squid -k rotate
> endscript
> }
>
>
>
>
> root at paproxy:/etc/logrotate.d# dpkg -s squid
> Package: squid
> Status: install ok installed
> Priority: optional
> Section: web
> Installed-Size: 7464
> Maintainer: Ubuntu Developers <ubuntu-devel-discuss at lists.ubuntu.com>
> Architecture: amd64
> Source: squid3
> Version: 3.5.12-1ubuntu7.3
> Replaces: squid3 (<< 3.5.12-1ubuntu1~)
> Depends: libc6 (>= 2.15), libcap2 (>= 1:2.10), libcomerr2 (>= 1.01),
> libdb5.3, libecap3 (>= 1.0.1), libexpat1 (>= 2.0.1), libgcc1 (>= 1:3.0),
> libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.10+dfsg~), libldap-2.4-2
> (>= 2.4.7), libltdl7 (>= 2.4.6), libnetfilter-conntrack3, libnettle6,
> libpam0g (>= 0.99.7.1), libsasl2-2, libstdc++6 (>= 5.2), libxml2 (>=
> 2.7.4), netbase, logrotate (>= 3.5.4-1), squid-common (=
> 3.5.12-1ubuntu7.3), lsb-base, ssl-cert, init-system-helpers (>> 1.22ubuntu5)
> Pre-Depends: adduser
> Suggests: squidclient, squid-cgi, squid-purge, resolvconf (>= 0.40),
> smbclient, ufw, winbindd, apparmor
> Breaks: squid3 (<< 3.5.12-1ubuntu1~), ufw (<< 0.35-0ubuntu2~)
> Conffiles:
> /etc/apparmor.d/usr.sbin.squid 08e05266f0ef7a9a4ac2c62be29a3ef2
> /etc/init.d/squid f67c63ce21e0ac57a4d16e90909b3e34
> /etc/logrotate.d/squid 2be386088ead3641de5401a9c73a7a57
> /etc/resolvconf/update-libc.d/squid 9968dc6f2fcde9f38a6faea7dfe95dd1
> /etc/squid/errorpage.css 7f1cc06116c222d49d641f0e830ff615
> /etc/squid/squid.conf e73b82ed9d76b47c8b5963175f0ada1e
> /etc/ufw/applications.d/squid 710e7b8ded49bbcd41eb072a0fe1691f
> Description: Full featured Web Proxy cache (HTTP proxy)
> Squid is a high-performance proxy caching server for web clients,
> supporting
> FTP, gopher, ICY and HTTP data objects.
> .
> Squid version 3 is a major rewrite of Squid in C++ and introduces a
> number of
> new features including ICAP and ESI support.
> Homepage: http://www.squid-cache.org
> Original-Maintainer: Luigi Gangitano <luigi at debian.org>
> root at paproxy:/etc/logrotate.d#
>
>
>
>
> root at paproxy:/etc/logrotate.d# ls -al /etc/cron.daily
> total 56
> drwxr-xr-x 2 root root 4096 Mar 13 16:30 .
> drwxr-xr-x 91 root root 4096 Apr 3 13:43 ..
> -rwxr-xr-x 1 root root 376 Mar 31 2016 apport
> -rwxr-xr-x 1 root root 1474 Oct 31 07:31 apt-compat
> -rwxr-xr-x 1 root root 355 May 22 2012 bsdmainutils
> -rwxr-xr-x 1 root root 1597 Nov 26 2015 dpkg
> -rwxr-xr-x 1 root root 372 May 5 2015 logrotate
> -rwxr-xr-x 1 root root 1293 Nov 6 2015 man-db
> -rwxr-xr-x 1 root root 539 Jul 16 2014 mdadm
> -rwxr-xr-x 1 root root 435 Nov 17 2014 mlocate
> -rwxr-xr-x 1 root root 249 Nov 12 2015 passwd
> -rw-r--r-- 1 root root 102 Apr 5 2016 .placeholder
> -rwxr-xr-x 1 root root 3449 Feb 26 2016 popularity-contest
> -rwxr-xr-x 1 root root 214 May 24 2016 update-notifier-common
>
>
>
>
>
>
>
>
> On Thu, Apr 20, 2017 at 12:24 PM, Eliezer Croitoru <eliezer at ngtech.co.il>
> wrote:
>
>> Hey CM,
>>
>> From the output I understand that there wasn't a change in squid
>> packaging for 16.04 and they still use rc\init.d startup scripts.
>> Also they probably use the same logrotate scripts from very long ago.
>> We first must understand if squid is running and it seems that systemd
>> sees it as running.
>> I do not see in lsof output any port listening mentioned so I assume this
>> is the reason for the issue in hands.
>> The first thing I would do is run a crontab that will check if squid is
>> alive using a cache manager info page fetch and check if it's listening
>> using netstat or ss.
>> (did you tried to see if squid is listening using netstat or ss??)
>> The next step would be to check your squid roatate script and to verify
>> it's doing what it suppose to do.
>> After all this I would recommend changing from the rc\init.d startup
>> script to a real system based one and abandon the old rotation scripts of
>> Ubuntu or fix them.
>>
>> If you are looking for a fix it's one path and if you are looking to get
>> the work done properly by Ubuntu it's a whole new wagon.
>> I have been working on squid packages for Ubuntu and Debian that uses
>> system scripts but the packages are not perfected yet.
>>
>> Let me know the path you want to choose and also your approach to things
>> and I will try to help you with which of the options you will choose to
>> resolve the issues.
>>
>> Eliezer
>>
>> ----
>> http://ngtech.co.il/lmgtfy/
>> Linux System Administrator
>> Mobile: +972-5-28704261
>> Email: eliezer at ngtech.co.il
>>
>>
>> From: Chee M Gui [mailto:cheemeng at function.com]
>> Sent: Thursday, April 20, 2017 7:17 PM
>> To: Eliezer Croitoru <eliezer at ngtech.co.il>
>> Cc: squid-users at lists.squid-cache.org
>> Subject: Re: [squid-users] Squid stopped working after cache.log and
>> access.log rotation
>>
>> Hi Eliezer
>>
>> Thank you for the response, and sorry for the late reply.
>>
>> As requested, here are the output of the commands you suggested:
>> root at paproxy:/# systemctl status squid
>> ● squid.service - LSB: Squid HTTP Proxy version 3.x
>> Loaded: loaded (/etc/init.d/squid; bad; vendor preset: enabled)
>> Active: active (running) since Wed 2017-03-15 14:47:53 PDT; 1 months 5
>> days ago
>> Docs: man:systemd-sysv-generator(8)
>> Tasks: 4
>> Memory: 54.4M
>> CPU: 2min 38.957s
>> CGroup: /system.slice/squid.service
>> ├─25482 /usr/sbin/squid -YC -f /etc/squid/squid.conf
>> ├─25484 (squid-1) -YC -f /etc/squid/squid.conf
>> ├─25485 (logfile-daemon) /var/log/squid/access.log
>> └─25486 (unlinkd)
>>
>> Warning: Journal has been rotated since unit was started. Log output is
>> incomplete or unavailable.
>> root at paproxy:/#
>>
>>
>>
>> top - 09:11:45 up 54 days, 23:51, 2 users, load average: 0.00, 0.00,
>> 0.00
>> Tasks: 141 total, 1 running, 140 sleeping, 0 stopped, 0 zombie
>> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si,
>> 0.0 st
>> KiB Mem : 6043140 total, 4844728 free, 135292 used, 1063120
>> buff/cache
>> KiB Swap: 6222844 total, 6222844 free, 0 used. 5556300 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 23355 root 20 0 41668 3776 3236 R 6.7 0.1 0:00.01 top
>> 1 root 20 0 37884 5968 4020 S 0.0 0.1 0:27.20
>> systemd
>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.22
>> kthreadd
>> 3 root 20 0 0 0 0 S 0.0 0.0 0:01.01
>> ksoftirqd/0
>> 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00
>> kworker/0:0H
>> 7 root 20 0 0 0 0 S 0.0 0.0 1:28.65
>> rcu_sched
>> 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
>> 9 root rt 0 0 0 0 S 0.0 0.0 0:00.14
>> migration/0
>> 10 root rt 0 0 0 0 S 0.0 0.0 0:05.53
>> watchdog/0
>> 11 root rt 0 0 0 0 S 0.0 0.0 0:05.75
>> watchdog/1
>> 12 root rt 0 0 0 0 S 0.0 0.0 0:00.14
>> migration/1
>> 13 root 20 0 0 0 0 S 0.0 0.0 0:02.48
>> ksoftirqd/1
>> 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00
>> kworker/1:0H
>> 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00
>> kdevtmpfs
>> 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
>> 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf
>> 19 root 20 0 0 0 0 S 0.0 0.0 0:01.26
>> khungtaskd
>> 20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00
>> writeback
>>
>>
>> root at paproxy:/# ps -aux | grep squid
>> root 23358 0.0 0.0 14224 976 pts/0 S+ 09:12 0:00 grep
>> --color=auto squid
>> root 25482 0.0 0.1 109272 6416 ? Ss Mar15 0:00
>> /usr/sbin/squid -YC -f /etc/squid/squid.conf
>> proxy 25484 0.0 0.7 166684 45184 ? S Mar15 2:27
>> (squid-1) -YC -f /etc/squid/squid.conf
>> proxy 25485 0.0 0.0 13280 1648 ? S Mar15 0:03
>> (logfile-daemon) /var/log/squid/access.log
>> root at paproxy:/#
>>
>>
>> root at paproxy:/# df -h
>> Filesystem Size Used Avail Use% Mounted on
>> udev 2.9G 0 2.9G 0% /dev
>> tmpfs 591M 60M 531M 11% /run
>> /dev/sda1 911G 1.9G 863G 1% /
>> tmpfs 2.9G 12K 2.9G 1% /dev/shm
>> tmpfs 5.0M 0 5.0M 0% /run/lock
>> tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
>> tmpfs 591M 0 591M 0% /run/user/1000
>> tmpfs 591M 0 591M 0% /run/user/0
>> root at paproxy:/#
>>
>>
>>
>> root at paproxy:/# df -h
>> Filesystem Size Used Avail Use% Mounted on
>> udev 2.9G 0 2.9G 0% /dev
>> tmpfs 591M 60M 531M 11% /run
>> /dev/sda1 911G 1.9G 863G 1% /
>> tmpfs 2.9G 12K 2.9G 1% /dev/shm
>> tmpfs 5.0M 0 5.0M 0% /run/lock
>> tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
>> tmpfs 591M 0 591M 0% /run/user/1000
>> tmpfs 591M 0 591M 0% /run/user/0
>> root at paproxy:/#
>>
>>
>> root at paproxy:/# lsof -n|egrep "proxy|squid" | more
>> squid 25482 root cwd DIR 8,1
>> 4096 2 /
>> squid 25482 root rtd DIR 8,1
>> 4096 2 /
>> squid 25482 root txt REG 8,1
>> 6430816 41946763 /usr/sbin/squid
>> squid 25482 root mem REG 8,1
>> 47648 10093387 /lib/x86_64-linux-gnu/libnss_nis-2.23.
>> so
>> squid 25482 root mem REG 8,1
>> 93128 10093399 /lib/x86_64-linux-gnu/http://libnsl-2.23.so
>> squid 25482 root mem REG 8,1
>> 35688 10093400 /lib/x86_64-linux-gnu/libnss_compat-2.
>> 23.so
>> squid 25482 root mem REG 8,1
>> 47600 10093403 /lib/x86_64-linux-gnu/libnss_files-2.2
>> 3.so
>> squid 25482 root mem REG 8,1
>> 25913104 41944015 /usr/lib/x86_64-linux-gnu/libicudata.s
>> o.55.1
>> squid 25482 root mem REG 8,1
>> 22520 10093098 /lib/x86_64-linux-gnu/libmnl.so.0.1.0
>> squid 25482 root mem REG 8,1
>> 26248 41948589 /usr/lib/x86_64-linux-gnu/libnfnetlink
>> .so.0.2.0
>>
>>
>> Thank you once again
>>
>> CM
>>
>>
>>
>> On Thu, Mar 23, 2017 at 1:59 PM, Eliezer Croitoru <mailto:
>> eliezer at ngtech.co.il> wrote:
>> There is another option!
>> The log rotate script is doing something nasty or the systemd service
>> file start up squid in a weird way.
>> The output of:
>> $ systemctl status squid
>> $ top -n1 -b
>> $ ps aux
>> $ df -h
>> $ netstat -ntulp
>> $ lsof -n|egrep "proxy|squid"
>>
>> How many clients this system has?
>> Is the system facing the Internet directly or behind some nat(aws or
>> another provider)?
>>
>> The above are the basic required data to understand the situation.
>>
>> All The Bests,
>> Eliezer
>>
>> ----
>> http://ngtech.co.il/lmgtfy/
>> Linux System Administrator
>> Mobile: tel:%2B972-5-28704261
>> Email: mailto:eliezer at ngtech.co.il
>>
>>
>> From: squid-users [mailto:mailto:squid-users-bou
>> nces at lists.squid-cache.org] On Behalf Of Chee M Gui
>> Sent: Wednesday, March 22, 2017 5:18 PM
>> To: mailto:squid-users at lists.squid-cache.org
>> Subject: [squid-users] Squid stopped working after cache.log and
>> access.log rotation
>>
>>
>> Hi All
>>
>> We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS. It
>> ran fine at first but stopped working after a while. telnet server 3128
>> still works, i.e., opens a blank window, but Squid is just not accepting
>> requests. Then we realized that there is no new access.log file. The
>> access.log file stopped rotated at 6:24AM on 3/17/2017. It looks like
>> Squid wasn't able to create a new access.log? We could not find any error
>> message in syslog or the cache.log. We haven't rebooted the server
>> because we want to know what went wrong. It isn't the firewall blocking
>> Squid because Squid was working fine all the while until recently. Also
>> after it stopped working, we disabled the firewall to see if it would work
>> but it still didn't work.
>>
>> root at paproxy:/var/log/squid# ls -alt
>> total 15536
>> drwxr-xr-x 2 proxy proxy 4096 Mar 21 06:25 .
>> -rw-r----- 1 proxy proxy 63 Mar 21 06:25 cache.log
>> drwxrwxr-x 9 root syslog 4096 Mar 21 06:25 ..
>> -rw-r----- 1 proxy proxy 63 Mar 20 06:25 cache.log.1
>> -rw-r----- 1 proxy proxy 83 Mar 19 06:25 cache.log.2.gz
>> -rw-r----- 1 proxy proxy 15759111 Mar 17 06:24 access.log.1
>> -rw-r----- 1 proxy proxy 117223 Mar 17 05:52 netdb.state
>>
>> Any ideas what went wrong?
>>
>> Thank you very much in anticipation.
>>
>> Gui
>> _______________________________________________
>> squid-users mailing list
>> mailto:squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
>>
>> --
>> Chee Meng Gui
>> Function Engineering
>> 650-833-0660
>>
>>
>
>
> --
> Chee Meng Gui
> Function Engineering
> 650-833-0660 <(650)%20833-0660>
>
--
Chee Meng Gui
Function Engineering
650-833-0660
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170420/fabb995b/attachment-0001.html>
More information about the squid-users
mailing list