<div dir="ltr">Hi Eliezer<div><br></div><div>Thank you for the fast reply. </div><div>Squid is listening on 3128 on the server. See netstat output below.</div><div>We would like to fix the logrotate script (if this is being used?) rather that wait for Ubuntu to fix the Squid package (which may take a while).</div><div>There is no cron job under root or proxy or any other users on the server.</div><div>The logrotate file in /etc/cron.daily is also a very old one.</div><div><br></div><div>Thank you once again</div><div><br></div><div>CMG</div><div><br></div><div><div><br></div><div>root@paproxy:/etc/logrotate.d# netstat -an | more</div><div><br></div><div>Active Internet connections (servers and established)</div><div>Proto Recv-Q Send-Q Local Address Foreign Address State</div><div>tcp 0 0 <a href="http://0.0.0.0:22">0.0.0.0:22</a> 0.0.0.0:* LISTEN</div><div>tcp 0 0 <a href="http://0.0.0.0:3128">0.0.0.0:3128</a> 0.0.0.0:* LISTEN</div><div>tcp 13382 0 <a href="http://192.168.5.244:3128">192.168.5.244:3128</a> <a href="http://192.168.5.103:49953">192.168.5.103:49953</a> CLOSE_WAIT</div><div>tcp 90 0 <a href="http://198.27.136.41:52652">198.27.136.41:52652</a> <a href="http://198.252.206.25:443">198.252.206.25:443</a> CLOSE_WAIT</div><div>tcp 232 0 <a href="http://192.168.5.244:3128">192.168.5.244:3128</a> <a href="http://192.168.5.126:63442">192.168.5.126:63442</a> CLOSE_WAIT</div><div>tcp 440 0 <a href="http://192.168.5.244:3128">192.168.5.244:3128</a> <a href="http://192.168.5.126:63423">192.168.5.126:63423</a> CLOSE_WAIT</div><div>tcp 221 0 <a href="http://192.168.5.244:3128">192.168.5.244:3128</a> <a href="http://192.168.5.103:54521">192.168.5.103:54521</a> CLOSE_WAIT</div><div><br></div><div><br></div><div>root@paproxy:/etc/logrotate.d# crontab -l</div><div>no crontab for root</div><div><br></div><div><br></div><div><br></div><div>root@paproxy:/etc/logrotate.d# crontab -u proxy -l</div><div>no crontab for proxy</div><div><br></div><div><br></div><div>root@paproxy:/etc/logrotate.d# vi squid</div><div>#</div><div># Logrotate fragment for squid.</div><div>#</div><div>/var/log/squid/*.log {</div><div> daily</div><div> compress</div><div> delaycompress</div><div> rotate 2</div><div> missingok</div><div> nocreate</div><div> sharedscripts</div><div> prerotate</div><div> test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports</div><div> endscript</div><div> postrotate</div><div> test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid || /usr/sbin/squid -k rotate</div><div> endscript</div><div>}</div><div><br></div><div><br></div><div><br></div><div><br></div><div>root@paproxy:/etc/logrotate.d# dpkg -s squid</div><div>Package: squid</div><div>Status: install ok installed</div><div>Priority: optional</div><div>Section: web</div><div>Installed-Size: 7464</div><div>Maintainer: Ubuntu Developers <<a href="mailto:ubuntu-devel-discuss@lists.ubuntu.com">ubuntu-devel-discuss@lists.ubuntu.com</a>></div><div>Architecture: amd64</div><div>Source: squid3</div><div>Version: 3.5.12-1ubuntu7.3</div><div>Replaces: squid3 (<< 3.5.12-1ubuntu1~)</div><div>Depends: libc6 (>= 2.15), libcap2 (>= 1:2.10), libcomerr2 (>= 1.01), libdb5.3, libecap3 (>= 1.0.1), libexpat1 (>= 2.0.1), libgcc1 (>= 1:3.0), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.10+dfsg~), libldap-2.4-2 (>= 2.4.7), libltdl7 (>= 2.4.6), libnetfilter-conntrack3, libnettle6, libpam0g (>= 0.99.7.1), libsasl2-2, libstdc++6 (>= 5.2), libxml2 (>= 2.7.4), netbase, logrotate (>= 3.5.4-1), squid-common (= 3.5.12-1ubuntu7.3), lsb-base, ssl-cert, init-system-helpers (>> 1.22ubuntu5)</div><div>Pre-Depends: adduser</div><div>Suggests: squidclient, squid-cgi, squid-purge, resolvconf (>= 0.40), smbclient, ufw, winbindd, apparmor</div><div>Breaks: squid3 (<< 3.5.12-1ubuntu1~), ufw (<< 0.35-0ubuntu2~)</div><div>Conffiles:</div><div> /etc/apparmor.d/usr.sbin.squid 08e05266f0ef7a9a4ac2c62be29a3ef2</div><div> /etc/init.d/squid f67c63ce21e0ac57a4d16e90909b3e34</div><div> /etc/logrotate.d/squid 2be386088ead3641de5401a9c73a7a57</div><div> /etc/resolvconf/update-libc.d/squid 9968dc6f2fcde9f38a6faea7dfe95dd1</div><div> /etc/squid/errorpage.css 7f1cc06116c222d49d641f0e830ff615</div><div> /etc/squid/squid.conf e73b82ed9d76b47c8b5963175f0ada1e</div><div> /etc/ufw/applications.d/squid 710e7b8ded49bbcd41eb072a0fe1691f</div><div>Description: Full featured Web Proxy cache (HTTP proxy)</div><div> Squid is a high-performance proxy caching server for web clients, supporting</div><div> FTP, gopher, ICY and HTTP data objects.</div><div> .</div><div> Squid version 3 is a major rewrite of Squid in C++ and introduces a number of</div><div> new features including ICAP and ESI support.</div><div>Homepage: <a href="http://www.squid-cache.org">http://www.squid-cache.org</a></div><div>Original-Maintainer: Luigi Gangitano <<a href="mailto:luigi@debian.org">luigi@debian.org</a>></div><div>root@paproxy:/etc/logrotate.d#</div><div><br></div><div><br></div><div><br></div><div><br></div><div>root@paproxy:/etc/logrotate.d# ls -al /etc/cron.daily</div><div>total 56</div><div>drwxr-xr-x 2 root root 4096 Mar 13 16:30 .</div><div>drwxr-xr-x 91 root root 4096 Apr 3 13:43 ..</div><div>-rwxr-xr-x 1 root root 376 Mar 31 2016 apport</div><div>-rwxr-xr-x 1 root root 1474 Oct 31 07:31 apt-compat</div><div>-rwxr-xr-x 1 root root 355 May 22 2012 bsdmainutils</div><div>-rwxr-xr-x 1 root root 1597 Nov 26 2015 dpkg</div><div>-rwxr-xr-x 1 root root 372 May 5 2015 logrotate</div><div>-rwxr-xr-x 1 root root 1293 Nov 6 2015 man-db</div><div>-rwxr-xr-x 1 root root 539 Jul 16 2014 mdadm</div><div>-rwxr-xr-x 1 root root 435 Nov 17 2014 mlocate</div><div>-rwxr-xr-x 1 root root 249 Nov 12 2015 passwd</div><div>-rw-r--r-- 1 root root 102 Apr 5 2016 .placeholder</div><div>-rwxr-xr-x 1 root root 3449 Feb 26 2016 popularity-contest</div><div>-rwxr-xr-x 1 root root 214 May 24 2016 update-notifier-common</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 20, 2017 at 12:24 PM, Eliezer Croitoru <span dir="ltr"><<a href="mailto:eliezer@ngtech.co.il" target="_blank">eliezer@ngtech.co.il</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey CM,<br>
<br>
>From the output I understand that there wasn't a change in squid packaging for 16.04 and they still use rc\init.d startup scripts.<br>
Also they probably use the same logrotate scripts from very long ago.<br>
We first must understand if squid is running and it seems that systemd sees it as running.<br>
I do not see in lsof output any port listening mentioned so I assume this is the reason for the issue in hands.<br>
The first thing I would do is run a crontab that will check if squid is alive using a cache manager info page fetch and check if it's listening using netstat or ss.<br>
(did you tried to see if squid is listening using netstat or ss??)<br>
The next step would be to check your squid roatate script and to verify it's doing what it suppose to do.<br>
After all this I would recommend changing from the rc\init.d startup script to a real system based one and abandon the old rotation scripts of Ubuntu or fix them.<br>
<br>
If you are looking for a fix it's one path and if you are looking to get the work done properly by Ubuntu it's a whole new wagon.<br>
I have been working on squid packages for Ubuntu and Debian that uses system scripts but the packages are not perfected yet.<br>
<br>
Let me know the path you want to choose and also your approach to things and I will try to help you with which of the options you will choose to resolve the issues.<br>
<span class=""><br>
Eliezer<br>
<br>
----<br>
<a href="http://ngtech.co.il/lmgtfy/" rel="noreferrer" target="_blank">http://ngtech.co.il/lmgtfy/</a><br>
Linux System Administrator<br>
Mobile: <a href="tel:%2B972-5-28704261" value="+972528704261">+972-5-28704261</a><br>
Email: <a href="mailto:eliezer@ngtech.co.il">eliezer@ngtech.co.il</a><br>
<br>
<br>
</span>From: Chee M Gui [mailto:<a href="mailto:cheemeng@function.com">cheemeng@function.com</a>]<br>
Sent: Thursday, April 20, 2017 7:17 PM<br>
To: Eliezer Croitoru <<a href="mailto:eliezer@ngtech.co.il">eliezer@ngtech.co.il</a>><br>
Cc: <a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.<wbr>org</a><br>
Subject: Re: [squid-users] Squid stopped working after cache.log and access.log rotation<br>
<div><div class="h5"><br>
Hi Eliezer<br>
<br>
Thank you for the response, and sorry for the late reply.<br>
<br>
As requested, here are the output of the commands you suggested:<br>
root@paproxy:/# systemctl status squid<br>
● squid.service - LSB: Squid HTTP Proxy version 3.x<br>
Loaded: loaded (/etc/init.d/squid; bad; vendor preset: enabled)<br>
Active: active (running) since Wed 2017-03-15 14:47:53 PDT; 1 months 5 days ago<br>
Docs: man:systemd-sysv-generator(8)<br>
Tasks: 4<br>
Memory: 54.4M<br>
CPU: 2min 38.957s<br>
CGroup: /system.slice/squid.service<br>
├─25482 /usr/sbin/squid -YC -f /etc/squid/squid.conf<br>
├─25484 (squid-1) -YC -f /etc/squid/squid.conf<br>
├─25485 (logfile-daemon) /var/log/squid/access.log<br>
└─25486 (unlinkd)<br>
<br>
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.<br>
root@paproxy:/#<br>
<br>
<br>
<br>
top - 09:11:45 up 54 days, 23:51, 2 users, load average: 0.00, 0.00, 0.00<br>
Tasks: 141 total, 1 running, 140 sleeping, 0 stopped, 0 zombie<br>
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
KiB Mem : 6043140 total, 4844728 free, 135292 used, 1063120 buff/cache<br>
KiB Swap: 6222844 total, 6222844 free, 0 used. 5556300 avail Mem<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
23355 root 20 0 41668 3776 3236 R 6.7 0.1 0:00.01 top<br>
1 root 20 0 37884 5968 4020 S 0.0 0.1 0:27.20 systemd<br>
2 root 20 0 0 0 0 S 0.0 0.0 0:00.22 kthreadd<br>
3 root 20 0 0 0 0 S 0.0 0.0 0:01.01 ksoftirqd/0<br>
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H<br>
7 root 20 0 0 0 0 S 0.0 0.0 1:28.65 rcu_sched<br>
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh<br>
9 root rt 0 0 0 0 S 0.0 0.0 0:00.14 migration/0<br>
10 root rt 0 0 0 0 S 0.0 0.0 0:05.53 watchdog/0<br>
11 root rt 0 0 0 0 S 0.0 0.0 0:05.75 watchdog/1<br>
12 root rt 0 0 0 0 S 0.0 0.0 0:00.14 migration/1<br>
13 root 20 0 0 0 0 S 0.0 0.0 0:02.48 ksoftirqd/1<br>
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H<br>
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs<br>
17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns<br>
18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf<br>
19 root 20 0 0 0 0 S 0.0 0.0 0:01.26 khungtaskd<br>
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback<br>
<br>
<br>
root@paproxy:/# ps -aux | grep squid<br>
root 23358 0.0 0.0 14224 976 pts/0 S+ 09:12 0:00 grep --color=auto squid<br>
root 25482 0.0 0.1 109272 6416 ? Ss Mar15 0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf<br>
proxy 25484 0.0 0.7 166684 45184 ? S Mar15 2:27 (squid-1) -YC -f /etc/squid/squid.conf<br>
proxy 25485 0.0 0.0 13280 1648 ? S Mar15 0:03 (logfile-daemon) /var/log/squid/access.log<br>
root@paproxy:/#<br>
<br>
<br>
root@paproxy:/# df -h<br>
Filesystem Size Used Avail Use% Mounted on<br>
udev 2.9G 0 2.9G 0% /dev<br>
tmpfs 591M 60M 531M 11% /run<br>
/dev/sda1 911G 1.9G 863G 1% /<br>
tmpfs 2.9G 12K 2.9G 1% /dev/shm<br>
tmpfs 5.0M 0 5.0M 0% /run/lock<br>
tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup<br>
tmpfs 591M 0 591M 0% /run/user/1000<br>
tmpfs 591M 0 591M 0% /run/user/0<br>
root@paproxy:/#<br>
<br>
<br>
<br>
root@paproxy:/# df -h<br>
Filesystem Size Used Avail Use% Mounted on<br>
udev 2.9G 0 2.9G 0% /dev<br>
tmpfs 591M 60M 531M 11% /run<br>
/dev/sda1 911G 1.9G 863G 1% /<br>
tmpfs 2.9G 12K 2.9G 1% /dev/shm<br>
tmpfs 5.0M 0 5.0M 0% /run/lock<br>
tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup<br>
tmpfs 591M 0 591M 0% /run/user/1000<br>
tmpfs 591M 0 591M 0% /run/user/0<br>
root@paproxy:/#<br>
<br>
<br>
root@paproxy:/# lsof -n|egrep "proxy|squid" | more<br>
squid 25482 root cwd DIR 8,1 4096 2 /<br>
squid 25482 root rtd DIR 8,1 4096 2 /<br>
squid 25482 root txt REG 8,1 6430816 41946763 /usr/sbin/squid<br>
squid 25482 root mem REG 8,1 47648 10093387 /lib/x86_64-linux-gnu/libnss_<wbr>nis-2.23.<br>
so<br>
</div></div>squid 25482 root mem REG 8,1 93128 10093399 /lib/x86_64-linux-gnu/<a href="http://libnsl-2.23.so" rel="noreferrer" target="_blank">http://<wbr>libnsl-2.23.so</a><br>
<span class="">squid 25482 root mem REG 8,1 35688 10093400 /lib/x86_64-linux-gnu/libnss_<wbr>compat-2.<br>
23.so<br>
squid 25482 root mem REG 8,1 47600 10093403 /lib/x86_64-linux-gnu/libnss_<wbr>files-2.2<br>
3.so<br>
squid 25482 root mem REG 8,1 25913104 41944015 /usr/lib/x86_64-linux-gnu/<wbr>libicudata.s<br>
o.55.1<br>
squid 25482 root mem REG 8,1 22520 10093098 /lib/x86_64-linux-gnu/libmnl.<wbr>so.0.1.0<br>
squid 25482 root mem REG 8,1 26248 41948589 /usr/lib/x86_64-linux-gnu/<wbr>libnfnetlink<br>
.so.0.2.0<br>
<br>
<br>
Thank you once again<br>
<br>
CM<br>
<br>
<br>
<br>
</span><span class="">On Thu, Mar 23, 2017 at 1:59 PM, Eliezer Croitoru <mailto:<a href="mailto:eliezer@ngtech.co.il">eliezer@ngtech.co.il</a>> wrote:<br>
There is another option!<br>
The log rotate script is doing something nasty or the systemd service file start up squid in a weird way.<br>
The output of:<br>
$ systemctl status squid<br>
$ top -n1 -b<br>
$ ps aux<br>
$ df -h<br>
$ netstat -ntulp<br>
$ lsof -n|egrep "proxy|squid"<br>
<br>
How many clients this system has?<br>
Is the system facing the Internet directly or behind some nat(aws or another provider)?<br>
<br>
The above are the basic required data to understand the situation.<br>
<br>
All The Bests,<br>
Eliezer<br>
<br>
----<br>
<a href="http://ngtech.co.il/lmgtfy/" rel="noreferrer" target="_blank">http://ngtech.co.il/lmgtfy/</a><br>
Linux System Administrator<br>
</span>Mobile: tel:%2B972-5-28704261<br>
Email: mailto:<a href="mailto:eliezer@ngtech.co.il">eliezer@ngtech.co.il</a><br>
<br>
<br>
From: squid-users [mailto:<a href="mailto:mailto">mailto</a>:<a href="mailto:squid-users-bounces@lists.squid-cache.org">squid-users-<wbr>bounces@lists.squid-cache.org</a>] On Behalf Of Chee M Gui<br>
<span class="">Sent: Wednesday, March 22, 2017 5:18 PM<br>
</span>To: mailto:<a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.<wbr>squid-cache.org</a><br>
<span class="">Subject: [squid-users] Squid stopped working after cache.log and access.log rotation<br>
<br>
<br>
Hi All<br>
<br>
We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS. It ran fine at first but stopped working after a while. telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests. Then we realized that there is no new access.log file. The access.log file stopped rotated at 6:24AM on 3/17/2017. It looks like Squid wasn't able to create a new access.log? We could not find any error message in syslog or the cache.log. We haven't rebooted the server because we want to know what went wrong. It isn't the firewall blocking Squid because Squid was working fine all the while until recently. Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.<br>
<br>
root@paproxy:/var/log/squid# ls -alt<br>
total 15536<br>
drwxr-xr-x 2 proxy proxy 4096 Mar 21 06:25 .<br>
-rw-r----- 1 proxy proxy 63 Mar 21 06:25 cache.log<br>
drwxrwxr-x 9 root syslog 4096 Mar 21 06:25 ..<br>
-rw-r----- 1 proxy proxy 63 Mar 20 06:25 cache.log.1<br>
-rw-r----- 1 proxy proxy 83 Mar 19 06:25 cache.log.2.gz<br>
-rw-r----- 1 proxy proxy 15759111 Mar 17 06:24 access.log.1<br>
-rw-r----- 1 proxy proxy 117223 Mar 17 05:52 netdb.state<br>
<br>
Any ideas what went wrong?<br>
<br>
Thank you very much in anticipation.<br>
<br>
Gui<br>
______________________________<wbr>_________________<br>
squid-users mailing list<br>
</span>mailto:<a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.<wbr>squid-cache.org</a><br>
<div class="HOEnZb"><div class="h5"><a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/<wbr>listinfo/squid-users</a><br>
<br>
<br>
<br>
<br>
--<br>
Chee Meng Gui<br>
Function Engineering<br>
<a href="tel:650-833-0660" value="+16508330660">650-833-0660</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Chee Meng Gui<br>Function Engineering<br>650-833-0660</div>
</div>