Every few months (about 2-3), this squid server will start spawning massive numbers of (squid) -DY processes. These spawn at a rate of 1 every 15 minutes, which is kinda suspicous. The original dnsserver processes continue to run, although the new squid processes complain about not being able to start the dnsserver child processes. The new squid processes are supposedly spawned by init. The orininal process is spawned using an rc script.
The following excerpt from the store log appears to be the first entry of one of these processes starting up:
2005/12/13 02:07:11| helperHandleRead: FD 8 read: (110) Connection timed out
2005/12/13 02:07:11| ipcacheParse: Got <NULL> reply
2005/12/13 02:07:11| WARNING: dnsserver #3 (FD 8) exited
2005/12/13 02:07:11| Too few dnsserver processes are running
2005/12/13 02:07:11| Starting new helpers
2005/12/13 02:07:11| helperOpenServers: Starting 5 'dnsserver' processes
2005/12/13 02:07:11| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:10:20| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:13:29| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:16:38| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:19:47| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:19:47| helperHandleRead: FD 9 read: (110) Connection timed out
2005/12/13 02:19:47| ipcacheParse: Got <NULL> reply
2005/12/13 02:19:47| WARNING: dnsserver #4 (FD 9) exited
2005/12/13 02:19:47| Too few dnsserver processes are running
2005/12/13 02:19:47| storeDirWriteCleanLogs: Starting...
2005/12/13 02:22:56| Finished. Wrote 57 entries.
2005/12/13 02:22:56| Took 0.0 seconds (218390.8 entries/sec).
FATAL: The dnsserver helpers are crashing too rapidly, need help!
Squid Cache (Version 2.5.STABLE10-20050523): Terminated abnormally.
CPU Usage: 165.870 seconds = 59.090 user + 106.780 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 466
Memory usage for squid via mallinfo():
total space in arena: 2489 KB
Ordinary blocks: 2369 KB 25 blks
Small blocks: 0 KB 0 blks
Holding blocks: 208 KB 1 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 120 KB
Total in use: 2577 KB 104%
Total free: 120 KB 5%
2005/12/13 02:22:59| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
The message about the dnsserver helpers only appears once. While the squid -DY processes spawn about once every 15 minutes, there are still only 5 dnsserver processes. The original "good" squid -DY process is also still running.
The following excerpt is from the ps auxf command:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1348 484 ? S Oct17 0:17 init [3]
...
root 1185 0.0 0.1 5412 1560 ? Ss Oct17 0:00 /usr/local/squid/sbin/squid -DY
squid 4013 0.0 0.1 5628 1668 ? S 11:04 0:00 \_ (squid) -DY
squid 4014 0.0 0.1 5628 1664 ? S 11:04 0:00 \_ (squid) -DY
squid 1189 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1190 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1191 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1192 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1193 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1194 0.0 0.0 1196 280 ? Ss Oct17 0:00 (unlinkd)
root 1234 0.0 0.0 1608 712 ? Ss Oct17 0:03 /usr/sbin/cron
...
ntp 1256 0.0 0.3 3576 3572 ? SLs Oct17 0:44 /usr/bin/ntpd -n -u ntp:ntp
squid 2394 0.0 0.5 8172 5144 ? S Dec12 0:00 (squid) -DY
squid 2407 0.0 0.5 8172 5144 ? S Dec12 0:00 (squid) -DY
squid 2408 0.0 0.5 8172 5144 ? S Dec12 0:00 (squid) -DY
...
The following excerpt from the command cat cache.log|grep "Starting Squid", shows that a new rogue process starts about every 15 minutes:
...
2005/12/13 11:20:14| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 11:36:02| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 11:51:50| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:07:38| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:23:27| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:39:15| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:55:03| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:10:51| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:26:39| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:42:27| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:58:15| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 14:14:03| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 14:29:51| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 14:45:39| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 15:01:28| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
...
This is a generic squid 2.5STABLE10 install on a gentoo machine using a 2.4 kernel. There's nothing particularly special about it, that I'm aware of anyway. Certainly, nothing that I can think of which would cause it to spawn rogue processes every 15 minutes. It seems curious that the system would run fine for months on end, and then suddenly exhibit this behavior. Any ideas about what could be causing this? Any help you could offer would be greatly appreciated.
Thanks,
GD
The following excerpt from the store log appears to be the first entry of one of these processes starting up:
2005/12/13 02:07:11| helperHandleRead: FD 8 read: (110) Connection timed out
2005/12/13 02:07:11| ipcacheParse: Got <NULL> reply
2005/12/13 02:07:11| WARNING: dnsserver #3 (FD 8) exited
2005/12/13 02:07:11| Too few dnsserver processes are running
2005/12/13 02:07:11| Starting new helpers
2005/12/13 02:07:11| helperOpenServers: Starting 5 'dnsserver' processes
2005/12/13 02:07:11| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:10:20| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:13:29| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:16:38| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:19:47| WARNING: Cannot run '/libexec/dnsserver' process.
2005/12/13 02:19:47| helperHandleRead: FD 9 read: (110) Connection timed out
2005/12/13 02:19:47| ipcacheParse: Got <NULL> reply
2005/12/13 02:19:47| WARNING: dnsserver #4 (FD 9) exited
2005/12/13 02:19:47| Too few dnsserver processes are running
2005/12/13 02:19:47| storeDirWriteCleanLogs: Starting...
2005/12/13 02:22:56| Finished. Wrote 57 entries.
2005/12/13 02:22:56| Took 0.0 seconds (218390.8 entries/sec).
FATAL: The dnsserver helpers are crashing too rapidly, need help!
Squid Cache (Version 2.5.STABLE10-20050523): Terminated abnormally.
CPU Usage: 165.870 seconds = 59.090 user + 106.780 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 466
Memory usage for squid via mallinfo():
total space in arena: 2489 KB
Ordinary blocks: 2369 KB 25 blks
Small blocks: 0 KB 0 blks
Holding blocks: 208 KB 1 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 120 KB
Total in use: 2577 KB 104%
Total free: 120 KB 5%
2005/12/13 02:22:59| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
The message about the dnsserver helpers only appears once. While the squid -DY processes spawn about once every 15 minutes, there are still only 5 dnsserver processes. The original "good" squid -DY process is also still running.
The following excerpt is from the ps auxf command:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1348 484 ? S Oct17 0:17 init [3]
...
root 1185 0.0 0.1 5412 1560 ? Ss Oct17 0:00 /usr/local/squid/sbin/squid -DY
squid 4013 0.0 0.1 5628 1668 ? S 11:04 0:00 \_ (squid) -DY
squid 4014 0.0 0.1 5628 1664 ? S 11:04 0:00 \_ (squid) -DY
squid 1189 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1190 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1191 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1192 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1193 0.0 0.0 2044 588 ? Ss Oct17 0:00 (dnsserver)
squid 1194 0.0 0.0 1196 280 ? Ss Oct17 0:00 (unlinkd)
root 1234 0.0 0.0 1608 712 ? Ss Oct17 0:03 /usr/sbin/cron
...
ntp 1256 0.0 0.3 3576 3572 ? SLs Oct17 0:44 /usr/bin/ntpd -n -u ntp:ntp
squid 2394 0.0 0.5 8172 5144 ? S Dec12 0:00 (squid) -DY
squid 2407 0.0 0.5 8172 5144 ? S Dec12 0:00 (squid) -DY
squid 2408 0.0 0.5 8172 5144 ? S Dec12 0:00 (squid) -DY
...
The following excerpt from the command cat cache.log|grep "Starting Squid", shows that a new rogue process starts about every 15 minutes:
...
2005/12/13 11:20:14| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 11:36:02| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 11:51:50| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:07:38| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:23:27| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:39:15| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 12:55:03| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:10:51| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:26:39| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:42:27| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 13:58:15| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 14:14:03| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 14:29:51| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 14:45:39| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
2005/12/13 15:01:28| Starting Squid Cache version 2.5.STABLE10-20050523 for i686-pc-linux-gnu...
...
This is a generic squid 2.5STABLE10 install on a gentoo machine using a 2.4 kernel. There's nothing particularly special about it, that I'm aware of anyway. Certainly, nothing that I can think of which would cause it to spawn rogue processes every 15 minutes. It seems curious that the system would run fine for months on end, and then suddenly exhibit this behavior. Any ideas about what could be causing this? Any help you could offer would be greatly appreciated.
Thanks,
GD