Apache2 troubleshooting

[Fred]

Just some general tips on monitoring Apache.

You can also try running "top" to see everything that's running on your 
server. If you have a script running for 10 hours, it'll show up there.

Having found such a beast, you can also use "strace" to get an idea of what 
it's doing. Strace can look at processes that are already running using the 
option to specify the pid.

I find it useful to keep a few continuous dumps open just so I can catch 
something weird going on. "ssh yourdomain.com tail -f /your/logfile" is your 
friend. You can at least do this while tracking down the problem, and with a 
number of log files to boot.

You might also want to monitor the traffic bandwidth of your server. This can 
tell you quite a bit -- espcially if you see any unusual spikes in activity 
that you can't account for. In that case it could mean that someone turned 
your machine into a DoS zombie.

Another bit is that you can turn on Apache's Server Status monitoring and get 
a real-time snapshot of what your server is doing:

ExtendedStatus On
...
#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
<Location /server-status>
    SetHandler server-status
# The following will restrict access to monitoring.
# Configure as you see fit.
#
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
</Location>

Then you can use your browser and do "http://yourdomain.com/server-status" to 
see how active your server really is. Here's the type of output you can 
expect to see:

Apache Server Status for yourdomain.com

Server Version: Apache/2.0.59 (Unix)
Server Built: Sep 15 2006 08:10:15

Current Time: Thursday, 26-Oct-2006 05:26:34 PDT
Restart Time: Sunday, 22-Oct-2006 04:51:47 PDT
Parent Server Generation: 1
Server uptime: 4 days 34 minutes 47 seconds
Total accesses: 1292714 - Total Traffic: 18.8 GB
CPU Usage: u243.44 s295.14 cu.01 cs0 - .155% CPU load
3.72 requests/sec - 56.8 kB/second - 15.3 kB/request
14 requests currently being processed, 13 idle workers

__.._.WK......W_KK_.W__W_..KKKW__W_W.C_......................._.
................................................................
................................................................
......................................

...

This may or may not help you find your current problem, but it's a good thing 
to have in your pocket anyway. Or you can install and set up Nagios, but 
that could be overkill just for doing simple monitoring. Too much time to 
muck about with the fancy tools when a simple command-line script will do.

Yes, you can tell I am very much into the KISS principle.

-Fred