iostat strangeness

[Bruce Dawson]

Its been several years (and major kernel versions) since I've played
with iostat, but perhaps my statements here will goad someone with more
recent experience to inject more accurate truths...

    * iostat used to "not work well" on SMP systems.
    * your "510kB/s average write on dm-5, but only 184.01kB/s average
      write on the enclosing PV?" observation may be due to "write
      behinds", caching, and seek/latency optimization.
    * I don't remember if Linux provides seek/latency optimization, but
      I suspect it does.
    * iostat essentially just does arithmetic on the counters kept by
      the kernel.
    * For long uptimes, counters can overflow and produce some *really
      strange* numbers. I would expect Linux to use 64 bit counters in
      recent kernels though.

Don't you just love documentation written by developers (I'm referring
to the iostat man page)?

Hopefully someone with more experience on later kernels (like: after
1.2) can chime in.

--Bruce

mike ledoux wrote:
> I have a server (RHEL 4u5 x86_64, 2.6.9-55.0.9.ELsmp kernel) that
> is hammering the disk subsystem, due largely to a broken-by-design
> application.  In the course of gathering stats to justify either
> replacing the application (unlikely) or upgrading the hardware to
> better cope, I've encountered some very strange output from iostat:
>
>   # uptime
>    14:21:07 up 43 days, 20:08,  6 users,  load average: 12.39, 15.20, 15.03
>   # iostat -x -k
>   Linux 2.6.9-55.0.9.ELsmp (courtmail01.jud.state.ma.us)  01/15/2008
>   
>   avg-cpu:  %user   %nice    %sys %iowait   %idle
>              7.92    0.90   14.84   13.23   63.11
>   
>   Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>   cciss/c0d0  45.66 390.89 221.06 219.62  164.22  368.01    82.11   184.01     1.21     1.02    2.31   1.32  58.00
>   dm-0         0.00   0.00  0.57 10.02   14.92   80.18     7.46    40.09     8.98     0.06    5.85   0.50   0.53
>   dm-1         0.00   0.00  0.01  0.02    1.24    0.13     0.62     0.06    49.49     0.00    6.09   2.99   0.01
>   dm-2         0.00   0.00  0.13  2.08    2.41   16.67     1.21     8.34     8.63     0.01    3.20   0.67   0.15
>   dm-3         0.00   0.00  0.03  0.18    0.24    3.42     0.12     1.71    17.73     0.00   14.98   1.78   0.04
>   dm-4         0.00   0.00  0.57 38.67   34.67  309.37    17.33   154.68     8.77     0.38    9.56   0.18   0.69
>   dm-5         0.00   0.00 194.32 553.65  475.39 1020.36   237.70   510.18     2.00     0.85    1.13   0.73  54.38
>   dm-6         0.00   0.00  0.12  0.10    0.96    0.81     0.48     0.41     8.00     0.00    8.49   2.03   0.04
>
>
> My understanding is that these numbers are averages for the entire
> uptime of the kernel.  Device cciss/c0d0 is the local RAID, as
> presented to the OS.  I've partitioned that into c0d0p1 (/boot) and
> c0d0p2 (LVM PV).  Devices dm-0 through dm-6 are LVM LVs, all coming
> out of that c0d0p2 PV.  There is a newer kernel available from RHN,
> but not a newer sysstat package.
>
> Here's the broken bit, that has me stumped.  How, exactly, is it
> possible to be seeing 510kB/s average write on dm-5, but only
> 184.01kB/s average write on the enclosing PV?  Why does c0d0p1
> (/boot) not show up at all?
>
> Have any of you seen anything like this?  If I can't trust iostat,
> I'm boned.  
>
>