du(1) for FTP sites

[Benjamin Scott]

On Sun, May 2, 2010 at 12:08 PM, Shawn O'Shea <shawn at eth0.net> wrote:
> ... provides a stat method to get info (including filesize) on the files
> (therefore should be FTP server agnostic).

  The problem with FTP is that the output of LIST is
implementation-specific; it varies from server to server.  There's a
fair bit of variation even within system types.  Just because you know
the system type is "UNIX" doesn't mean you're going to get exactly the
same format as some other server.  Some provide a "total" line; the
date format varies; etc.

  According to the docs for ftputil, it tries to parse output for UNIX
and MSFT style directory listings, which is more than my script does,
but it's still vulnerable to the vagaries of output formats.  (And
won't work at all for a VMS server.  ;-)  )

  RFC-3659 (2007) defines some new commands, notably MLST, which is
intended to give a directory list for machine processing, in a
well-defined format.  I don't know how widely implemented MLST is,
though.  (i.e, if you do actually find a VMS FTP server, it's unlikely
to be running software new enough to implement MLST.)

  It's unfortunate that it took to 2007 for a machine-friendly format
to be defined.

  Of course, HTTP file servers are even worse.

  Sigh.

-- Ben