shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

[Ben Scott]

On Tue, Oct 21, 2008 at 2:36 PM, Michael Pelletier <mvpel at hushmail.com> wrote:
> An important thing to remember with Perl is that the Perl
> interpreter must be loaded before the script can proceed.  This may
> explain the difference in your case, Ben.

  I ran several successive trials of each variant.  The numbers I
posted were typical results.  The first invocation was frequently a
little bit to quite a bit slower.  The very first run of anything I
tested was more than twice as slow -- I assume because nothing was
cached yet.

> On UNIX/Linux, this issue is typically addressed by setting the
> sticky bit on the Perl executable ...

  According to everything I've ever read, Linux ignores the sticky bit
on executables.  Wikipedia has a good summary:

http://en.wikipedia.org/wiki/Sticky_bit

> I'm frankly shocked to see a shell script operating faster than a Perl script

  I was surprised, too, at first, since everyone says Perl is faster.
But I think parallelism (dual core CPU) might explain it.

  Consider: If the RPM database is in cache, then the
"package-cleanup" utility is going to mainly be reading data from
already cached pages (memory mapped file I/O).  In the Perl script,
everything is serialized.  But in the bash script, both
package-cleanup invocations can -- I'm guessing -- run concurrently.
One on each core.  I think the "package-cleanup" runs are the slowest
thing by far in this process, so if they can both run at the same
time, that might do so well as to cut wall clock time *in half*.  That
might well make-up for otherwise slow tasks.

  Again, this is speculation.  :)

-- Ben