shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

[Michael Pelletier]

I used multithreading to very good effect in a project I worked on a while
back - it was almost a textbook example, I reckon.

The code needed to scan every inode in each filesystem to handle UID and GID
changes, but when dealing with an NFS server, where the round-trip network
latency is added to each filesystem operation, the speed of processing was
unacceptably slow.

So, in the main thread I had the "find"-like code (FTS, as I recall, from
BSD) walking the directory tree, and for each filesystem element it
identified, I spawned a new thread which ran the stat() on the file, looked
up the UID & GID in the change table to check if a change was needed, then
performed the change if necessary.

This way, all the main thread had to wait on was NFS calls to walk the
filesystem by reading directory inodes, not any NFS stat or update calls.
Needless to say, the NFS client and server needed to be tuned as well to
allow for a large number of simultaneous pending client calls for best
performance.

Properly tuned, the performance was about an order of magnitude better for
NFS-mounted filesystems.

It was a very useful application for threading, though it was the first and
last time in my programming career where it was worth the trouble of
implementing it.

	-Michael Pelletier.

-----Original Message-----
From: gnhlug-discuss-bounces at mail.gnhlug.org
[mailto:gnhlug-discuss-bounces at mail.gnhlug.org] On Behalf Of Ben Scott
Sent: Wednesday, October 22, 2008 11:09 AM
To: Greater NH Linux User Group
Subject: Re: shell, perl, performance, parallelism, profiling,etc. (was:
Upgrade guidance)

On Wed, Oct 22, 2008 at 9:39 AM, Bill Ricker <bill.n1vux at gmail.com> wrote:
> MP in any language is tricky ...

  I've done very little MP work, but that has been my impression.
Fundamental to the thinking of most programmers (myself included) is that
variables won't change unless the code writes to them.

> Perl has tried a couple times to make it safe and easy, but MP 
> intrinsically isn't ...

  Ah, much like security.  :)

> A shell pipe with sort in it won't be doing more than loading the 
> executable / spawning the process in parallel, since the sort won't 
> write until it's done reading.

  Right.  The case-in-point is, perhaps, an unusual situation.  There aren't
many forks involved, just a handful of programs, and the two big ones can
(in theory) run concurrently on the same input data set.

-- Ben
_______________________________________________
gnhlug-discuss mailing list
gnhlug-discuss at mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/