FOSS benefits the field of computer engineering (was: Malware "best practices")

[Fred]

On Friday 28 July 2006 10:48, Ben Scott uttered thusly:
> On 7/27/06, Jason Stephenson <jason at sigio.com> wrote:
> >>  The MySpace "worm" does highlight something important: Programmers
> >> keep making the same stupid mistakes, over and over and over and over
> >> and over again.
> >
> > As a programmer, I can tell you why. Most programmers are not well
> > versed in the art or the science (if there really is any) of
> > programming.
>
>   Indeed.  From what I've seen, most aspects of computing are not
> approached with any kind of engineering discipline.  That includes not
> only traditional programming, but software design, data design, UI
> design, integration, systems management, etc. People just throw stuff
> together and hope it works.  Small wonder that things fall down so
> often.

More often than not, management does not want to wait to see the results of 
real software engineering. They want *immediate* gratification. They want to 
see bells and whistles and pretty flashy things. If you actually take the 
time to *do it right*, you'll be fired. Thus, there is constant pressure to 
get something going at all, and then to run with *that ball*, not allowing 
time for real design, etc.

On the other hand, I've been at other places that placed too much emphasis on 
"doing it right" and nothing ever got done -- mainly because focus would 
shift faster than the design process, which becomes annoying after doing all 
the hard work to get it right.  As an engineer, you want to see a thing to 
completion. As a business, you want to capitalize on an ever-shifting 
market. This pits engineering and marketing at loggerheads.

...
> > In the commercial realm of closed source
> > software most programmers only get to see the code of the project(s) to
> > which they are assigned. They never get to see much code that's better
> > or worse than what they are used to seeing.
>
>   I think you've got a very good point there.  Not just the code,
> either, but the whole experience.  Learning a FOSS project means you
> have to learn the data structures and program flow of someone else's
> code.  You get to appreciate the value of good design and good
> comments/documentation, and/or see how hard the lack of same makes
> picking up a project.  You also get to see, first hand, how software
> evolves over time, and the consequences of bad work.  You also see how
> abuses and bad assumptions lead to software failures in the field.

And not just with FOSS, either. It is a problem endemic in the entire 
software industry. FOSS does have the nice benefit of NOT being so 
profit-driven so you actually do have time to do it right -- or to do a 
re-write if it were done wrong.

>   Wow.  This is actually pretty deep stuff.  Another benefit of FOSS.
> I haven't seen this particular angle "sold" before.  Wow.  This is a
> huge point.  FOSS is good because it improves the overall quality of
> software *everywhere*.
>
> -- Ben

Yes, very good thing. Now, as the complexity of software grows seemingly 
without bound, the meaning of "good software" will undoubtedly shift 
greatly.

Looking at how biological systems evolve, develop, and function can give you 
even deeper insights about software and where it could head. Biological 
systems are kinda like very badly written software that has become so robust 
that it can stand up to its own bad design!!!! It seems very contradictory, 
but holds true. There have been some attempts at doing software this way -- 
the whole genetic programming and genetic algorithm development over the 
years. But that's more "pie in the sky" today -- I am not aware of any major 
projects that uses genetic programming, though there are quite a few that 
makes use of genetic algorithms -- but usually in limited contexts, such as 
time series predictions, etc.

The truth about software is that, even in a well designed system, software 
tends to be very fragilistic. A single wrong symbol somewhere in the code 
can bring an entire system down, as was the case many years ago with the 
5ESS switch problem that brought down the phone system in NYC -- I believe 
it was a single '+' that shoud've been a '-' or something equally subtle, 
out of millions of lines of code. If you are talking about C code, a single 
uninitialized pointer can create subtle bugs that are difficult to track 
down. Yes, there are tools to combat this, but sometimes the tools 
themselves get in the way. Higher level languages like Python, Perl, Java, 
etc. eliminate some of these problems but introduce others. I'm having to 
deal with a site or two of mine seg faulting mysteriously running 
essentially the same PHP code as many other sites working flawlessly. And 
what's even stranger is that the problem goes away when APC is disabled. So 
I don't know if the problem is in APC, PHP, some subtle interaction between 
the two, or maybe involving Apache as well. And do I have the time to track 
this down? Nope. I have access to all the source code for all these FOSS 
tools and yet I simply don't have the time to ferret out such a subtle 
systems-level bug.

FOSS is great, but man, when things goes wrong, OUCH!

-Fred