General dependencies discussion (was: ARTICLE - ESR gives up on Fedora)

[Nigel Stewart]

>  Libraries enable code re-use.  Now programmers don't have to
> continuously re-invent the wheel; they can build on the word of
> others.  Shared libraries mean you only have to update one .so to fix
> a bug or security hole; you don't have to rebuild/update everything
> that uses it.  Sounds like a win, right?  But getting everything you
> need together in one place, all configured the same way at once, often
> proves an incredibly daunting task.
> 
>  Is there a general solution to this problem?

	I think there is a way forward, but it's not out there
	in deployment yet.  It's a concept I have been thinking
	about since at least 2002, and the discussion here prompted
	me to take a fresh look it.  It amounts to a fairly sweeping
	change to the way a system is organised, and I'm not sure
	there is a tidy migration path from the current situation.
	Anyway, here goes...

	An operating system and applications collectively occupy
	a complex filesystem structure.  If any of those individual
	files go astray, are modified, or the incorrect version,
	or havn't been tested in combination, overall stability
	and reliability suffers.  For a particular
	application there is no automatic way to audit all the
	dependencies to diagnose a problem - the granularity is
	just too fine.  Software development has resulted in a
	vastly complex environment in which we hope our programs
	will operate reliably.

	From a security point of view we also have a situation where
	all programs have access to the entire filesystem, far in
	excess of what is actually required for that program to
	operate.  There exists various ways to "sandbox" an environment,
	but it is not used as the default mode of operation.

	The essence of the proposal is that packages should form the
	granularity of system, rather than individual files.  Each
	package contains an organised collection of files, in
	a zip archive manner.  In addition to package-specific files,
	there is a means to link to files that exist in other packages.
	In this manner the package forms a self-contained file system
	that provides everything necessary for that package to run.

	There are several advantages to this arrangement:
	- We know at startup if all dependencies are satisfied,
	  this makes package testing easy.
	- Dependencies are explicit at the package level, rather
	  than being buried in the internals of the software, or
	  being runtime dependent.
	- Packages don't need to be installed as an administrator,
           or even extracted.  The system could transparently cache
	  the appropriate working set of decompressed files.
	- Packages can be a mixture of root and user space.
	- Various security mechanisms can be applied at the package
	  granularity - digital signatures, limits to user data,
	  network and device access, etc.  A "hello world" program
	  has no need for network access, or user documents.
	- Multiple versions of applications can be accomodated,
	  given that each runs in it's own environment.
	- Packages can be added to the system without affecting
	  the availability of existing (known working) packages.
	- Packages can depend on specific versions of libraries
	  or interpreters (Python 2.3 vs Python 2.4, Qt3 vs Qt4, etc)
	- Being able to configure various self-contained environments
           such as "shell scripts", "gcc 2.96, Qt3, and "gcc 4, Qt4"
	  "Java 1.4.1, Eclipse" that each operate independently.

	Unresolved issues:
	- How to provide access to user data - deciding which files
	  can be visible, or opened for input or output.
	- How packages might populate themselves into desktop menus,
	  right-click actions, and so on.
	- How packages might be updated, flagged or disabled in
	  response to security issues, big fixes, or new features.
           (It may be appropriate to download by default and selectively
            activate, for maximum user interactivity)

If all of this sounds deluded, I am relying on you all here
to point out the show-stopping flaw... :-)

Nigel