Dev Ops - architecture (local not cloud)

[Joshua Judson Rosen]

"Greg Rundlett (freephile)" <greg at freephile.com> writes:
>
> We are replacing a monolithic software development IT infrastructure where
> source code control, development and compiling all take place on a single
> machine with something more manageable, scalable, redundant etc.  The goal is
> to provide more enterprise features like manageability, scalability with
> failover and disaster recovery.
>
> Let's call these architectures System A and System B.  System A is
> "monolithic" because everything is literally housed and managed on a single
> hardware platform.  System B is modular and virtualized, but still running in
> a traditional IT environment (aka not in the cloud).  The problem is that the
> new system does not come close to the old system in performance.  I think it's
> pretty obvious why it's not performing: user home directories (where
> developers compile) should not be NFS mounted. [1]  The source repositories
> themselves should also not be stored on a NAS.
>
> What does your (software development) IT infrastructure look like?


Central build server in a VM for controlled autobuilds and releasable
binaries. Developers have accounts with NFS-exported home-directories
on that server, but they're not `their authoritative $HOME'; developers'
home-directories on their desktops are local disks on those desktops,
with desktops backed up to a (different) central server in case
someone's desktop dies. Distributed version control using Bazaar;
developers have complete local copies (with history) of trunk and
whatever other branches they're working on, on their desktops
(or laptops, as the case may be). Changes are developed, built,
and tested primarily via the individual desktops (which are
collectively *very fast* with *lots of fast storage*), and task
branches are folded in and push up to the central trunk (and/or
whatever release branch people are working in) when they're ready
for `release' to QA. The DVCS analog of checking out an svn branch
into the developer's desktop-local home-directory is *fast*,
requires little space on their desktop, and imposes basically
no load on the central server or on the network. And in
the event that one machine does go down (either one of
the developer desktops, or one of the servers; either as
as a failure or for routine maintenance), the impact to everyone
is minimal.

It's hard to know what the significance of `large svn repositories'
is due to the way that svn repositories are used; how big are the
*checkouts* from the repository (in all of the terms you've already
used--number of files (aside from svn metadata files), file-sizes...)?

The idea of making everybody's everything happen on one big
`micro mainframe' server sounds crazy to me these days--the cost/benefit
ratio we get from distributing the load and reliability is just
too good (not even considering the other benefits that may
or may not apply outside of our specific business context).


> One of the specific problems that prompted this re-architecture was disk
> space.  Not the repository per se, but with 100+ developers each having one or
> more checkouts of the repos (home directories), we have maxed out a 4.5TB
> volume.
>
> More specifically, here is what we have: 
> system A (old system)
> single host
> standard Unix user accounts
> svn server using file:/// RA protocol
> 4.5TB local disk storage (maxed out)
> NFS mounted NAS for "tools" - e.g. Windriver Linux for compiling our OS
>
> system B (new system)
> series of hosts managed by VMWare ESX 5.1 (version control host + build
> servers connected via 10GB link to EMC VNXe NAS for home directories and tools
> and source repos
> standard Unix user accounts controlled by NIS server (adds manageability
> across domain)
> svn server using http:/// RA protocol (adds repository access control and
> management)
> NFS mounted NAS for "tools", the repositories, the home directories
>
> Notes: 
> The repos we're dealing with are multiple "large" repositories eg. 2GB 43,203
> files, 2,066 directories.
> We're dealing with 100+ users
>
> [1] http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/
> com.ibm.aix.prftungd/doc/prftungd/misuses_nfs_perf.htm

-- 
"'tis an ill wind that blows no minds."