Recommendations...

[Benjamin Scott]

On Tue, Jun 15, 2010 at 10:32 PM, Michael ODonnell
<michael.odonnell at comcast.net> wrote:
>> The Linux 32-bit kernel supports PAE (the extension that allows
>> access to more than 3GB RAM).
>
> PAE ... allows the kernel to rig the page tables such that they can
> address more than the 4Gb that they'd normally be restricted to when
> using only 32bit addresses, but at the cost of an extra level of
> indirection in the page tables and a measurable performance penalty.

  Just to be clear: With PAE, virtual address space remains 32 bit (4
GiB).  Physical address space is 36 bit (64 GiB).  So each process is
still limited to 4 GiB of virtual address space (RAM, swap, hardware,
mmap'ed files, whatever), but the CPU itself can directly address more
than 4 GiB of RAM.  The MMU and page tables will give each process its
own unique mapping, including up to 4 GiB of that RAM.

  Processes can potentially indirectly access more than 4 GiB of RAM
by using memory windowing/bank swapping/etc.  This would be similar to
"Expanded Memory" from the days of the 8086.  Reserve some range of
process-addressable memory.  A special library/system call exchanges
that block of memory with another block from the
not-directly-addressable RAM.  It works, but now processes have to do
their own memory management.  Not really a good idea unless you're
stuck with a 32-bit address space for some reason.  (Much less likely
with FOSS.)

> IIRC, PAE is not necessary until you want to address *more* than
> 4Gb of RAM, though I have a nagging, fragmentary memory that some
> other issue (maybe related to PCI mappings or other I/O stuff?)
> can make PAE advisable even when you don't have more than 4Gb.

  It gets complicated.

  The physical address space is used for any memory-mapped hardware.
That includes RAM, but can also include PCI configuration space,
option ROMs, video RAM, AGP aperture, etc.  All of the latter eat up
physical address space.  There are video cards on the market with 512
MiB or more dedicated RAM.  Put a couple of those into a box and
they'll eat up over 1 GiB by themselves.

  When computers were limited to "only" a few hundred MiB of RAM max,
this wasn't a big deal.  But as you start approaching 4 GiB, you start
running out of physical address space.  This can block access to some
RAM.

  In theory, enable PAE (and modify your OS (and drivers) to support
PAE) and you're all set.  But.

  The motherboard, core logic, and mainboard firmware all have to
support PAE.  There's four more physical address lines, and the BIOS
may (I don't remember) get involved to set things up at power on.

  Not all peripheral hardware supports a physical address > 4 GiB.
Some option cards (or their firmware) misbehave if given addresses >
32 bits.  On a 32-bit PCI bus (most of them), the cards have to
support DAC, Dual Address Cycle, to transfer a 64 bit address word
using two 32 bit bus cycles.

  I'm told there is something called "memory hoisting", where the MMU
can put the peripherals below 4 GiB, and the rest of the RAM above 4
GiB.  This again goes back to motherboard/BIOS support.

  There's one other thing PAE gets used for: The NX (no execute) bit.
Software can set NX to mark a page as not executable.  Doing this
right can help defend against buffer overflow attacks.  To get that
extra bit, though, you need the bigger page tables that PAE gives you.

  Also, if you're working in the MS Windows world, most 32-bit
versions of 'doze do *not* support hardware addresses above 4 GiB,
regardless of PAE being enabled or not.  IIRC, they may even limit you
to 3 GiB or less in some cases.  This is done because most 32-bit
Windows drivers are completely unprepared to handle a physical address
word > 32 bits.  Bad things happen when you smash the stack in kernel
mode.  So if you have even 3 GiB or more RAM, on 'doze, you'll need to
go 64-bit to use it.

  (This is especially interesting because as I mentioned, the NX bit
needs PAE.  Win XP SP2 introduced support for the NX bit (Microsoft
calls it "Data Execution Prevention", continuing their tradition of
coming up with new names for no reason).  So XP enables PAE and takes
the performance hit, but then can't actually use more than 4 GiB of
hardware anyway.)

  Almost all of this stuff goes completely out the window once the
processor is running in "Long Mode" (64 bit virtual address space).
You only have to worry about dodgy hardware then.

> There used to be a config option that made a full 4GB of virtual
> address space available to every process by forcing the kernel to use
> its own page tables; support for this capability has been deprecated,
> apparently based on the attitude that anybody who wants more than
> 3Gb virtual address space should be running a 64bit OS, a rather
> condescending stance IMO.

  It may be condescending, but it also strikes me as realistic,
especially given all the gyrations the kernel must have to go through
to support it.  Why *would* you want to run a 32-bit kernel if you're
trying to do large memory stuff?

-- Ben