Start reading here:
http://msdn.microsoft.com/en-us/library/ff556232.aspx
and try /3GB and/or /PAE (search for it in the above) if you’re on XP or 2003. If you’re on Vista or 7, go here:
http://msdn.microsoft.com/en-us/library/ff542202.aspx
and try PAE and very possibly IncreaseUserVA.
I have found very good performance of 32-bit Windows as virtual guest, in a VirtualBox environment hosted on 32-bit Fedora 14 Linux, using the PAE kernel on Linux and /PAE (or BCDEdit /SET PAE) in the virtual. If you use PAE in the kernel, do make sure to use PAE in the virtual, or there will be an extra and expensive memory processing step required at all times. This is to be done even if the virtual is configured with smallish RAM, e.g., 1G.