> Make sure you have 15 MB of application memory free before starting it.
This was the problem. Interestingly enough, I had enough free RAM total, just not enough free program RAM allocated. The Memory control panel applet kept setting the slider back to 1/2 way, too. Once I freed up enough RAM so that the 1/2 way mark left enough RAM for MAMEBoy, the slider was set to the "high water mark", and I can run it again without problems.
Is there not a way for MAMEBoy to request enough RAM from the system when starting up? I thought the OS was supposed to dynamically allocated RAM between program and storage.