If the base resolution is not 4:3, when you scale it to 4:3 there is no way to make the pixels square. That's just the way it is. A 384x224 game is closer to 16:9 than it is 4:3, but the game's true AR Is 4:3, so therefore, if you're squishing those pixels into a 4:3 AR, they won't be square.
If you want less "blurring" the best setting would be use 1.) Your monitor's default resolution (1680x1050), 2.) Use your monitor's default AR (16:10), 3.) Keep AR correct in MAME options 3.) Use D3D9, 4.) Up the prescale to 4 or greater.
That's going to be as pixel perfect as you're going to get. I'm running a 1680x1050 LCD and it works absolutely fine at those settings.