As title said, I definitely think it's worth it, but to be fair it depends on what else you have, and personal tastes. In general (as you can guess from the below post) I am quite fond of portable consoles of any kind; I find them much better than smartphones for gaming, mainly due to their form factor. The GP2X was fairly unique in that for a while it was the only linux-based, fully open handheld. In other words you can develop on it and install whatever you want without hacking the firmware. It runs a version of linux and all the libraries for it are open-source. The company that made the machine is not going to try to block you with a firmware update, or try to sue you if you sell programs you made yourself for it (rather, they actually encourage it!). There are other alternatives now, but the GP2X's strength was and is the development community around it.
Compared to commercial consoles
The NDS can also load ad-hoc software (via one of the miniSD cartridges), but it is quite underpowered for emulation, and the screen resolution is very small (eg. can't do NES full screen, need to crop). GBA is even less powerful, but NES emulation does work ok (I've played Zelda2 to completion on it).
Perhaps the best alternative to GP2X is the PSP, as long as you are willing to hack the firmware. Most of the good emulators for GP2X also exist on PSP (e.g. Picodrive, gPSP). The PSP is also able to play PSX games (either from the PS Store or "fallen off the mule"). Therefore if you have a hacked PSP, you might not find the GP2X too interesting for emulation alone. Aside from the firmware, the downside of the PSP is apparently (don't own one so can't confirm) that it suffers from screen blurring (GP2X screen is very good and sharp) and the D-pad is apparently worse on the PSP.
Aside from emulation, the GP2X has attracted many more devs so it has a very large homebrew community with very decent games. It's probably the homebrew platform that has the most development on it (aside from the PC - definitely larger than the DreamCast which was quite big). Stuff is still coming out for it.
Compared with other "Open" Handhelds
There are actually 3 families of the "GP2X".
There is also one machine made by a Chinese company (the Dingoo A320), and one upcoming (?) device called the OpenPandora.
I've never owned a GP32, but I believe it was somewhat underpowered to emulate much (same issue as the GBA). It's no longer available as the original company went under (the main engineers split and made the company that produced the GP2X and Wiz). Worth mentioning though since this machine started it all as it was the first successful open handheld.
This is the machine that made the name famous. For a while, it was the only machine available that gave you decent power. I've bought mine quite late (when it was well established) and have been very happy with it.
Genesis and earlier machine are emulated 100% perfectly (60fps with 44KHz sound). NeoGeo/CPS2 get 50-60 Fps with 11KHZ sound. Snes and GBA are playable with sound, but expect lower framewates (20-30fps).
Form factor is similar to the GBA, and it takes 2 AA batteries. Battery life is 5-6 hours of gameplay using 2800mHa rechargeable batteries. CPU is clocked at ~200 Mhz, overclockable to ~250-280 Mhz for more demanding emus.
I'm not sure it is still produced but you might find it on sale on eBay. There are a different models though; the F200 is the latest, which is identical to the previous one except that its white and features a touch screen.
The Wiz is the current model. It is almost as good as the GP2X with a few caveats.
The good is that it features a beautiful OLED touchscreen. It has a smaller form factor (closer to the GB micro) and has more powerful hardware than the older GP2X; CPU runs at ~533 Mhz. It takes SDHC cards, and features and internal battery (~6-7 hour life). Like the older model, it also has a very active homebrew community. Files from older versions are not directly compatible (at binary level) so games and emus need to be recompiled from source. Most devs have followed-up but of course it means there is less choice than for the GP2X. That being said, dev tools are still opensource and free, so more will come as long as the machine keeps being available.
The bad is that the hardware is apparently not as well documented as the GP2X, so devs have not yet unlocked all the potential of the machine. For example it is theoretically capable of running a PSX emulator but until people figure out how to code the OpenGL on it well, it will remain slow (emu exists). This unfamiliarity can also create other issues, e.g. the screen is a 240x320 phone screen which can produce a "tearing" effect if it is not synced properly. As a result many direct ports from the GP2X can have some display issues if the devs did not do it right. SNES emulation is faster but This is getting better now though.
In spite of the issues, I do believe the Wiz is the best option out there. My only gripe with it is the smaller size. I have big hands, and the size of the older GP2X was perfect. A Wiz of the same size would be heaven.
This is a cheaper machine made by a Chinese company. It is interesting because originally it costed about half the price of the Wiz. Some people prefer it over the Wiz. (caveat emptor - it has better margins than the Wiz so reviews from online stores tend to say it's better. this is not true, read on).
The machine takes microSD cards. It does not run Linux natively but it can be installed (bit annoying, I actually haven't tried). I'm not sure of the hardware specs, but I would say it's between the GP2X and Wiz. It natively comes with several emulators, but except the NES one they're quite flaky.
The machine has however one big hardware flaw which killed it in my book (I stopped using it other than backup video player if my Wiz runs out of juice). The X and B buttons cannot be pressed simultaneously. This is really annoying for some games (eg. Super Mario).
It still works other than that, so it's an ok machine. I would say worth getting for somebody does not want to pay for a Wiz and can't find an older GP2X either.
There is talk of a new handlhed which supposedly is very powerful (800Mhz w/3D hardware, similar specs of the Nokia N900). It looks very good on paper and the pictures are nice, but the big problem is that it has not been released yet. I do hope it comes out though, because it has a lot of good ideas in it (double SD cards, WiFi, keyboard + digital pads + analog pads, 800x480 screen). Prototype units do exist so it does not seem like a scam, but until it's widely available I won't count on it.
Where to get programs
For software for any these machines, (including A320 and Pandora), check out http://gp32x.com . They have dedicated archives for each.
Phew, longer post than what I expected to type, but hope it helps!
[download a life]