I got sucked into the Linux/Android handheld rabbit hole since I’ve been playing through some of my NES/GB backlog. I bought an RG34XXSP, was surprised it was significantly better than my real GBA, and suddenly got interested in the space.
My personal preferences include:
I play older games and don’t want joysticks or 16:9 screens. Raw performance rarely matters to me for that reason too. I also don’t use features like wifi or video out.
If you don’t want to read, these are my favorites that I own:
Everything 4:3: Anbernic 476H
GB, GBC, GBA, SNES: Anbernic RG CubeXX
Actually pocketable: TrimUI Smart
The top two don’t have clicky controls unfortunately. So begins the treadmill of searching for the “best” handheld I guess.
For one, it doesn’t have a nice d-pad. But I also own the original Analogue Nt and Nt Mini and found incompatibilities with both despite how much praise they get. The Analogue Pocket, for example, uniquely has a save corruption issue with certain Pokemon ROM hacks.
People who think “FPGA = Perfect! Software emulator = Fake!” are being misled by marketing. Check any Analogue product release notes to see the compatibility issues fixed in each version. If an FPGA console was magically perfect, they wouldn’t need bug fixes.

Ideal Platforms: GB, GBC, GBA, SNES (8:7), Pico-8, NGPC
Software: MuOS
This is a jack of all trades, master of Game Boy. This display works perfectly for GBA since 240px scales perfectly into 720px. It’s also ideal for SNES at an 8:7 aspect ratio. I prefer this to the RG34XXSP for GBA because it’s more comfortable, but both are great devices.

Ideal Platform: Anything 4:3
Software: F-Droid and Retroarch
There are a million handhelds that look like the 476H, but the super nice 4:3 display and d-pad focused ergonomics set it apart as a higher end device for old games, since many of the popular candy bar handhelds have 16:9 screens now. This is my only Android device. You can remove the bloatware and install Retroarch via F-Droid, then disconnect it from the internet and it’s not too bad. Still, I don’t think I’ll buy another Android device in the future.

Ideal Platform: GBA
Software:: MuOS
This is a perfect dedicated GBA due to the 3:2 screen. I like it more than than my real, modded GBA. As far as I understand, this has a nicer display than the larger RG35XXSP as well.

Ideal Platforms: Anything 4:3
Software: Quark
This is the largest device I’d consider pocketable without being an inconvenience. This is also the most “all around” device I own. It isn’t the smallest or the best at anything specific, but it is the one that’s good enough at everything.
I understand the display is small, but it’s baffling this isn’t more popular. It’s also rather old and I don’t think it’s been topped. With a 2.8” 480p screen, this would be the perfect device for me.

Ideal Platforms: Anything 4:3
Software: MinUI Legacy
This is larger than the almost-credit card sized Arduboy but it’s still the most pocketable thing in this class and I’m sad there isn’t more competition in this space. The light, circular d-pad is more comfortable than most handhelds I own. The screen is tiny and integer scaling makes it ever tinier.

Ideal Platforms: Anything 4:3
Software: MinUI with PakUI
This thing is an internet favorite, but I don’t like it because of the d-pad. This d-pad looks identical to the TrimUI Smart but it feels worse. If you press a direction, you can roll the d-pad without feeling the second click or with the second click being muted. The “unclick” of releasing a direction can also feel inconsistent. I don’t have this issue with any of my other d-pads. For this reason alone, this is the only device I own I wouldn’t recommend to anyone.
I bought the metal Hammer model because of the metal hotspot on the original, but unless you like the novelty of a premium little tank the extra weight isn’t worth it.

Ideal Platforms: Kind of not ideal for anything
Software: Onion
If you want something smaller than the Brick, with bigger controls, better software, or don’t like clicky buttons, this is the one. The only thing worse about the Miyoo Mini to the Trimui Brick is the screen resolution. The screen is unfortunately the weak point. If you’re bothered by the shimmering of non-integer scaled games or blurriness of shaders, this isn’t the handheld for you.