If anything, I would assume that, all other things being equal, a speaker with a stronger magnet would be more sensitive and hence need less power than one with a less strong magnet. It's not necessarily a question of the type of the magnet, but simply the build of the speaker. Some are deliberately more or less sensitive than others.

Have you tried passive speakers like the kind that used to be popular in the late 1990s? Before they started making them with built-in amplifiers. Basically nothing but overblown headphones. You can still get them in charity or pound shops.

Another possibility is that your speaker simply does not work. Have you tried it with some other audio source?