http://maemo.org/packages/view/rtorrent/

as you can see there should be a package in devel-repository...

but be warned... enabling devel repository can cause huge problems because the apps there are only mentioned for developers and not the endusers... they can be unoptified and unstable....

i haven't tested it... so you could enable the rep, install rtorrent and disable it immediately... (do NOT update any of your apps when devel rep is enabled!)

this said you may want to stuck to transmission... or you have no fear to reflash your device... then go straight ahead and try