Mainly the issues I am running into are that it seems to require all sorts of libraries just to build it, many of which aren't in the Jolla repos and I'm getting tired of hunting them down.

I've started experimenting with using Julius instead, as it is written in C++ with almost no dependencies. It's a bit more work creating the language models as there is no equivalent to CMU's lmtool, but I think it might be a valid alternative on the Jolla.