I reset all databases and settings and reinstalled. Not sure which one did it, but that worked for me (keyboard still took a while to load the first time).

Edit: I'm using Fairphone 2, so I doubt it's a port specific problem.