QML ListView has serious issues if it's delegates aren't unifromed, all the same height and width.
To jump ahead, proper solution would be to use a Repeater with a Column all wrapped up in Flickable (what basically makes it a ListView, but with more control)
Inside a Repeater you do calculate it by hand, or leave it to Text/Label/Item component to calculate it's own implicitHeight. Then you just use it as height of your delegate.
That's what I would try.