Seems to be fixed in my case by changing the connection settings to Automatic or None.
Something about the search interval?
Though Automatic seems to make the battery drain faster then it ever did before so my guess there is it's always searching for new access points in this mode.