Yeah, most Swedes speak English fluently. So its rather the way around, that its hard to learn fluent everyday Swedish just because "those bastards don't speak Swedish with foreigners".
Well, at least according to a Italian guy I met last summer, that apparently had a hard time with that.