There's a couple of issues - firstly (and most importantly) a lot of files in /usr are accessed before the other filesystems are mounted, so you won't be able to boot after moving the files. Secondly, as said, there's performance impacts - that's unlikely to be significant for just reading files, but some operations were benchmarked at something like 30x slower.