I dont think there is any problem with huge calendars on n9.
I sync my 2003 - 2012 google calendars witch is very big: about 45minutes for the first sync.
(in 2003-2007 I was using palmpilot devices)

I use calDav as you do, but did you try to use "Mail for exchange" account instead?

I tryed first and as far as I remember it works fine too..