Start with a pure Debian system and add all the packages needed. It's similar to Mer in concept, but based on Debian and MeeGo; instead of Ubuntu and Maemo.
Then your project is very different from my idea. Although your project idea could use the parts created by the project I have planned.