I would advise against the toolbar: it takes valuable screen estate. Just overlay a couple of transparent buttons on the map, the same way as built-in N810 mapping application does it: this way the icons will be easily usable, they will not steal screen estate, and they will not obstruct the map.
The idea is to catch drags that end up crossing themselves and treat only these drags as zoom operations. In other words, if the end of your drag is in <epsilon pixels from the start, and if the drawn shape has an extent (i.e. it's not just a segment), then you zoom, otherwise you pan.