Well, it's all about semantics after all. For you, not taking action means actually killing these people, for me it just means not saving them, which is entirely different.
What's the morally right thing to do? Morality is a largely subjective matter, and while in some very clear scenarios there seems to be universal agreement (eg, killing a man, in a vacuum, is almost universally perceived as a bad thing), in most cases (especially in extreme cases like the ones presented) there is room for interpretation, semantics, and case by case analyses.
I don't think there is a right or bad thing to do, but I do know what would work for me. I don't claim to have the answers, in fact I don't think there's a right answer (actually those that claim to have all the right answers tend to scare the hell out of me )