Check out Bernard Williams, A Critique of Utilitarianism. He discusses negative responsibility: if you have the capacity and responsibility to stop harm, you ought to. But, you shouldn't be blamed for the outcome you fail to prevent.
I'm not saying that you should take the baby in. The fact of you calling the police is already an admittance of responsibility.
You're right, no one can be blamed for your predicaments - and I clearly didn't say that. But if you fail to pull the lever, you are quite plausible responsible, in the negative sense, for the death of those five people. But I wouldn't blame you for their deaths.
Hopes this helps.