The prime minister’s badly timed holiday has become a source of anger but while the reaction is valid it’s the wrong thing to focus on