A number of disastrous missteps – a holiday in Hawaii and a forceful handshake – have played into a view that the prime minister is out of touch