Committee has delayed decision after some Liberals criticised proposal but Keith Pitt says it should ‘do its job’