Boris Johnson could hamper Tory electoral success, says poll