Growing number of Canadians want refugee ban