Prime minister’s stance comes as Japan is considering raising visa issuance fees for foreign visitors for first time since 1978