Brexit left 42% of Britons worse off, survey suggests