'Brexit will hit ethnic minorities the hardest'