Brexit voters pulling up the drawbridge