Residents of village to be doubled in size by asylum seekers say concerns "stonewalled" by gov.