Census workers worried about contact with public