Residents concerned about losing homes to stadium