Neighbors say Measure B could cost them their homes