Billionaire buying up properties in Calif. town 'cannot understand' why locals are 'terrified'