Homeowners push against more granny flats