Struggle to maintain affordable housing downtown