Sri Lankans have stormed and taken over the Prime Minister's office, the army has stood down.