Several workers expected to quit when asked to return to the office