Stay Warm with Fewer Layers