Even Policies Are Not Set-It-And-Forget-It