Fr. Philip Dyer-Perry tells his congregation he won't be going to Allen Hall