Jeff Lewis says daughter, 5, has threatened to not see him