Family of inmate worried about his safety