'This was never about one speech' -Rep. Swalwell