Noel Pearson has made a fiery speech - calling for the abolition of all state Indigenous departments.