He's seen it all. The good, the bad, the really, really ugly. Riley might have said 'None of that touches you', but what did he know? Spike on the other hand can say it, knowing that Buffy has to take him seriously. He isn't a lovestruck Angel, or a clueless Riley. He's Spike - the one who tried to pull her into the darkness, but who can also acknowledge that her light still shines brightly. So brightly that it brought him out from the dark and into the light.
Yup, he's the one who was in that alley, in that smashed basement, in that bathroom and on that balcony; and in the end, none of it mattered because she was "the one", the truly effulgent thing in his life.