Do you believe we use words like "racist" around too much?

Calling a person a racist is a serious allegation. Why are we over using it so much with people like Don imus, Dog, and of course illegal immigration? Do you notice that people who oppose illegal immgration rarely even bring up race but people promote it do?