Hi there, the actual phrase used in Han Yu's text is 夫佛本夷狄之人. Unfortunately, 夷狄 is impossible to translate, though it has similar meaning with 异地. It could be translated to "outsider" or "foreigner" as well. But since 夷狄 had a political import, I think "barbarian" is apt. Classically, it was used to describe the territories beyond the control of Imperial China.
I didn't know that "barbarian" doesn't have such a condemning tone in American English. Perhaps you're right, the extent of condemnation is cultural.