Well, it's not called 'Little white town', that's the welcome visitors encounter when they enter Bideford because it's written on the signs. Plus, it's not exactly traditional or particularly historical, it was a description in a novel from 1855. Given this, yeah, it sets the wrong tone, and the town might want its history back rather than this terrible slug line on its signs.
In terms of re-writing history, I find that interesting, mostly because I'm a trained historian. People often seem to think 'racist memes' have roots stretching much further back into history than they do. For example, black face morris - you know, the folks with sticks and bells. It relatively recent. It gained popularity round about the same time as the black and white minstrels and, with the best will in the world, they're racist.
I think the starting point here is trying to understand why and how people might be offended. The why is easy. Slavery (from which Britain profited massively) was a form of genocide. Millions upon millions died. All because the white man thought the black man was little better than a savage. Another example, when the slave trade was banned, slave owners realised they wouldn't be able to import any more labour, and so they forcibly bred their women slaves. Forcibly. Like animals. They did this because they had the power to do it. Imagine if this was your history. And this brings us on to the how. There's such a thing called 'ancestral trauma'. It's where the descendants and the survivors still feel it, because it was so awful, so indefensible, that it sort of defies closure. It's triggering, in pretty much the same way as antisemitism triggers people these days. Maybe they're oversensitive, but I wouldn't like to draw that conclusion because it's not me who's been subjected to the trauma. If they're asking me to do x, y or z because it would make them, in their trauma, feel more comfortable, then I'm willing to listen to that.
Finally, as for rewriting history. I live in Brighton. If we were to accurately describe this town our signs would say 'Built off the back of slaves, profits from the plantations'. Pretty sure none of us are demanding that kind of accuracy.