Brown is a nickname that stuck. In English it usually pointed to someone with brown hair, a brownish complexion, or a habit of wearing brown cloth—useful labels in a village where many people shared the same first name. Color nicknames of this kind (Brown, White, Black, Gray) are among the oldest surname habits in English.
The idea is so ordinary that many separate families acquired Brown independently; it is not a clue to one ancestral village. German Braun and similar forms sometimes became Brown in America, but the English descriptive sense is the one that made the name ubiquitous. If your family’s story is Brown, the most honest first answer is: someone, long ago, was the brown one.