Brown attended Middlebury College in Vermont, where he became the first African American member of Sigma Phi Epsilon, collegiate fraternity and as result, the national charter of SPE at Middlebury was rescinded and the fraternity became known as Sigma Epsilon, the Harlemworldmagazine.com reported.

Brown became involved in politics during his 12 years stint with the Urban League where he served as Deputy Executive Director, General Counsel, and Vice President of the Washington bureau.

In 1989, he was elected chairman of the Democratic Party National Committee, becoming the first African American to hold the top position in a major political party in the United States.

In 1993, Clinton appointed Brown to be the first African American secretary of commerce, a position he held until his death in 1996.

On January 8, 2001, Brown was presented, posthumously, with the Presidential Citizens Medal by President Bill Clinton.