In 1902, the Nathaniel Felton Junior and Senior Houses were purchased by Joseph Newhall Smith (1839-1912), a wealthy shoe manufacturer from Lynn and owner of the Boston Woven Hose and Rubber Company. Smith also purchased several abutting properties and established Brooksby Farm.
The barn, now named the Smith Barn, was built in 1903-1904 to store apples harvested from the orchards, grain for the animals and farming implements. He planted orchards and cultivated pastureland. He built a caretaker’s cottage, called the Orchard House, which still stands next door. The family also built additions onto the Senior and Junior houses and used the property as a summer residence.
The property was then passed down to Joseph N. Smith’s, daughter Annie Breed Smith (1873-1949) and her husband Reverend William Austin (also named) Smith (1872-1922). They had 5 children: Mary Estey (1905-1909), Austin (1907-1969), Anne Josephine (1910-1975), Miriam Fuller (1911-1913) and Janet Fuller (1914-1982).
Rev. Smith was a Harvard educated Episcopalian minister and editor of the Churchman Magazine. Upon Rev. Smith’s death, his children, Austin, Janet and Josephine, took the reins of the property. Austin, in particular, worked diligently to cultivate the land and orchards. Austin’s wife Eleanor Frothingham Smith and his sisters Janet Smith and Mrs. Anne Josephine Walsh are responsible for preserving the Felton-Smith Historic Site. They sold 200 acres of the land to the City of Peabody as open space, now Brooksby Farm. The remaining 16.5 acres and the historic buildings were given to the Peabody Historical Society and Museum, a private, non-profit organization, which operates separate from the City of Peabody.