Although Baker is said to have built the present house, the structure is typical of houses built before 1720, which suggests he may remodeled the home of the Rev. John Rogers.
The oldest part of the house dates to 1677 when the military officer and entrepreneur Captain John Whipple constructed a town house on Saltonstall St. near the center of Ipswich