The house at 22 Elm Street sits behind other houses on the street, its foundation resting in the Ipswich River. The Ipswich assessors site gives the date of construction as 1840. The 1856 Ipswich map shows this building as part of a grist mill complex. Several mills operated at the County St. Bridge in the mid-19th Century, and this building may have originated as worker housing or part of the grist mill.
Thomas Franklin Waters wrote that in 1832 the lower grist mills and land on Elm Street were purchased by the Ipswich Manufacturing Company with the intention of building a canal from the river above the upper dam, across the Heard estate to the lower river, but the project was never enacted. In the 1884 map the owner is “Mr. and Mrs. Parsons.”