The James Brown house at 50 North Main Street in Ipswich was built in 1720 (1st / 2nd Period). This 3/4 acre property had two houses and came to be owned by two families at the same until a petition to the town to divide the property was accepted. The long house on the property adjacent to the James Brown house was sold to Thomas Morley and to James Damon. Morley cut off his end of the house, turned it end to the street and made it into a separate dwelling at 48 North Main. Mr. Damon took down the remainder of the old dwelling and built the fine house at 46 North Main. Thus one lot became three, explaining why the houses sit close together.

In Tales of Olde Ipswich, Volume III, Harold Bowen relates that Albert Hills had his store in this building, and that it was the home of long-time selectman Brainard Wallace, father of David Wallace who has the popular Facebook group, I Love Ipswich MA