The house at 3 Mineral Street was constructed between publications of the 1856 and 1872 Ipswich maps. According to the present owners, the deed to the house states that it was constructed in 1858. The earliest owner to be identified is Charles H. Baker who owned the house in the 1884 map. Baker is listed in town directories as a “flagman,” probably for the local branch of the Boston & Maine Railroad, which runs next to the house.