The house at 3 Mineral Street was constructed between publications of the 1856 and 1872 Ipswich maps. The earliest owner to be identified is Charles H. Baker who owned the house by 1884. Baker is listed is town directories as a “flagman”, possibly for the local branch of the Boston & Maine Railroad.