Arthur L. Sweetser, a Boston banker, built Greystone his summer home on Labor in Vain Road in Ipswich. His wife was Laura Attwill.

The architectural style was apparently influenced by Henry Hobson Richardson (1838-1886), a prominent architect who designed nearly 80 buildings, including churches, libraries, railroad stations, and private homes, many of them in Massachusetts. His buildings often included massive towers and rugged stone walls, with blocks often laid in bands of contrasting color. The “Richardsonian Romanesque” style was widely copied.