The exact location of the Ipswich Custom House has never been verified. Shown above is a photograph of the Town Wharf area taken by Arthur Wesley Dow, sometime between 1905 and 1910. Alice Keenan wrote that the small building on the right had been the customs house. Later photos show a sign attached to the front that reads, “P.R. Eames, Wood, Coal, Lime and Cement.”

Two buildings that once stood on Water Street were moved to Brown Square. The small Stellene Oil Company office is the building shown in the photo above. William J. Barton wrote that the former grain building to the right of Tedfords had once been the custom house and was later used by William G. Brown as a lumber barn before being moved to Brown Square.
In 1789, Congress established the United States Custom Service as a branch of the Treasury Department. A minor port between two major points of entry at Newburyport and Gloucester, Ipswich became a port of delivery under the Salem/Beverly Customs district. Seven years later, Ipswich was established as an independent customs district. The Surveyor position was abolished and Asa Andrews was installed in the new role of Collector. In 1803 there were 23 vessels, weighing a total of 1,362 tons, registered at the Ipswich Custom House.
In 1829, Andrews was succeeded by Timothy Souther, “a man of prominence and influence in his native town of Ipswich and one of the old line Democrats who held office there under President Andrew Jackson.” Souther resigned in August, 1840 after being charged with graft. He was replaced by Asabel Wildes. Ipswich was annexed to the Newburyport Customs District in July, 1844. Wildes remained in Ipswich as Surveyor following the annexation, with Daniel L. Wilcomb as Inspector during this period. In 1861 Reuben Daniels was appointed surveyor of the port of Ipswich.
Customs collection ended in Ipswich and the Newburyport District in 1910. Susan Howard Boice wrote that the records of the Ipswich customs house were sent to Newburyport and were subsequently lost.
Timothy Souther
The first person in Ipswich by the name of Timothy Souther arrived with his wife in 1763. The family was impoverished, and was “warned out,” He died at age 27. Timothy Souther, possibly the son of the former, bought half a house on South Main Street, near the Choate Bridge and died in the West Indies at 36 years of age in 1804. By then had sold “half of the half” he owned, but his wife Elizabeth Badger Souther continued living in the northwest corner of the house until her death on December 31, 1841 in Ipswich at age 74. The old Souther house was removed when the Dr. Joseph Manning house was moved to that location.
Timothy Souther, son of Timothy and Elizabeth, was born in Ipswich on April 7, 1800. He became involved in the town’s Democratic party, and in 1829 was appointed collector of customs for the district and inspector of the revenue for the port of Ipswich at the old Custom House. Souther purchased a lot from the executors of Moses Treadwell’s estate on North Main Street on January 1, 1935. He mortgaged to Otis Holmes, who foreclosed the mortgage on March 1, 1842. Accusations of graft became public knowledge at about that time, and Souther moved with his family of five sons and two daughters to Alton, Illinois, where he was employed as postmaster from 1846 to 1854. He died on Jun. 20, 1871, and is buried in Alton.
Accusations of Patronage and Fraud
(Thanks to Melissa Berry for finding this opinion article, dated August 1840)
The Salem Gazette printed an extract from the speech of Otis P. Lord, Esq. of Ipswich, delivered at Salem on the 4th of July, 1840, in which Mr. Lord touched upon the abuses practiced by the Locofoco office-holders all over the country, and stated that public money was being converted to improper and fraudulent uses in the conduct of affairs in the Ipswich Custom House, a charge for which he claimed to have full and satisfactory proof:
“Our little town, although situated as it were in the bush, is nevertheless a Port of Entry, and it is necessary that some person should officiate as Collector, and see that the business of the Custom House (should there be any business is properly attended to…The whole of the imports since the election of Gen. Jackson to the presidency have been but about twelve or fifteen hogsheads of molasses, and for several years past, no such thing as a foreign entry has been known. In 1837, the whole amount of revenue received at the Ipswich Custom House was about twelve dollars…Yet to collect this sum for the United States, three or four officers have been employed, whose salaries amount in the aggregate, to about twelve hundred dollars a year!”
“Some time ago, a friend of the Collector Mr. Souther obtained a loan at the Bank, of four hundred dollars, for which Mr. Souther was surety. The principal could not pay the note at its maturity, and of course the endorser was held. And what did the Collector do? Why, he appointed the delinquent as Inspector in the Custom House, at a salary of $400 a year, with the express understanding that the salary should be paid over to cover the amount of the note at the Bank. The man was not wanted in the office; there was nothing for him to do, but the Collector did not want to lose the money, and the United States treasury was able to pay it.
The Collector built himself a house in Ipswich, and how did he build it? He employed a man to work about the building, digging the cellar, etc. at $11.00 a month. When he engaged the man to dig the cellar, the Collector told him he was about to appoint him a Boatman. The man was astonished at the idea—he did not know anything about being a boatman, or what boating the Collector had to do. He was appointed, but never had occasion to reef or steer at all, as there was no business wherein to employ boat or boatman, except in one or two instances he did go down the river with parties of pleasure! He went on and dug the cellar, and quarterly was induced by the Collector to sign receipts for fifteen dollars a month as boatman, receiving at the same time, only eleven dollars, the price of his digging per month.”
On April 15, 1845, Otis Holmes sold Timothy Souther’s former property at 16 North Main Street to Stephen Coburn, who built or finished the house still standing. Souther moved to Alton, Illinois, and was hired as postmaster for that city. Coburn was postmaster of Ipswich from 1823 to 1860 and the owner of Coburn’s Block which stood on the site of Dr. John Manning’s failed factory until it was destroyed by fire and was replaced by the Caldwell Building. After the death of the Coburns, the house became the Lucy Coburn home for the Elderly, a benevolent institution. It is now the Kaede Bed and Breakfast.
Sources:
- Newspaper clipping, circa 1838, provided by Melissa Berry
- Peabody Essex Museum: Ipswich Customs House
- History of Essex County
- Ipswich in the Massachusetts Bay Colony
- Over Three Hundred & Fifty Years of Ipswich History by Elizabeth H. Newton, Alice Keenan and Mary P. Conley
Hello! These are wonderful posts about. Ipswich. Do you have any information about the Robert Brown of Ipswich who emigrated back to England in 1645 to fight in the New Model army?
Presumably he attended the Meeting House currently occupied by First Church? Thanks for these posts!
I don’t have information about Robert Brown, but Edward Brown came to Ipswich in 1642, and his house still stands. https://storiesfromipswich.org/edward-brown-house-27-high-street/