Before the rise of textile mills, the fast-flowing water of the Jones Falls instead powered gristmills supplying Baltimore's lucrative flour trade. Whitehall Mill was established as a gristmill in the late 1700s and owned by James Ellicott, a member of the same family that settled Ellicott City. In 1839, David Carroll, Horatio Gambrill, and their associates purchased the mill from Ellicott and converted it to a textile mill for weaving cotton duck, a tightly woven canvas used to make ship sails.
Over the years, the mill was expanded, burned, rebuilt, renamed, and converted to a number of different commercial uses. To house their workers, Carroll and Gambrill built Clipper Village, a cluster of homes located across from Whitehall for the mill's workers. The capacity of the mill was doubled in 1845 and the mill was converted to steam power to keep up with manufacturing demand. By 1850, forty men and sixty-five women were working at Whitehall Mill with an output of 220,000 yards of cotton duck. Carroll and Gambrill quickly expanded by converting other gristmills along the Jones Falls to textile mills.
The three-story granite factory burned in 1854 and, after it was rebuilt, renamed Clipper Mill in recognition of the ships that used the cotton duck cloth for sails. By this point, William E. Hooper, a sailmaker who expanded his business to selling raw cotton to the textile mills, had joined as a partner. In the 1860s, Gambrill sold his shares in the company to Hooper and opened Druid Mill. After another fire in 1868, Clipper Mill was rebuilt at twice its size. The mill was sold in 1899 to the Mount Vernon-Woodberry Cotton Duck Company, a national conglomerate. In 1902, the mill manufactured the cotton duck for Kaiser Wilhelm's yacht, which was christened by Alice Roosevelt as the Meteor III. In addition to ship sails, the mill manufactured other heavy canvas items such as mail bags for the U.S. government.
In 1925, the mill was sold to Purity Paper Vessels, a firm that manufactured paper containers that could hold semi-liquid foods. The mill's cotton manufacturing machinery was shipped to Mount-Vernon-Woodberry Company's Southern mills in Tallassee, Alabama and Columbia, South Carolina. During the year of the sale, several elegiac articles appeared in the Baltimore Sun that looked back on the time when Baltimore's cotton duck manufacturing was at its peak and its clipper ships dominated international trade. Purity Paper Vessels later sub-leased part of the building to the Shapiro Waste Paper Company. In 1941, half the building was leased by the Army Quartermaster Office to be used as a warehouse for the Third Corps Area.
By the 1940s, the I. Sekine Brush Company, a maker of men's grooming products and toothbrushes, occupied the mill. The company was founded in 1906 and had been operating plants in Baltimore since 1928. After the attack on Pearl Harbor, H.H. Sekine, who had been living in the United States for over twenty years, was arrested and interrogated, along with dozens of foreign-born Baltimoreans connected to nations on the Axis side. At the time, Sekine was operating a factory in Reservoir Hill that the government shut down for two weeks. When it reopened shortly before Christmas, Sekine paid all his employees in full for the time they lost during the closure. Over time, portions of the Clipper Mill property were leased to other companies, including Penguin Books, The Maryland Venetian Blind Manufacturing Corporation, and Star Built Kitchen Units. Sekine maintained operations at the Whitehall mill location until 1992 when it was sold to Komar Industries.
Most recently, developer Terra Nova Ventures transformed the building into a mixed use development with a planned market. Architects Alexander Design Studio restored much of the long neglected mill, bringing new life to the historic structure. Numerous improvements were made for flood prevention, including the construction of a pedestrian bridge over Clipper Mill Road.