The mineral spring that gave Bath its name was near the intersection of Tracy St. and Forbes Ave. today. The 1876 Beers Map shows "Mineral Street" there, named for the mineral spring. The story is that in 1790 Elkanah Watson proposed the idea of the Bath to Patroon Stephen Van Rensselaer III. (A year later, Watson proposed the Erie Canal, which was a bit more successful than the bath house.)
Watson had traveled to the spas at Bath in England, and was "filled with delight" at their magnificence. In comparison, he found the scene at Saratoga in 1790 to be "enveloped in rudeness and seclusion, with no accommodations appropriate to civilized man".
The American Universal Geography, by Jedidiah Morse, 1793
In the new town of Rensselaer, nearly opposite the city of Albany, a medicinal spring has lately been discovered, combining most of the valuable properties of the celebrated waters of Saratoga. Should further experiments confirm the favorable opinion already entertained of this spring, it will prove a fortunate discovery for the city of Albany and for the country adjoining, as well as for the invalids who annually resort to Saratoga, under many inconveniences and at great expense.The American Gazetteer, by Jedidiah Morse, 1797
Bath, a village in the co. of Rensselaer, New-York, pleasantly situated on the east bank of Hudson river,... A mineral spring has been discovered here, said to possess valuable qualities; and a commodious bathing-house has been erected, at a considerable expense, containing hot, cold, and shower baths.History of the Towns of Rensselaer County:, A. J. Weise, 1880
Before the close of the last century the wonderful efficacy of the waters of several mineral springs at Bath had made the little hamlet somewhat conspicuous as a place of cure. John Maude, an English traveler, visiting places of note in the United States, in 1800, thus speaks of its notoriety, in his journal... "Crossed the river to Bath, a town lately laid out by the patroon; it at present consists of about thirty houses... The medicinal springs and the baths, at one time so much vaunted, are now shut up and neglected ; yet, as a watering place, it was to have rivaled Ballstown..."The bath house built on the Hudson was very innovative for the 1790's - it had plumbing, unlike the other mineral springs in the region (in Ballston, you sat in a wooden barrel sunk into the mud). I'm not sure why the bath house was already abandoned by 1800. Three Villages speculates the Patroon's lease terms may have dampened the commercial potential of the enterprise.