James Schoolcraft Sherman was born in Utica, New York, on October 24, 1855, to Richard Updike Sherman and Mary Frances Schoolcraft Sherman. His father owned a canning factory and was active in Democratic politics. Sherman attended Hamilton College and graduated in 1878 after distinguishing himself in oratory and debate. He studied law there as well and received his law degree the year after his graduation. In 1880, he was admitted to the New York bar, and he married Carrie Babcock in 1881.

Breaking with his father, Sherman became a Republican. His father was displeased, but his displeasure was mitigated when Sherman was elected mayor of Utica at the age of twenty-nine. Two years after his election as mayor, in 1886, he won election the U.S. House of Representatives. Aside from two years after an unsuccessful reelection attempt in 1890, Sherman would remain in public office for the rest of his life.

During his twenty-year career in the House, Sherman chaired the Indian Affairs Committee and was a member of the Committee on Interstate and Foreign Commerce. Although he never chaired a major committee or held a position of party leadership, Sherman held significant power in the House. He worked closely with Speakers Thomas Reed and Joe Cannon during his tenure and was trusted to hold the gavel in their absence. He was noted for his parliamentary knowledge and as a reliable ally of the "Old Guard" conservatives who resisted the progressivism of more liberal members of the Republican Party, namely Theodore Roosevelt.

In 1908, Republican presidential candidate William Taft selected Sherman as his running mate for the presidential election. Sherman was chosen partially as a compromise pick to appease conservative Republicans and with hopes he would help Taft carry New York in the election. The Taft-Sherman ticket won in a landslide over the Democratic ticket of William Jennings Bryan and James Kern.

Shortly after the election, Taft requested that Sherman serve as his liaison between the White House and congressional leaders, particularly Speaker Cannon. Sherman testily replied that he would do no such thing, and that "acting as messenger boy was not part of the duties as Vice President." Taft began meeting Speaker Cannon on a regular basis in the wake of Sherman's refusal and began a drift towards the conservative wing of his party that would come to divide the Republicans and ultimately cost Taft reelection.

Sherman was an unwavering conservative and did not seek compromise with the progressives. In fact, he advocated punitive actions against them for disloyalty and entered in a contentious battle with Roosevelt to head the nomination convention for the party's nominee for governor of New York. When Roosevelt defeated Sherman, and thereby secured the power to select the party's nominee, it was widely perceived as a political defeat for the Taft administration.

The growing divide between the conservative and progressive elements of the Republican Party culminated in 1912, when Roosevelt announced his intention to pursue the Republican nomination for President. When he failed to wrest it from Taft, he announced his attention to run as a third-party candidate. Sherman was renominated as Taft's running mate, although Roosevelt's entry into the election essentially ruined their chance at victory.

Sherman had suffered from Bright's disease, a serious kidney illness, for years and was too sick to campaign. On October 30, 1912, just days before the election, he died. His name remained on the ticket as Taft was decidedly beaten, receiving just eight electoral votes in an election that split the Republican Party vote and gave Democratic candidate Woodrow Wilson an overwhelming victory.