Treasury Secretary Timothy F. Geithner is catching heat from both parties on Capitol Hill after a stinging report from the top auditor of the $700 billion Troubled Asset Relief Program — so much so that some members are calling for his resignation.

Rep. Kevin Brady , R-Texas, told Geithner during a Joint Economic Committee hearing Thursday that “the public has lost all confidence in your ability to do your job. It really is time for a fresh start and I would urge you to consider that.”

Geithner wasted little time swiping back at Brady, telling him he “would not take responsibility” for the policies of the previous Republican administration that he said led to the economic collapse of 2008 and the subsequent bailout of Wall Street firms.

“If you look at any measure of confidence in the financial system, it is substantially higher today than when the president of the United States took office,” Geithner added, referring to President Obama.

But Geithner is facing a growing wave of criticism on Capitol Hill in the wake of a report produced by the Special Inspector General for TARP that pinned the blame on Geithner for taxpayers covering the losses of Goldman Sachs, Merrill Lynch and other “counterparties” who engaged in high-risk transactions with failing insurance giant American International Group Inc.

As the president of the Federal Reserve Bank of New York at the time, Geithner was the point man in the AIG rescue, which resulted from the collapse in the value of complex credit default swaps in which the company traded.

Lawmakers have jumped on the inspector general’s report in recent days, calling for investigations of the AIG payouts.

Two senior House Republicans — Rep. Spencer Bachus , the ranking Republican on the House Financial Services Committee, and Rep. Roy Blunt , the lead GOP negotiator during the creation of the TARP in 2008 — circulated a letter Thursday demanding a hearing on the topic.

Rep. Peter DeFazio, D-Ore., said Wednesday night in an interview on MSNBC that he thought Geithner should resign. He also indicated that he and other members of the Congressional Progressive Caucus were considering their options for pursuing his resignation.