People may revert to hiding their cash "under the mattress" amid fears that their money is not safe if controversial plans enabling the taxman to seize money directly from accounts are implemented, the Building Societies Association (BSA) has warned.
The BSA is one of several bodies to raise concerns over proposals which would allow HM Revenue and Customs (HMRC) to gain access to money held in a debtor's bank or building society account without needing to apply to a court.
Others who have voiced warnings about the plans and their potential impact on public trust in the tax system include the British Bankers' Association (BBA), the Law Society and the Federation of Small Businesses (FSB).
In its response to a consultation on the proposals, the BSA warned that by by-passing the courts, HMRC would be acting as "judge, jury and executioner".
It raised fears about the impact of legal challenges and " reputational damage resulting from the wrongful application of these powers".
The BSA said: "The impact of aggrieved customers upsetting other customers and not to mention branch staff cannot be underestimated.
"It could lead to customers thinking their money is not safe and cashing out their savings, leading to a return to cash 'under the mattress'."
The BSA said it wants to work with the Treasury and HMRC "to find a way of achieving the Government's objectives to collect tax that is due whilst ensuring the appropriate safeguards for UK citizens".
HMRC has argued that its plans would only affect a "small core" of around 17,000 people a year who typically owe £5,800 in tax and tax credit debts. The cash would not be taken under the plans unless it still leaves the debtor with at least £5,000 across all their accounts.
The proposals would apply to current accounts, joint accounts and Isas. In cases where someone who owes nothing to the taxman but shares a joint account with someone who does, the taxman would recover debt from up to half the funds in the joint account.
People whose accounts would be targeted would have already ignored a string of correspondence asking them to make arrangements to pay the debt.
In the BBA's response to the consultation over the plans for the direct recovery of debts, Anthony Browne, chief executive of the BBA, has also raised concerns about the possibility of blunders.
Mr Browne said: "We are concerned about the lack of judicial oversight in the proposals and believe that the lack of checks and balances could create problems for citizens, particularly if HMRC makes mistakes.
"We believe that the current proposals could have a negative impact on the frail and vulnerable and could undermine public faith in the tax system."
The proposed powers would be aimed at taxpayers who owe more than £1,000 and the debt could be owed against just one tax or a range of them.
But the BBA has argued that the £1,000 threshold could mean that "a significant number" of people who are caught up in the rules are "unintentionally non-compliant, vulnerable taxpayers".
The BBA noted that the plans would mean that a debtor is contacted by HMRC at least four times before money is seized, and possibly up to nine times, including by letter and telephone.
But it said that HMRC should perhaps be putting more focus on the quality of how it gets in touch with consumers rather than on the number of times, in the light of anecdotal accounts of issues with HMRC's systems, record keeping and handling of information or payments.
Bodies representing consumers, businesses, banks and lawyers highlighted concerns over the plans in a letter recently published in The Sunday Times, which described the plans as " a power too far".
Its signatories included the Money Advice Trust, the Law Society, the FSB, Liberty, the BBA, the BSA and the Institute of Chartered Accountants in England and Wales.
HMRC launched a consultation on the proposals in May, with legislation expected to be taken forward as part of the 2015 Finance Bill.