Just from my memory, I believe their biggest concern was trust. For Democracy to work, the people have to have transparency of government. They talked about fiat-currencies making trust more difficult.

I asked ChatGPT for a few quotes to help back me up. I was a bit disappointed in the quantity and quality of them. Perhaps they weren't as outspoken about it is I'd thought. I read that Jefferson became louder about it much later, during the 1810's.

“The pestilent effects of paper money on the necessary confidence between man and man, on the necessary confidence in the public councils, on the industry and morals of the people…” — Federalist No. 44 (1788)

“Banks have done more injury to the religion, morality, tranquility, prosperity, and even wealth of the nation than they can have done or ever will do good.” — Letter to Thomas Jefferson, 1819