Personally, I think that in an ideal Libertarian society the FDIC and the Federal Reserve would be abolished. This would result in fewer loans and as a result less debt. While the FDIC is a good idea, I'm positive that deposits could be insured more efficiently in the private sector than in the public sector, as is almost always the case.

The FED is a whole other story. For a great explanation of why the FED is highly unproductive and generally a hazard to the economy, check out this video: http://www.03808.com/meltdown-thomas-e-woods-jr ; The video is about an hour long but extremely logical and informative. If you don't care to watch it, I'll give you a short summary.

Thomas Woods likens the economy to a builder, while savings which can be loaned are the bricks which the house is made of. By artificially setting interest rates, the Fed makes the builder think he has more bricks than actually exist resulting in a house being built only part of the way before the builder (economy) realizes that it physically can't build a house that big. This would result in a "bubble" where the whole house must be torn down and the builder would have to go back to the drawing board.

I also believe that normally fiscally conservative Libertarians probably have less debt than the average American, as a matter of principle.

As far as banks go, they would operate in a similar capacity but would give out far fewer loans, which would keep their customers' finances more secure.