It's probably small, but there would be some trust lost in America's ability to maintain the dollar if it becomes known that there's less gold in reserve than claimed.

Otherwise, I totally agree.