“but how many of those transactions are going to contain same address”

One of many reasons why you don't reuse addresses...

If you aren't reusing addresses, this is a total non issue. And reusing the same server over and over again is not best practice anyway (unless you control it).