You're hitting on it pretty well. Your work has value to you, because there are other things you'd rather be doing. That means the product of your time needs to be valuable enough for your employer/customer to be willing to pay for it.

The easiest way to see the flaw in thinking work itself adds value is to just imagine how much you'd still get paid if technology made it possible to produce the same goods from the same inputs without your work.