I have further narrowed down the issue and found a partial solution to create truly amount less offers but still paying them fails, I opened a github issue for the LNDK repo (responsible for offer handling that I had forked and integrated into my app, hopefully they will also help resolving this issue.