Misusing the sitemap implementation was a bad idea and broke for every major update. I now finally migrate to custom implementation which should remove the issues for future.