And even if I change the Header template - the clones of the button persist and I can't delete them too!

In other threads, I found out that there should be a "Delete" button like this:

But in my case - it is missing for some reason! Here is my interface:

And Odoo support helped me with resolving this issue, sharing this with community:

To delete the multiple "Contact Us" buttons displayed on your website, please follow these steps:

    Open "Settings" > Scroll down and click on "Activate the Developer mode"

    Open "Settings" again > Click on "Technical" from the menubar > Scroll down and click on "Views"

    Search View for "header call to action" > Open the "Header Call to Action" view that does not have an External ID

    Click on the gear icon > Click on "Compare/Reset" > Select "Reset to file version (hard reset)" > Click "Reset View"

This should resolve the duplicate Contact Us buttons and only one will be left.

Please refer to this video: https://drive.google.com/file/d/1-h_piUAUR9eKEbGaeUunsU921jNKipdf/view