Hmm that was actually easier than i first thought. Seems like there are 2 places where a breadcrumb is generated and they are not in sync ...

This fixes the breadcrumb behavior for the product page:

    <template id="product" inherit_id="website_sale.product">
        <xpath expr="//li[@t-if='category']" position="replace">
            <t t-if="category and category.parents_and_self">
                <t t-foreach="category.parents_and_self" t-as="cat">
                    <li class="breadcrumb-item">
                        <a t-att-href="keep('/shop/category/%s' % slug(cat), category=0)" t-field="cat.name"/>
                    </li>
                </t>
            </t>
        </xpath>
    </template>