Correct, the (Internal) Reference field (or default_code as it is called in the database) is meant to be an unique identifier on a Product.

Now, here 'Product' can mean two things:

• a Product (Template) (technical name product.template) without Variants
• a (Product) Variant (technical name product.product) itself

It's important to note that every Product Template has at least on Product Variant. In other words, a product.template record is either directly linked to a single product.product record (=> a Product without Variants) or to multiple product.product records (=> a Product with Variants); In return, each Product Variant is definitely linked to one Product Template.

In the light of calling '(Internal) Reference' a 'SKU' this makes actually sense, since you don't keep track of the stock of what all a product could be (Product Template with multiple Variants) but rather what it actually is (a Product Template representing just one Variant or the different Variants of a Product Template).

From a more technical point of view, the behavior of what a Product Template's Reference is, is defined here: https://github.com/odoo/odoo/blob/18.0/addons/product/models/product_template.py#L423-L428
This section basically says that when there's just a single Variant, the Variant's Reference is applicable to the Template as well, otherwise it isn't (this then is combined with some view logic of whether or not to show the field).
This is also true for other fields, such as for example the weight of a Product - again, here you would be interested in the physical item rather than the 'definition container' or 'group' (aka Product Template) when dealing with different Variants.