Ray, I agree with Nacho's point of view in the sense that it would make the flow more intuitive.
So, as long as you have not entered the supplier invoice, you should be able to return with reference to that PO for the PO lines and quantities not yet invoiced, and the system should reduce the quantity to invoice. Quantity received - quantity returned = quantity to invoice. Because you, as a customer, do not acknowledge these quantities to be included in the supplier invoice.
In this same sense, if the supplier uses Odoo, he should be able to return the quantities delivered, provided that they have not been invoiced yet, reducing the quantity to invoice.
If the customer invoice have been posted and sent at the supplier side, then I agree that it is the obligation of the customer to accept the invoice. The initiating supplier invoice should then be posted with the original quantity received. Then the return should be part of a separate process, with separate refund invoice.
So, that is, adjustments in the quantity to invoice resulting from disputes between the supplier and customer, before invoices have been generated, should become part of the same transaction. But once the invoice has been sent, a new transaction flow must be initiated.