group by can use a callable or a field to group, so you can create a computed field to get the domain and then use the computed field to group.

something like this I suppose but I didn't test it

# Python
    email_domain = fields.Char(compute='_compute_email_domain')

    @api.depends('email')
    def _compute_email_domain(self):
        for record in self:
            record.email_domain = record.email.split('@')[1] if '@' in record.email else None
# XML

https://github.com/odoo/odoo/blob/17.0/odoo/models.py#L6106