One way is to use an Automated Action that is run during create and/or update of a Transfer:

Note:  This snippet assumes you have two System Parameters setup:

1) rma.allowable_category = the ID of the product category users ARE allowed to add to a Return

2) rma.admin_group = the External ID of the Group that IS allowed to add ANY items to a Return 

allowable_category = env['ir.config_parameter'].sudo().get_param('rma.allowable_category')
admin_group = env['ir.config_parameter'].sudo().get_param('rma.admin_group')

for records in records:
  if not record.user_has_groups('admin_group'):
    for move in record.move_ids_without_package:
      if move.product_id.categ_id.id != allowable_category:
        raise Warning("You can only add USED products to a Return")

Hi, 

What i would suggest is that you add a context to your M2O product (context is_rma: parent.rma_field) and then Override _search function. 

Inside that function, check your context if it has the rma information. If it does, add to args ('categ_id', '=', REF_USED.id)

Hope this helps. If it does, you can upvote.

Regards.