For this you can use a record rule for example in my case i want my users to see only those records in which they are manager so i simply use

[('manager_id.user_ids','=',user.id)]
manager_id is my custom field in the view, you can use your custom field which you use for assigning users to each record. Also you can create a group for which this record rule may only apply.

Hope it helps.