I wanted to restrict the partners (contacts) to a user based on partner tags (technical name of field is "category_id"). As the partner tag field is a many2many field, I'm not able to achieve the required output. An example of requirement is described below - 

Partner Tags - KA, TN, AP, NCR

Users - user 1, user 2, user 3, user 4

Requirement - A record rule which restricts a user to view only those contacts for which he has the respective tag access. For example - user 1 should be able to view only those contacts for which the tag is 'KA'.

I tried few record rules to achieve this requirement but wasn't able to do. Please help me out on this.