Domain or Context in the Windows Action will "lock in" the selection of records.  If necessary, create different menu options for different groups of users, each with different domain or context.

https://odootricks.tips/windows-actions/#filters

Add 'groups' in that filter. In group, add relevant res.group record. And from group add users whatever you want to add. 

I think you have to pass the domain on action instead of set a default filter.
when you pass the domain in action it only displays record which satisfied the given domain.

ref. https://github.com/odoo/odoo/blob/11.0/addons/project/views/project_views.xml#L663-L664