There is no direct relationship between User (Assigned To) and Department.

You will have to add a department_id field in the user so that you can use if to filter the records.

Ex:

<field name="user_id" domain="[('company_id', '=', company_id), ('department_id', '=', department_id)]"

You can use the "domain" attribute. For example:

<field name="user_id" domain="[('login', 'ilike', 'test.com')]"/>

In the example above, only users with an email "test.com" will be available for selection in the many2one drop down.

You can return domain from onchange method, which solve your issue. 

@api.onchange('company_id', 'department_id') 
def onchange_m2o(self): 
    if self.company_id and self.department_id: 
        users = self.department_id.member_ids.mapped('user_id') 
        if users:
            return {'domain' : { 'user_id' [('id','in', users.ids)] }}


Make sure department must have employees. 

This method add domain to the Assigned to field at runtime based one the value you selected on the task.