I solved the problem. In case any one face the same issue, here shares my solution. By default, in the file crm_security.xml, there is one rule controlling the crm phone call access.

<record id="crm_rule_all_phones" model="ir.rule">
    <field name="name">All Phones</field>
    <field ref="model_crm_phonecall" name="model_id"/>
    <field name="domain_force">[(1,'=',1)]</field>
    <field name="groups" eval="[(4, ref('base.group_sale_salesman_all_leads'))]"/>
</record>

It allows the phone call records to be viewed by all the users no matter he is in this company or not. So in order to fix the issue, you could change the rule to the following:

<record id="crm_rule_all_phones" model="ir.rule">
    <field name="name">All Phones</field>
    <field ref="model_crm_phonecall" name="model_id"/>
    <field name="domain_force">['|',('company_id','=',False),('company_id','child_of',[user.company_id.id])]</field>
</record>

That will protect the records to be viewed by the members out of this company.

Regards Sage