As this is not an out-of-the-box feature you might need to reproduce these steps when a database update is applied, but still works without customizations or development this is a standard solution*

To limit the access of certain users to the contacts we need 3 configurations 

1. Contact tag for internal users
2. Group to define the users for this restriction
3. Record rule with the restriction

We need the developer mode for this configuration

Steps to reproduce… 

Contact tag for internal users

This label would help to maintain the visibility among our users contacts, as many functionalities use the contact related to the user. (example: the chatter)

At the contacts application, create a tag (category_id) with the name “INTERNAL”; you can name your tag as you wish but this is a recommendation. 

Assign this tag to all the contacts related to your internal users and companies.

Once the tag is created, get the tag ID.  Still into the contacts application, at the configurations menu go into the “Contact tags” model, select the tag “INTERNAL” and go to the developers tool menu to the View metadata option and see the ID of the record

We will use this value later. 

Group to define the users for this restriction

At the settings in the Users and Companies menu, into the groups model. Create a new group, in this group will be all the users that the restriction applies.   

Under the Users tab, add the users that will be restricted with this configuration. 

Create the group just with the name and add the users that will be restricted at the users tab

The admin should NOT be included in this group.

Record rule with the restriction

At the settings in the Technical menu, into the Record rules model. We need to create a new rule with the following configuration: 

• Name: Contacts: Only contacts assigned
• Model: Contact (res.partner)
• Active: True
• Rule definition (Domain filter): ['|', ('user_id', '=', user.id), ('category_id', '=', 9)] (The number "9" is the ID that ge got from the tag at the first step)
• Group name: Settings / Restrictions for users

The domain at this rule is filtering the contacts this way: 
Contacts where the user (user.id) equals the salesperson (user_id) specified at the contact 

OR
Contacts with the tag (category_id) with the ID 9, (The value that we got from the “INTERNAL” tag).

With this they can only see their contacts assigned and also the internal users to keep the communication among them.

Additional notes: 

- For the administrators you don’t need to do anything, as this rule only applies to the users specified in the group.
- If a contact is not assigned to any salesperson, only the administrators will be able to see this contact. 
- This rule applies to specific contacts, it does not affect partner or child contacts, is necessary to assign each contact to the salesperson. 
- The users that will create a new contact can assign themselves to the salesperson field during the contact creation. 

- When creating new users The admin won't have any problems,  as this only applies to the users at the Group defined, just add the tag "INTERNAL" to the contact related once the user is created. 

Hope this is useful for you.
Greetings