So I kinda solved my issue, but not on 100%. Here is a piece of code of "base.view_partner_form" that you can get by clicking on "Edit FormView" in the developer mode:

<form string="Contact / Address">                                    
    <sheet>                                        
        <field name="type" required="1" widget="radio" options="{'horizontal': true}"/><hr/>                                        
        <group>                                            
            <group attrs="{'invisible': [('type','=', 'contact')]}">                                                
                <label for="street" string="Address"/>                                                
                    <div>                                                    
                        <div class="o_address_format" name="div_address">                                                        
                            <field name="street" placeholder="Street..." class="o_address_street"/>                                                        
                            <field name="street2" placeholder="Street 2..." class="o_address_street"/>                                                        
                            <field name="city" placeholder="City" class="o_address_city"/>                                                        
                            <field name="state_id" class="o_address_state" placeholder="State" options="{&quot;no_open&quot;: True}" context="{'country_id': country_id, 'zip': zip}"/>                                                        
                            <field name="zip" placeholder="ZIP" class="o_address_zip"/>                                                        
                            <field name="country_id" placeholder="Country" class="o_address_country" options="{&quot;no_open&quot;: True, &quot;no_create&quot;: True}"/>                                                    
                        </div>                                                
                    </div>                                            
            </group>

and here is what i did to add my custom field "street3" after "street2":

<xpath expr="//div[@name='div_address']/field[@name='street2']" position="after">             		
    <field name="street3" placeholder="Street 3..."/>             	
</xpath>

But here is what I noticed. If you add "street3" field to the main form and the same field to the form that appears after you click the button in "Contacts" tab then you'll see that they're not linked. What i mean is if you type in "street" and "street2" fields in the main form then you'll see the same input in the same fields on the appeared form after you click a button, but it's not like that when you add your own custom fields.

I think the same field needs to be added somewhere else to make it happen.