You need to change your button type to "action" , like its done in invoice form when opening a refund form:

 <button name="%(action_account_invoice_refund)d" type='action' string='Ask Refund' states='open,paid' groups="account.group_account_invoice"/>

and define the action in xml:

<record id="action_account_invoice_refund" model="ir.actions.act_window">
            <field name="name">Refund Invoice</field>
            <field name="res_model">account.invoice.refund</field>
            <field name="view_type">form</field>
            <field name="view_mode">tree,form</field>
           <field name="view_id" ref="view_account_invoice_refund"/>
           <field name="target">new</field>
</record>

You can check the code example at addons/account/account_invoice_view.xml and addons/account/wizard/account_invoice_refund_view.xml