The return value of your button function should be a window action with target attribute 'new'.

XML:

<button name="your_button_function", string="My Button", type="object"/>

or

 <button name="your_xml_act_window_name", string="My Button"/>

If your button triggers a python function (the 1st case):

def your_button_function:
    return {
        'name': 'My Window',
        'domain': [],
        'res_model': 'my.model',
        'type': 'ir.actions.act_window',
        'view_mode': 'form',
        'view_type': 'form',
        'context': {},
        'target': 'new',
    }

Otherwise: Window Action