It is not perfect solution but work around to what you want to achieve:
Odoo gives us two classes

1. 'oe_edit_only' : to show element in edit mode only
2. 'oe_read_only' : to show element in read mode only

so I defined a duplicate field of my one2many field and show that in only read mode and original only in edit mode.
Note: we can not use same field in xml more than one time because odoo populates value only in lastly defined field. all previous fields with same name remain blank.
 

Try writting invisible="True" in the button xml line, i think it works

@Giovanny: I wanted to leave comment but this website is not allowing me to do so. but with your solution button will be hidden permanently. I need to hide only in Non-Editable mode.
 

You can pass the class to the template by finding the button template "ListView.row.button" in addons/web/static/src/xml/base.xml. Just add "#{widget.class}" to the button t-attf-class attribute.