Dear Mathieu,

Yes, You can do it.
First create your form view. Then inherit the view also specify the groups.
For example in sale order form view, i want to make the customer reference field invisible for group user.

<record id="view_order_form_cust_ref_readonly" model="ir.ui.view">
    <field name="name">sale.order.form.readonly.cust</field>
    <field name="model">sale.order</field>
    <field name="inherit_id" ref="sale.view_order_form"/>
    <field name="groups_id" eval="[(6, 0, [ref('base.group_user') ])]"/>
    <field name="arch" type="xml">
        <field name='client_order_ref'" position="attributes">
            <attribute name="invisible">1</attribute>
        </field>
    </field>
</record>

Hope above code help for you.

Best Regards,

Ankit H Gandhi.