Option 1: Define two graph views with different fields

Create two graph views (with priority) and load one based on context:
<record id="view_graph_with_field" model="ir.ui.view">

  <field name="name">model.graph.with_field</field>

  <field name="model">your.model</field>

  <field name="priority">20</field>

  <field name="arch" type="xml">

    <graph string="Report">

      <field name="date" type="row"/>

      <field name="amount" type="measure"/>

      <field name="some_field" type="measure"/>

    </graph>

  </field>

</record>

<record id="view_graph_without_field" model="ir.ui.view">

  <field name="name">model.graph.no_field</field>

  <field name="model">your.model</field>

  <field name="priority">10</field>

  <field name="arch" type="xml">

    <graph string="Report">

      <field name="date" type="row"/>

      <field name="amount" type="measure"/>

    </graph>

  </field>

</record>

Option 2: Use groups attribute

If visibility should depend on user group, this works:
<field name="some_field" groups="base.group_user" />