This seems to be hardcoded in the graph_view.js.. But one way you can hack around this is to use 
def fields_get override in python file.

If the fields are not needed in any of Measures, GroupBy, Export, AddCustomFilter, then you can use the allFields parameter of the super to control the list. To control individually, you can use different attributes of each field. Example (V 13.0):

@api.model
def fields_get(self, allfields=None, attributes=None):
   res = super(ResPartner, self).fields_get(allfields, attributes=attributes)
   not_selectable_fields = set(res.keys()) - set(self.selectable_fields)
   for field in not_selectable_fields:
      res[field]['selectable'] = False # to hide in Add Custom filter view
      res[field]['sortable'] = False # to hide in group by view
      res[field]['exportable'] = False # to hide in export list
      res[field]['store'] = False # to hide in 'Select Columns' filter in tree views
   return res