Stemphane,

Here is one solution if you want to hide particular field for specific group.

Follow these steps this might help you.

1) Take one boolean field  (Computational field)

     The method defined for this field will check the current user has your X(Test) group than it will set true to your boolean field.

2) Apply attrs on your field if that boolean is true than make it visible.

Hope this will help.

Rgds,

Anil.