You need more than 1 step.

First I imported the attributes with the corresponding values

This csv would import an attribute 'color' with 2 colors and there hex code. Name and type only in the first row.

name,value_ids/name,value_ids/color,type

color,red,#123456,Color

,blue,#123457,

Now you can create a product with variants

name,product_variant_ids/attribute_line_ids/attribute_id,product_variant_ids/attribute_line_ids/value_ids

"Tshirt","color","red,blue"

If you want to add for example an EAN to the variants you would have to export the variants now, add the EAN and reimport.