Potentially out-dated:
Since you're dealing with a Sale Order in state 'Sale' (confirmed), you can not remove lines once added. This is regardless of the way how a product was added (be it directly via the Sale Order form, or from the Project Task Catalog). You can only set the Quantity of that Product in the Sale Order to 0 at that point (or reset the Sale Order to 'draft', if that works for you and reconfirm it once the line has been deleted).
You can't remove a Product (quantity) via Catalog once it has been delivered. The product is delivered, thus should be invoiced (unless the Field Service is set as 'Under Warranty'). Now, simply removing the Product via the Catalog would just completely corrupt your inventory count:

You probably want to perform a Return action on the delivery, to first of all get it back from your customer - so your on hand quantity is correct again.
Once this is done, you can manually set Quantity in the Sale Order to 0. This should eventually remove the Product form the Task's Catalog.
Hello, thank you for your quick answer.
My problem is that when adding a product, the sales order is automaticaly set in order, and not in quote or concept. Is there a way to prevent the order to be confirmed directly?
No, not in Odoo standard. Projects can be linked to confirmed Sale Orders only by default (thus you would either need to modify that logic or set up an automation to reset confirmed Sales to draft, once linked to a project for example). Note that only confirmed Sale Orders can be invoiced on the other hand.