Clean up filestore

We're trying to clean up our filestore since its gradually growing and way beyond 100GB right now.
We are removing the avatar from the res.partner record, then observe that the 'image' related records are deleted automatically from the ir_attachment table.

However the files in the filestore related to the ir_attachment that is deleted remain in the filestore.
Do they ever get deleted, or are permanent?

The "Base: Auto-vacuum internal data" doesn't seem to do the job.