Thank you for your answer, but it just explains how to add a custom report, ie. for products. In my case I want to override the invoice report and for that I need direct instructions about how to register the record (from my understanding), like
<record id="custom_invoice_layout" model="ir.actions.report">...
or how to register my qweb template file
<template id="invoice_layout_custom">...

from your suggestion it would be something like

<record id="custom_invoice_layout" model="ir.actions.report">
<field name="name">Custom Invoice Layout</field>
<field name="model">account.move</field>
<field name="report_type">qweb-pdf</field>
<field name="report_name">custom_odoo_templates.invoice_layout_custom</field>
<field name="report_file">custom_odoo_templates.invoice_layout_custom</field>
<field name="report_file">account.report_invoice</field>
<!-- <field name="print_report_name">'Invoice_%s' % (object.name)</field> -->
<field name="binding_model_id" ref="account.model_account_move"/>
<field name="binding_type">report</field>
</record>

if I use that, I get the error
odoo.addons.base.models.ir_qweb.QWebException: Error while render the template
KeyError: 'doc'
Template: custom_odoo_templates.quote_layout_custom

so for me it looks like the invoice report is being rendered using the (overridden) quote (sale.order) template.

I now deleted the <record> and just work with
<template id="invoice_layout_custom" inherit_id="account.report_invoice_with_payments">
I also added t-if="doc" everywhere in the <template id="quote_layout_custom"> tags and it now looks like the rendering reaches the default invoice templat as I now get the KeyError
KeyError: 'o'
Template: account.report_invoice_with_payments
Questions arise like: Why does it renders the sale order template before? And why is there no o-Object any more?

If you need to create a new invoice report try this <record id="action_invoice_report" model="ir.actions.report">
<field name="name">Invoice Report</field>
<field name="model">account.move</field>
<field name="report_type">qweb-pdf</field>
<field name="report_name">invoice_report.report_invoice_template</field>
<field name="report_file">invoice_report.report_invoice_template</field>
<field name="print_report_name">'Report - %s' % (object.name)</field>
<field name="binding_model_id" ref="account.model_account_move"/>
<field name="binding_type">report</field>
</record>

otherwise, if you need to inherit the default invoice report and make changes use inherit_id and call this template account.report_invoice_document
<template id="report_invoice_document" inherit_id="account.report_invoice_document">
inside you can add the updates
</template>

trying to inherit the default invoice report with

<record id="custom_invoice_layout" model="ir.actions.report">
<field name="name">Custom Invoice Layout</field>
<field name="model">account.move</field>
<field name="inherit_id">custom_odoo_templates.invoice_layout_custom</field>
</record>

I get the error

ValueError: Invalid field 'inherit_id' on model 'ir.actions.report'

inherit_id is in the report template, not in ir.actions_report model