you can revoke delete access right from the user to prevent cancel activity 
Settings / Technical / Security / Access Rights
search mail.activity model and uncheck uncheck boolean from types / Internal User

Note: it will revoke access from all the internal users, if you want to provide cancel access to some users then create a new user group and provide full access  

Hi,

1. To remove cancel button in the chatter of activity, Please add the following code static/src/xml directory,

<?xml version="1.0" encoding="UTF-8"?>
<templates id="template" xml:space="preserve">

    <t t-inherit="mail.Activity" t-inherit-mode="extension">
        <xpath expr="//i[hasclass('fa fa-times)]"" position="replace">
         
        </xpath>
    </t>

</templates>

2. Give XML path in the manifest

Hope it helps,

Remove button is way more easier, just inherit the xml template and remove it. If you need removing it, i'll help, you just need to let me know odoo version