You can make the Priority field visible (in form view) to certain groups only and let your user can be part of that group. So you will only able to change the priority from form view. 

From kanban view remove the widget from priority field so nobody will able to change it. Just show priority field as normal field with some extra designs. 

I hope you get an Idea.