The problem is that there's pretty much never a reason to have your character wear armour they aren't proficient with because of the disadvantage and not casting spells, you'd have to rework all the armour numbers, and it would interact weird with heavy armour which is designed to just give you a flat AC number (same as Tortle's natural armour, etc) plus potentially a shield.
