What sounds like a good idea at first will take revenge on you sooner or later. The mean thing is that, long after the installation of the system, something might break and you won't even be able to figure out that a missing installation set is the reason for this. Spare yourself the hassle and follow the sane defaults of OpenBSD by installing all the sets on every system.