If you set Group wide settings > Maximum number of allowed recipients per private message: to say 2 for the Registered User group, this works fine when adding individual recipients.

But users can still add multiple groups to the Private Message recipient list.

It is only once 3 individual user names have been added that the system seems to check at which point it reverts back to 2 groups with the expected message "You tried to send a private message to too many recipients." If you don't add any individual names then the user can keep adding groups beyond the maximum.