There is a conceptual problem with the group/permission system: the "Never"-permission is useless for groups in most cases since a group member can remove himself from any group except his main group.
To give a real live example: consider a forum where every user has permission to use the pm- and the signature-feature. But you want to disallow these functions for a few users having misused these features.
If you create a group "No pm usage" giving the "Never"-permission for pm-usage and a second one for the signature-feature you have to add a user to both groups to disallow both features for him. But for at least one of these groups the user can remove himself from the group and recover the permission.
Creating a group "restricted permissions" with both permissions unset and adding the permissions with groups like "allow pm usage" won't work since a "no" will be overwritten by the "Yes" of all registered users and a "never" cannot be overwritten.
So you either have to create three groups (one for no pm-usage, one for no signature-usage and one for both) or have to assign the permission on a user basis. Both not a good idea.
Solution might be to introduce a special group type / setting which disallows self-removing for group members. Might be a good idea to add a flag to hide the group for the users, too.