Would it be possible to just completely remove permissions from categories and show them only if the user can see any categories under them? Basically like how the modules are setup to display the category only if you can see any under it.

Also, one other quick thing I thought of was having a per-user cache of the list of what forums/categories they can see if this change would add some extra load. Something like what the permissions cache in the users table does. It would just need to be cleared when changing permissions for the forums/user or changing the forums.