"Maximum file size" setting in "Manage extension groups" ACP module has a priority over the same setting in "Attachment settings" ACP module. It leads, for example, to the fact users are able to upload files belonging to certain extension groups larger than common max_filesize value.
This behavior is not obvious as well as not mentioned anywhere in ACP.

Additionally, the code responsible is in includes/functions_posting.php:

		if (!empty($extensions[$file->get('extension')]['max_filesize']))

		{

			$allowed_filesize = $extensions[$file->get('extension')]['max_filesize'];

		}

		else

		{

			$allowed_filesize = ($is_message) ? $config['max_filesize_pm'] : $config['max_filesize'];

		} 

So one of suggestions by nickvergessen (IRC discussion) is:

nickvergessen: and maybe use min() to determinate the setting, so the lowest value is taken?