GRKERNSEC_VM86 Restrict VM86 Mode

The description of this configuration option is very clear here.

config GRKERNSEC_VM86
	bool "Restrict VM86 mode"
	depends on X86_32

	help
	  If you say Y here, only processes with CAP_SYS_RAWIO will be able to
	  make use of a special execution mode on 32bit x86 processors called
	  Virtual 8086 (VM86) mode.  XFree86 may need vm86 mode for certain
	  video cards and will still work with this option enabled.  The purpose
	  of the option is to prevent exploitation of emulation errors in
	  virtualization of vm86 mode like the one discovered in VMWare in 2009.
	  Nearly all users should be able to enable this option.

The VM86 mode was used in various situations with the latest one being the well known VMware issue of 2009. Now, if we have a look at arch/x86/kernel/vm86_32.c file we can see how this patch is applied…

int sys_vm86old(struct vm86_struct __user *v86, struct pt_regs *regs)
{
        struct kernel_vm86_struct info; /* declare this _on top_,
                                         * this avoids wasting of stack space.
                                         * This remains on the stack until we
                                         * return to 32 bit user space.
                                         */
        struct task_struct *tsk;
        int tmp, ret = -EPERM;

#ifdef CONFIG_GRKERNSEC_VM86
	if (!capable(CAP_SYS_RAWIO)) {
		gr_handle_vm86();
		goto out;
	}
#endif

        tsk = current;
     ...
out:
        return ret;
}

So, before executing the code of sys_vm86old() system call, it will perform a capability check to ensure that the calling process has ‘CAP_SYS_RAWIO’ (System Raw I/O) POSIX capability set. However, if it doesn’t, it will invoke gr_handle_vm86() grsecurity’s routine and then jump to ‘out’ label that returns with a return value set to ‘-EPERM’ (Permission Denied). The same approach is used in the sys_vm86() too.

int sys_vm86(unsigned long cmd, unsigned long arg, struct pt_regs *regs)
{
        struct kernel_vm86_struct info; /* declare this _on top_,
                                         * this avoids wasting of stack space.
                                         * This remains on the stack until we
                                         * return to 32 bit user space.
                                         */
        struct task_struct *tsk;
        int tmp, ret;
        struct vm86plus_struct __user *v86;

#ifdef CONFIG_GRKERNSEC_VM86
	if (!capable(CAP_SYS_RAWIO)) {
		gr_handle_vm86();
		ret = -EPERM;
		goto out;
	}
#endif

        tsk = current;
        switch (cmd) {
        case VM86_REQUEST_IRQ:
     ...
out:
        return ret;
}

And as you might have guessed, gr_handle_vm86() is nothing more than a logging routine located at grsecurity/grsec_mem.c.

void
gr_handle_vm86(void)
{
	gr_log_noargs(GR_DONT_AUDIT, GR_VM86_MSG);
	return;
}