DEP and Virtual Machines

Data Execution Prevention (DEP) is a security feature included in all versions of Windows since XP SP2. It’s intended to prevent an application or service from executing code from a non-executable memory region. This helps prevent certain exploits that store code via a buffer overflow, for example.

DEP runs in two modes: Hardware-enforced DEP for CPUs that support it, and software-enforced DEP for CPUs that don’t. Software DEP is performed by the operating system, and as such, has a (small) performance hit.

It may make sense to disable DEP in virtual machines (especially test VMs) to eek out a little more performance. Read on for an explanation of how to do this.

Software DEP configuration is controlled through switches in the Boot.ini file.

There are four options to set the DEP mode are:

OptIn - Enables DEP only for OS components, including the Windows kernel and Windows drivers. Administrators can enable DEP for selected executable files with the Application Compatibility Toolkit (ACT).
OptOut - Enables DEP for the OS and all processes, including the Windows kernel and Windows drivers. However, administrators can disable DEP on selected executable files with the Control Panel System applet.
AlwaysOn - Enables DEP for the OS and all processes, including the Windows kernel and Windows drivers. All attempts to disable DEP are ignored, and all DEP configuration options are disabled.
AlwaysOff - Disables DEP. Attempts to enable DEP selectively are ignored, and the DEP GUI is disabled.

In Windows Server 2008 and Vista, you use bcdedit to set the DEP mode. The DEP configuration can be viewed using the bcdedit /enum osloader /v command. To configure DEP, use the /set nx switch. For example, to set the currently booted OS to DEP AlwaysOff, you would use the command: