How to solve the "The Win 16 Subsystem has insufficient resources to continue running" problem on Windows XP?

BIG DISCLAIMER: This Technical Note is not of my own creation. It is attributed to Dan Fulger, and I obtained it from http://www.qwr391.home.ro/wow32.htm. Since I have no idea how long that Romanian Web site will stay up, I decided to reproduce it here.

Many further useful suggestions (written for users of the Oxford English Dictionary, but applicable to other 16-bit applications) can be found on a page maintained by by R. J. Holmgren.

(November 20, 2007): A patch program is now available that allows you to modify your own copy of WOW32.DLL. Please scroll to the bottom of this page.

PLEASE DO NOT ASK FOR COPIES OF THE DLL DISCUSSED HERE. I DO NOT DISTRIBUTE MODIFIED COPIES OF COPYRIGHTED MICROSOFT MATERIAL. USE THE LINKS PROVIDED HERE OR THE PATCH PROGRAM BELOW TO MODIFY YOUR OWN COPY.

Cause:

Windows NT 4.0 and Windows 2000 have a limit of 16384 GDI objects per session. This means that the only way to get around this is to log on in a different new session using Terminal Services/Remote Desktop.

The console session (session 0) controls the actual physical display and input devices (keyboard mouse), and there is no way to get around this limit. Please note that this has nothing to do with the per-process limit for GDI objects, which can be tuned from the registry. As far as I know, there is no way to get around this per-session limit for GDI objects, other than recompiling Windows from source code ;-)

In Windows XP and later versions (2003, etc.) the limit was extended to 65536 GDI objects per session.

This poses a serious problem for the Win16 subsystem, because, for some reason, it always shifts the GDI handles to the right by two bits, when converting them from 32 to 16 bits. This means that the actual handle cannot be larger than 14 bits.

When the limit was updated for Windows XP, rather than eliminate this bizarre shift to the right, Microsoft programmers decided to simply crash the 16-bit application that happens to be served a handle larger than 16384 by the GDI system.

We know this was a deliberate decision because of this error message, "The Win 16 Subsystem has insufficient resources to continue running", which always accompanies this type of crash and nothing else (it is specific to this type of crash), and is a new error message in Windows XP.

When does the problem occur:

The problem occurs when a 16-bit application is served a handle larger than 16384 by the GDI.

Because the GDI allocated handle numbers sequentially, and always prefers to reuse a freed handle rather than allocate a never-used higher handle, this can only happen if at some point during the current session the GDI handle count exceeded 16384. Please note that usually you are using session 0, the same that displays the logon screen when you start your computer, and so the only way to reset it is to reboot the computer (just like the second part of the error message says).

The error message is wrong when it says you can solve the problem by closing applications. Though closing applications could theoretically reduce the probability of hitting this problem, in practice it doesn't. The only thing that works is to reboot the computer (or, maybe, the patch below).

An illegal, dangerous, patch that could solve the problem for you

I made a version of wow32.dll that doesn't shift the handles to the right. You can download it from here. It is based on, and should work with, Windows XP Service Pack 2 (now also works with 3 -- VTT.)

Installing it is quite tricky, because wow32.dll is both a KnownDll and WFP protected. I believe the simplest idea is to use some sort of parallel installation of Windows, and using it to replace both copies of wow32.dll in your main installation (the first in windows\system32, the second in windows\system32\dllcache) with the one provided here, making sure you rename it to wow32.dll first.

Having a parallel installation of Windows is always a good thing to have around; for instance, you can use it in a disaster situation to try and repair your main installation of Windows. Of course, having a second computer and moving hard drives from one another is almost as good as a parallel installation of Windows. Microsoft recommends using parallel installations in many circumstances, but (correctly) warns people to install on a separate partition, so that the parallel installation will have its own "Documents and Settings" and "Program Files" folders, safely isolated from those of the main installation.

There are more advanced ways to install this file, without using a parallel installation or even rebooting your computer; I used a combination of a simple, one byte write in live kernel memory, to get around KnownDlls protection, and wfpreplace to get around WFP protection.

The only danger with using this file, besides the fact that it is illegal, is that it could, sometimes, crash your 16-bit programs, though I have never seen this. But if you came here, your 16-bit programs are probably crashing all the time anyway, or refusing to run at all, so this is certainly an improvement.

There are some serious dangers with installing the file however; I think that Windows will refuse to boot if it can't find wow32.dll where it expects it. As I said, installing the file is tricky. Having a parallel installation will help in that you can boot from it and put any version of wow32.dll in its correct place inside the main installation.

A patch program

Apparently, the Web page where I found this solution is no longer available. I am not in the business of redistributing copyrighted DLLs. However, I have created a patch program that edits the DLL.

Using this patch program requires knowledge of the command prompt. Here's how to do it:

  1. Open the command prompt.
  2. Copy the file WOW32.DLL from the Windows SYSTEM32 directory to a directory of your choice.
  3. Copy the program WOW32_PATCH_EXE for Service Pack 2 or for Service Pack 3, downloaded from here, to the same directory. (NB: Some browsers may not allow you to download an executable file. Find a browser that does.)
  4. Issue the command
    WOW32_PATCH WOW32.DLL WOW32-PATCHED.DLL

This should do it. If WOW32_PATCH executed correctly, it will say so. If it fails, it will also say so, but no, please don't ask for help, I can't afford the time to provide support.

In particular, this program will fail if your version of WOW32.DLL differs from mine.

If the patch was successful, you can proceed with Dan Fulger's instructions above*, using the patched copy of WOW32.DLL instead of one downloaded from his no longer functioning Web site.

For reference, here is a list of bytes changed by WOW32_PATCH.EXE:

Address   Old (SP2)   Old (SP3)   New
0x00000138  0x78  0x6b  0xee
0x00000139  0xa7  0xf4  0xd7
0x00003155  0xc1  0x90
0x00003156  0xf9  0x90
0x00003157  0x02  0x90
0x00013c87  0x3f  0xff
0x00013c88  0x0f  0x90
0x00013c89  0x86  0xe9
0x00013d68  0xc1  0x90
0x00013d69  0xe0  0x90
0x00013d6a  0x02  0x90

*Alternatively, as I was informed in an e-mail by Harald Pfeiffer, you can use a utility called MoveOnBoot by Gibin Software. As I have never used this tool myself, I cannot say whether or not it works as advertised, though as with all tools that manipulate the system like this, caution is advised. Another alternative, suggested by Gary P., is to use the boot disk tools from bootdisk.com, which include Avira's NTFS4FOS, allowing one to boot into a command line and manipulate files on an NTFS partition. (Avira NTFS4DOS is also available directly from Avira, albeit in a less convenient package.)