Measuring memory usage in Windows 7

Historically, measuring the amount of memory in use by a Windows system has been a somewhat confusing endeavor.  The labels on various readouts in Task Manager, among other places, were often either poorly named or simply misunderstood.  I’ll tackle a prime example of this, the “commit” indicator, later in this post.  But first, let’s look at a simple way to measure the amount of physical memory in use on your system.

In Windows 7, the folks building the Task Manager performance tab tried to make it a little easier to understand the usage of physical memory on your system.  The most interesting bits are here:

What do these values tell us?

- We are looking at a machine with 4GB of physical memory installed.

- 71% of that physical memory is currently in use by applications and the system.

- That leaves 29% of memory “available”, despite the indication that only 16MB of physical memory is totally “free.”

Here’s a description of the four labels, from the bottom:

Free – This one is quite simple.  This memory has nothing at all in it.  It’s not being used and it contains nothing but 0s.

Available – This numbers includes all physical memory which is immediately available for use by applications.  It wholly includes the Free number, but also includes most of the Cached number.  Specifically, it includes pages on what is called the “standby list.”  These are pages holding cached data which can be discarded, allowing the page to be zeroed and given to an application to use.

Cached – Here things get a little more confusing.  This number does not include the Free portion of memory.  And yet in the screenshot above you can see that it is larger than the Available area of memory.  That’s because Cached includes cache pages on both the “standby list” and what is called the “modified list.”  Cache pages on the modified list have been altered in memory.  No process has specifically asked for this data to be in memory, it is merely there as a consequence of caching.  Therefore it can be written to disk at any time (not to the page file, but to its original file location) and reused.  However, since this involves I/O, it is not considered to be “Available” memory.

Total – This is the total amount of physical memory available to Windows.

Now, what’s missing from this list?  Perhaps, a measurement of “in use” memory.  Task Manager tells you this in the form of a percentage of Total memory, in the lower right-hand corner of the screenshot above.  71%, in this case.  But how would you calculate this number yourself?  The formula is quite simple:

Total – Available = Physical memory in use (including modified cache pages).

If you plug in the values from my screenshot above, you’ll get:

4026MB – 1150MB = 2876MB

This matches up with the 71% calculation.  4026 * .71 = 2858.46MB.

Recall that this number includes the modified cache pages, which themselves may not be relevant if you are trying to calculate the memory “footprint” of all running applications and the OS.  To get that number, the following formula should work

Total – (Cached + Free) = Physical memory in use (excluding all cache data).

On the example system above, this means:

4026MB – (1184 + 16) = 2826MB

By looking at the difference between these two results, you can see that my laptop currently has 50MB worth of disk cache memory pages on the modified list.

So what is “commit?”

Earlier I said that measuring physical memory usage has been tricky in the past, and that the labels used in Windows haven’t necessarily helped matters.  For example, in Windows Vista’s Task Manager there is a readout called “page file” which shows two numbers (i.e 400MB / 2000MB).  You might guess that the first number indicates how much page file is in use, and the second number indicates the amount of disk space allocated for use – or perhaps some sort of maximum which could be allocated for that purpose.

You would be wrong.  Even if you disabled page files on each of your drives, you would still see two non-zero numbers there.  The latter of which would be the exact size of your installed physical RAM (minus any unavailable to the OS because of video cards, 32-bit limitations, etc).  Unfortunately, the label “page file” didn’t mean what people thought it meant.  To be honest, I’m not quite sure why that label was chosen.  I would have called it something else.

In Windows 7, that label changed to “Commit.”  This is a better name because it doesn’t lend itself as easily to misinterpretation.  However, it’s still not readily apparent to most people what “commit” actually means.  Essentially, it is this:

The total amount of virtual memory which Windows has promised could be backed by either physical memory or the page file.

An important word there is “could.” Windows establishes a “commit limit” based on your available physical memory and page file size(s).  When a section of virtual memory is marked as “commit” – Windows counts it against that commit limit regardless of whether it’s actually being used.  The idea is that Windows is promising, or “committing,” to providing a place to store data at these addresses.  For example, an application can call VirtualAlloc with MEM_COMMIT for 4MB but only actually write 2MB of data to it.  This will likely result in 2MB of physical memory being used.  The other 2MB will never use any physical memory unless the process reads from or writes to it. It is still charged against the commit limit, because Windows has made a guarantee that the application can write to that space if it wants.  Note that Windows has not promised 4MB of physical memory, however.  So when the process writes there, it may use physical memory or it may use the page file.

This is a great example of why disabling your page file is a bad idea. If you don’t have one, Windows will be forced to back all commits with physical memory, even committed pages which are never used!

Further, processes may be charged against the commit limit for other things.  For example, if you create a view of a file mapping with the FILE_MAP_COPY flag (indicating you want Copy-On-Write behavior for writes to the file view), the entire size of the mapped view will be charged as Commit… even though you haven’t used any physical memory or page file yet.  I wrote a simple scratch program which demonstrates this:

int wmain(int cArgs, PWSTR rgArgs[])
{ 
if (cArgs == 2)
     {      
         HANDLE hFile;
         hFile = CreateFile(rgArgs[1], GENERIC_READ | GENERIC_WRITE, 0, nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
         if (hFile != INVALID_HANDLE_VALUE)
         {
             HANDLE hMapping;
             hMapping = CreateFileMapping(hFile,
nullptr, PAGE_READWRITE, 0, 0, nullptr);
             if (hMapping != nullptr)
             {
                 void *pMapping = MapViewOfFile(hMapping, FILE_MAP_COPY, 0, 0, 0);
                if (pMapping != nullptr)
                 {
                     wprintf(L"File mapped successfully.  Press any key to exit and unmap.");
                     getwchar();
                     UnmapViewOfFile(pMapping);
                 }
                 CloseHandle(hMapping);
             }
             CloseHandle(hFile);
         }
     }
    return 0;
}

Before running this program, let’s take a look at Task Manager again.

Now, if I run this scratch program and pass it the path to my Visual Studio 2010 Beta 2 ISO image (a 2.3GB file), the Task Manager readout changes to:

Notice how my physical memory usage is unchanged, despite the fact that Commit has now increased by the full 2.3GB of that file.

In fact, my commit value is now 6GB, even though I have only 4GB of physical memory and less than 3GB in use.

Note: It is not common for applications to commit enormous file mappings in this way.  This is merely a demonstration of Commit and Used Physical Memory being distinctly different values.