In Windows 7, the folks building the Task Manager performance tab tried to make it a little easier to understand the usage of physical memory on your system.  The most interesting bits are here:

What do these values tell us?

- We are looking at a machine with 4GB of physical memory installed.

- 71% of that physical memory is currently in use by applications and the system.

- That leaves 29% of memory “available”, despite the indication that only 16MB of physical memory is totally “free.”

Here’s a description of the four labels, from the bottom:

Free – This one is quite simple.  This memory has nothing at all in it.  It’s not being used and it contains nothing but 0s.

Available – This numbers includes all physical memory which is immediately available for use by applications.  It wholly includes the Free number, but also includes most of the Cached number.  Specifically, it includes pages on what is called the “standby list.”  These are pages holding cached data which can be discarded, allowing the page to be zeroed and given to an application to use.

Cached – Here things get a little more confusing.  This number does not include the Free portion of memory.  And yet in the screenshot above you can see that it is larger than the Available area of memory.  That’s because Cached includes cache pages on both the “standby list” and what is called the “modified list.”  Cache pages on the modified list have been altered in memory.  No process has specifically asked for this data to be in memory, it is merely there as a consequence of caching.  Therefore it can be written to disk at any time (not to the page file, but to its original file location) and reused.  However, since this involves I/O, it is not considered to be “Available” memory.

Total – This is the total amount of physical memory available to Windows.

Now, what’s missing from this list?  Perhaps, a measurement of “in use” memory.  Task Manager tells you this in the form of a percentage of Total memory, in the lower right-hand corner of the screenshot above.  71%, in this case.  But how would you calculate this number yourself?  The formula is quite simple:

Total – Available = Physical memory in use (including modified cache pages).

If you plug in the values from my screenshot above, you’ll get:

4026MB – 1150MB = 2876MB

This matches up with the 71% calculation.  4026 * .71 = 2858.46MB.

Recall that this number includes the modified cache pages, which themselves may not be relevant if you are trying to calculate the memory “footprint” of all running applications and the OS.  To get that number, the following formula should work

Total – (Cached + Free) = Physical memory in use (excluding all cache data).

On the example system above, this means:

4026MB – (1184 + 16) = 2826MB

By looking at the difference between these two results, you can see that my laptop currently has 50MB worth of disk cache memory pages on the modified list.

So what is “commit?”

Earlier I said that measuring physical memory usage has been tricky in the past, and that the labels used in Windows haven’t necessarily helped matters.  For example, in Windows Vista’s Task Manager there is a readout called “page file” which shows two numbers (i.e 400MB / 2000MB).  You might guess that the first number indicates how much page file is in use, and the second number indicates the amount of disk space allocated for use – or perhaps some sort of maximum which could be allocated for that purpose.

You would be wrong.  Even if you disabled page files on each of your drives, you would still see two non-zero numbers there.  The latter of which would be the exact size of your installed physical RAM (minus any unavailable to the OS because of video cards, 32-bit limitations, etc).  Unfortunately, the label “page file” didn’t mean what people thought it meant.  To be honest, I’m not quite sure why that label was chosen.  I would have called it something else.

In Windows 7, that label changed to “Commit.”  This is a better name because it doesn’t lend itself as easily to misinterpretation.  However, it’s still not readily apparent to most people what “commit” actually means.  Essentially, it is this:

The total amount of virtual memory which Windows has promised could be backed by either physical memory or the page file.

An important word there is “could.” Windows establishes a “commit limit” based on your available physical memory and page file size(s).  When a section of virtual memory is marked as “commit” – Windows counts it against that commit limit regardless of whether it’s actually being used.  The idea is that Windows is promising, or “committing,” to providing a place to store data at these addresses.  For example, an application can call VirtualAlloc with MEM_COMMIT for 4MB but only actually write 2MB of data to it.  This will likely result in 2MB of physical memory being used.  The other 2MB will never use any physical memory unless the process reads from or writes to it. It is still charged against the commit limit, because Windows has made a guarantee that the application can write to that space if it wants.  Note that Windows has not promised 4MB of physical memory, however.  So when the process writes there, it may use physical memory or it may use the page file.

This is a great example of why disabling your page file is a bad idea. If you don’t have one, Windows will be forced to back all commits with physical memory, even committed pages which are never used!

Further, processes may be charged against the commit limit for other things.  For example, if you create a view of a file mapping with the FILE_MAP_COPY flag (indicating you want Copy-On-Write behavior for writes to the file view), the entire size of the mapped view will be charged as Commit… even though you haven’t used any physical memory or page file yet.  I wrote a simple scratch program which demonstrates this:

int wmain(int cArgs, PWSTR rgArgs[])
{ 
if (cArgs == 2)
     {      
         HANDLE hFile;
         hFile = CreateFile(rgArgs[1], GENERIC_READ | GENERIC_WRITE, 0, nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
         if (hFile != INVALID_HANDLE_VALUE)
         {
             HANDLE hMapping;
             hMapping = CreateFileMapping(hFile,
nullptr, PAGE_READWRITE, 0, 0, nullptr);
             if (hMapping != nullptr)
             {
                 void *pMapping = MapViewOfFile(hMapping, FILE_MAP_COPY, 0, 0, 0);
                if (pMapping != nullptr)
                 {
                     wprintf(L"File mapped successfully.  Press any key to exit and unmap.");
                     getwchar();
                     UnmapViewOfFile(pMapping);
                 }
                 CloseHandle(hMapping);
             }
             CloseHandle(hFile);
         }
     }
    return 0;
}

Before running this program, let’s take a look at Task Manager again.

Now, if I run this scratch program and pass it the path to my Visual Studio 2010 Beta 2 ISO image (a 2.3GB file), the Task Manager readout changes to:

Notice how my physical memory usage is unchanged, despite the fact that Commit has now increased by the full 2.3GB of that file.

In fact, my commit value is now 6GB, even though I have only 4GB of physical memory and less than 3GB in use.

Note: It is not common for applications to commit enormous file mappings in this way.  This is merely a demonstration of Commit and Used Physical Memory being distinctly different values.

By creating a new folder in Windows 7 and renaming it with a certain text string at the end, users are able to have a single place to do everything from changing the look of the mouse pointer to making a new hard drive partition.

Apparently people decided to call this “God Mode” because to enable this “trick” you make a folder called GodMode.{ED7BA470-8E54-465E-825C-99712043E01C} and double-click on it.  Now you can see… the control panel.  With a slightly different view than you normally see it in.

So first off, why is this completely silly?  Well, the text ”GodMode” has nothing to do with making the trick work.  You can call the folder “ILikePuppies.{ED7BA470-8E54-465E-825C-99712043E01C}” and now you’ve discovered the magical “ILikePuppies” feature hidden in Windows.

Well, not really.  What you’ve actually discovered are two things:

First, you’ve discovered a documented feature of the shell whereby filesystem folders can be easily made into namespace junctions, as described here on MSDN.  Basically, any folder named <DisplayName>.<CLSID> will show up with just the <DisplayName> portion visible in Explorer, and navigating into the folder will take you to the namespace root defined by the <CLSID> portion of the name.  This isn’t a user feature, it’s a developer feature.

The second thing you’ve discovered is the “All Tasks” folder.  This is a special shell folder which is used as the source of the “Control Panel” search results seen in the Start menu.  This folder was not designed to be browsed to directly, as the normal Control Panel folder (accessible via Start -> Control Panel) contains all the same items but with a custom view designed to be easier to navigate.  The “All Tasks” folder has no custom view, so you just see the standard Explorer list view and little else.

The existence of this folder and its CLSID are implementation details and should not be relied upon by anybody for any purpose*.

God Mode?  Hardly.

Props to Ed Bott for his earlier post about this “trick.”

* = Update: The intended meaning of this statement was, “please don’t use this folder / CLSID in your app for anything, since it will probably break in the future.”  If you like using this view and are happy with an unsupported untested trick for your own personal use, then by all means, enjoy it.

If Vista had gotten the reception it actually deserved, and become a brand worth keeping, it seems highly likely that the name would have been retained. And Microsoft knows it, which is why internally, Windows 7 is only version 6.1. Sure, the company has made specious claims that this is to avoid breaking applications with bad version checks, but the logic doesn’t really hold; many of those applications are just as broken by “6″ as they would be by “7,” and if that were such a concern then the minor version wouldn’t change either.

This might make sense to a lay person on a cursory reading, but as usual with software development, reality is full of subtle but important complexities. 

Pop quiz

Consider the following rather common version checking mistake:

// Our app is designed for XP (5.1) and later
if (majorVersion >= 5 && minorVersion >= 1)
{
    // Succeed
}
else
{
    // Fail
}

Given the returned version values from Windows XP (reports 5.1), Windows Vista (reports 6.0), and Windows 7 (reports 6.1) – which OSes will this program succeed on?

You can see a bit from the Windows ship room sign-off event at the end of this cool RTM video clip:

You can also read more about it here:

Windows 7 Team Blog – Windows 7 has been released to manufacturing

Engineering 7 Blog – Our next engineering milestone: RTM

Woohoo!  Time to go do a bit more celebrating

There’s also a note in there confirming speculation that a Family Pack offering for installation on up to 3 PCs is in the works.

This week we published an update to the Federated Search Implementer’s Guide.  This update includes some new information as well as some corrections to the original document.  Most crucially, the original document erroneously indicated that the URL template contained an attribute called “format” – when it was referring to the “type” attribute.  So if you followed the guide exactly, your OSDX file wouldn’t work properly with Windows 7 (or anybody else following the OpenSearch spec).

This update also includes additions such as information about providing “collection” templates to display results without the user entering a query, details about URL protocol support, and more.

Link: Windows 7 Federated Search Provider Implementer’s Guide

The new page roughly sums up its use and purpose:

A client-side OpenSearch data source that sits in between the Microsoft Windows OpenSearch provider and the external data source.  With a search connector (a .searchconnector-ms file), Windows Explorer calls your implementation with the query parameters. Your implementation returns results formatted in RSS or Atom format. That allows your implementation to provide custom authentication UI and connect to the data source using its proprietary API.

More details at:  http://msdn.microsoft.com/en-us/library/dd378289(VS.85).aspx

Unfortunately we still don’t have any details on how you actually hook up an implementation of this.  The doc / SDK folks are working on this but it’s part of a larger effort that’s taking a bit of time.  In the meantime I’m going to see if I can post an example here or on CodePlex.  Most likely I’ll be posting a detailed .NET example followed by a simpler C++ version.

If you have questions about whether this API may end up being useful to you, let me know, and hopefully I can help clarify what is or isn’t possible.

Check it out and let us know what you think!

This document covers subjects such as crafting the best OpenSearch Description File for your site, returning files and custom properties, customizing the results view, optimizing your source for performance, and more. If you’re interested in developing an OpenSearch source for Windows 7, check it out! And as always, please share your feedback!

Link: Windows 7 Federated Search Provider Implementer’s Guide

Ed Bott does a nice job covering the announcement.

The main changes, which I am very happy about, are:

• Two main Editions for consumers.  Home Premium, and Professional.
• Professional now contains Media Center and everything else in Home Premium, as well as the traditional Pro feature set (Remote Desktop, domain support, etc).

Not only is this simpler, but I think it may constitute a price cut for people who needed to buy the Ultimate version of Vista in order to get both Media Center and Remote Desktop (or Domain support) in the same package.  Now they can get a machine with Pro and be all set.

Here’s the official press release / Q&A with Windows General Manager Mike Ybarra.

General

Windows Explorer

Taskbar modifiers

Note that this list focuses on stuff that’s specific to Windows 7. For a complete list of Windows Vista hotkeys (which should be largely unchanged except where noted above), visit this page on Microsoft.com or look in the Windows Help file.

If there’s something you think I should add, post a comment!

One commentor wrote:

“Maybe it’s finally time for MS to stop catering to bad ignorant developers that skip on reading the documentation? I suppose that’s what big A did.”

This, of course, is making an assumption that the design had anything to do with ignorant developers or developers not following some documentation.

But think about it… exactly which documentation would they not be following? I mean, if a developer built a 32-bit application that hardcoded windows\system32 – then clearly they wouldn’t have a problem if the 32-bit binaries were left in that directory. Their 32-bit application would continue to find the binaries its looking for just as before. On the other hand, if they’re going through the trouble to recompile their application for 64-bit, they could probably handle fixing that little issue along the way, right?

So it stands to reason that the decision was made on a different basis. Let’s see, if it wasn’t to support existing 32-bit apps, and it wasn’t for apps that are recompiled as 64-bit, then who would this arrangement benefit?

Well, I wasn’t around when WOW64 came about, but I have a theory. There’s another little kind of “program” out there that isn’t really 32-bit or 64-bit, in fact is isn’t even compiled at all. They aren’t sold on store shelves, but there are millions of them, often vital to the businesses and developers who use them.

They’re called scripts, and the chosen WOW64 model keeps most of them chugging along quite happily.

It looks like a work-in-progress, and since his service works as a proxy to talk to Flickr, performance isn’t what it would be if Flickr had a native implementation.  But for something he threw together over the last night or two I’m impressed!

We’ll be talking more about Search Federation on the Engineering Windows 7 blog soon, and will have much more to share when we deliver the upcoming Windows 7 beta release.