Brandon Live!

I’m pleased to announce that the second tool to join the BrandonTools.com collection is now available!  It’s a new Sidebar Gadget for those who want to see what the indexer is up to and to easily control its behavior.

Click here for details.

Note that the screenshot depicts the gadget running on WS4.  The "index now" button is not available on versions prior to Windows Search 4.

Following up on the Windows Search 4.0 Preview release, I will be writing several posts about some of the new features and changes enabled by this release.  One such feature, and this first one I will dive into here, is the capability to remotely search the index of another Windows PC.

This features isn’t entirely new.  Windows Vista shipped nearly a year and a half ago with the ability to query the index of another Vista machine when searching file shares.  The same capability extends to and from Windows Server 2008.

Windows Search 4.0 brings this capability to Windows XP machines, as well as Server 2003 - and perhaps more importantly, Windows Home Server.

So how does it work?  First let’s take a look at how the user sees it.  Let’s say I have a folder on Machine A called “Cool Stuff” that I want to share out.  One simple way to do that is to browse to the folder in Explorer, select it, and click “Share.”

You’ll then get a friendly dialog that asks you who you’d like to share with.

“Everyone” is a simple answer for information you want to be accessible to everyone.  Select it from the drop-down and click “Add” to add Everyone to the list of people the folder is shared with.

What else do I have to do on Machine A?  Nothing!  Windows Search 4.0 will automatically index any folders you share out, on both XP and Vista.

On Machine B, you simply navigate to the share as you normally would.  That could mean typing a UNC like \\MachineA\Cool Stuff\ or it could mean using a mapped drive, redirected User folders, the Network browser, etc.  Once there, just type a query in the Search box (or on XP, click the “Search” button to bring up the Search Pane) and you’re off!

Unfortunately I don’t have any XP machines to get a screenshot from, but I’ll try to add one soon.

Today we made available the WS 4.0 preview release for Windows XP, Vista, and Server 2003/2008.  You can read details about WS 4.0 at the following sites:

Vista Team Blog - Announcing the Windows Search 4.0 Preview

KB Article describing Windows Search 4.0 (with download links)

This release is mainly an update to the Windows Search indexer, and provides countless performance improvements, bug fixes, and reliability / recoverability features.

The XP/2003 version has been updated with more features previously exclusive to Vista - such as the ability to search remote indexes for network shares, and the ability to host Vista-style preview handlers in the preview pane.

WS4 also provides some cool new query capabilities for developers, which I will describe and give some examples of in future posts.

The most noticeable difference is probably how fast it is.  Those geniuses down the hallway in indexer land really pulled off some impressive feats with this release.

Since there are six different downloads depending on your OS, I’ll just refer you to the KB article for downloading the preview release.

Let us know what you think!

Let’s say you want to optimize your system by only indexing certain data.  For example, a reader recently e-mailed me and said “I only want to index my media files.”  Seems like a valid choice.  At first glance, it might seem like you could achieve this by telling Windows to only index files with extensions like .mp3 and .avi.  Ultimately, this is a very bad idea.

First, let me tell you why this is a bad idea.  Second, I’ll tell you the right way to achieve what you want.

Let’s begin by looking at how the Windows Vista shell and the indexer work together.

The indexer maintains a list of “start paths” - which are locations in the shell namespace that it cares about.  By default, it is set up to index the x:\Users directory - and thus all of the default Documents / Music / Pictures folders of all user accounts on the system.  When you install Outlook, it sets up a start path for your mail accounts.  OneNote sets one up for your OneNote data.  And so on.  This means that the indexer will try to index all items under that path*, and ignore everything else.

When you browse to a folder in Explorer, the shell asks the indexer if the current path is covered by the index.  If it is, Explorer will use the index exclusively for search / filter / grouping operations against that location.  It does not ask the index if it covers all the file types in that location.  It assumes the index is the authoritative source for information about that part of the namespace.

On the other hand, if the path is not covered by the index, Explorer walks the entire namespace starting at that location (so, the current folder and all subfolders) and enumerates every single item, performing all operations like filtering / sorting / grouping in-memory.  By default, it does not crack open any files being enumerated - so all filtering operations happen only against the basic properties like file name.  You can then click the “Search in File Contents” button (what some of us call the “try harder” button), and it will repeat the operation - stopping at every file and cracking it open with the appropriate IFilter and property handlers, doing essentially the same thing that happens when a file is indexed.  It loads the file, cracks it open, extracts all the properties and content, checks to see if it matches the current filter, and then decides whether or not to add that item to the view or ignore it.  If you change the filter, the whole process starts over again.  Needless to say, this is rather slow if you have to do it for more than a few files.  That’s why the “Search in File Contents” button is there, since in most unindexed locations (like C:\windows) you are probably only searching for a filename.

Armed with this information, let’s take another look at the original question.  Let’s say you go into the Advanced options for the indexer and tell it not to index .doc files at all.  Then you go save a new document called Something.doc inside of your Documents folder, which is still indexed.  The indexer will be notified that a new file was created there, but since you disabled indexing of that extension, it will ignore it.  Then when you go to your user folder or the Documents folder and search for “something” - you don’t find the document.  Even though it’s right there in the name.  The folder said “Hey, I’m indexed” and the file is not in the index, so as far as Explorer is concerned, it doesn’t exist.

A much better approach, if you really don’t care about indexing your .doc files, is to tell Windows not to index the Documents folder (or wherever you keep your .doc files).  That way it will fall back to slow GREP search when you look there, which will at least find what you’re looking for, albeit more slowly.

You can do this from the Indexing Options control panel, and it’s pretty easy to do.  Only want your music and videos indexed?  Then tell the indexer to only crawl the places where you store those files.  That it’s, mission accomplished.

The end result is the same.  The indexer isn’t doing any additional work, unless you mix .doc files and media files in the same folder.  And even then, at least you’ll be able to find them.

Another option available to you is to set certain extensions to “Index File Properties only.”  That way you’ll at least be able to find the item by its name.  Why would you want to do that?  I have no idea.  It’s not like indexing files incurs a significant overhead on any reasonably modern PC.  The option is mainly there because there are some file types the indexer can’t search inside.  So instead it indexes all the basic stuff that applies to every file (like name, date modified, and size).

* = It’s actually more complicated than that, as there can be nested inclusion/exclusion rules, files or folders excluded based on attributes, etc.  But that’s not particularly relevant to this discussion

PKEY_Identity (or “System.Identity”) is used to store the identity GUID associated with an Outlook Express or Windows Mail (Vista) account.

Tom Laird-McConnell, who used to be my boss until a few months ago, wrote one of his extremely rare blog posts last week on this subject.  To celebrate the occassion, I will link you there for the detailed answer to this question.

Definition of the PKEY_Identity / System.Identity property at Tom’s Handy Dandy Space

If you’re running a 64-bit version of Windows Vista, or a 64-bit version of WDS 3.x on Windows XP/2003, you may notice that the new Office 2007 document formats (.docx, .xlsx, etc) don’t show up when you search using the “Documents” filter in the search UI, or the kind:document Advanced Query Syntax.

This is a known issue with the 64-bit property system, and happens because the 64-bit shell only looks in the 64-bit section of the registry for a set of keys that map file extensions to various “kinds” for filetypes that don’t emit their own “kind” information.  Because Office 2007 is a 32-bit application, it registers its kinds in the 32-bit section of the registry, where the shell never sees it.

In a future release, the shell / search engine will be updated to better handle this situation.  For now, I have uploaded a .reg file which will fix the KindMap for Office 2007 documents on 64-bit machines.

Disclaimer Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. I cannot guarantee that these problems can be solved. Modify the registry at your own risk.

Click here for the workaround reg file

There’s been plenty to write about lately… the most obvious probably being WSUS accidentally installing WDS on unsuspecting machines and Google’s launch of OpenSocial.  Sadly, I’ve been neglecting my blog for a while now, so today I’m going to try and catch up a bit.

I hope that the WSUS issue has already been addressed for everybody affected by it.  But for the record, here’s the post I made in a couple of forums the day it happened:  

Here is my understanding of what happened. No one on the WDS product team knew about this until this morning (I was the first to know, because of an extremely impolite e-mail sent to my personal e-mail account).

• It was a screw-up, and everyone involved is deeply sorry about the trouble this has caused
• It ONLY affects WSUS systems where admins had approved an earlier WDS Update package
• The previous packages only updated existing WDS installations, and wouldn’t install it on new systems
• It was intentional to offer a package that would install WDS on machines without it
• It was NOT intentional for the approval of the previous update to be “inherited” by this package. This was a mistake in the publishing of this package to WSUS.

They have suspended deployment of the WDS package via WSUS while they fix the problem, and provided instructions for how admins can most easily disable and remove the WDS software.Believe me, Microsoft and the WDS team did not intend for this behavior. There was never any secret plan to force WDS onto unsuspecting machines. It was simply an error in the WSUS publishing process, which everyone deeply regrets.

As someone who used to work in IT, I feel the pain of these admins. This is also pretty embarassing for our team even though it could have happened to any group at Microsoft, as the WSUS publishing process is completely out of our control. That said, please don’t think that anyone here is taking this mistake lightly.

Even though our official responses may have appeared slow throughout the day, you can be sure that today was a non-stop fire drill for all involved. Once we identified and understood the problem, it took time to coordinate an official response and go through all the necessary approval processes. Unfortunate as that is, it’s the reality of a business this size, and our guys did their best to push through it and get this handled as best as we could after figuring out what happened.

For any further details, I refer you to the WSUS Blog.

If you have installed WDS on Windows XP / 2003 and wish to uninstall it, read the instructions below.  Also, please post a comment with your reason for uninstalling - user feedback is very important to all of us.

Standard answer:

1. Click on the Start button
2. Click “Control Panel”
3. Click “Add / Remove Programs”
4. Check the “Show Updates” box
5. Look for “Windows Desktop Search” in the list.

If that doesn’t work:

Try looking for the uninstaller, it normally resides in:

C:\WINDOWS\$NtUninstallKB917013$\spuninst\spuninst.exe

If it’s not there

Then you probably deleted it, or some ill-conceived ”tweak” or “disk cleaner” did.  Unfortunately, this puts you in a tough spot, since it’s really not a supported scenario or something we can design for (”I deleted the uninstaller - now I can’t uninstall!  Help!”).

Your best bet in this case might be to look and see if you have a System Restore point that you can revert back to, from before WDS was installed.  That should remove everything it added to the registry. 

I’ve added a new FAQ page to the site as well as a few FAQ entries to get things started.  I’ll keep adding more as time goes on.  If you have a question you’d like to see answered, post it in the comments or e-mail me.  But just because you ask, doesn’t mean an answer is guaranteed or that it will be timely.

The Indexer 

At its core, the Windows Search indexer doesn’t really know anything about files, e-mails, or anything like that.  In fact, all it really knows is how to do the following things:

1. Index contents and metadata associated with a URL and store it in a row.
2. Retrieve rows that match a specific query.
3. Shape the results in interesting ways (sorted, grouped, etc)
4. Retrieve properties / metadata associated with a row.

The indexer relies on other Windows Search components to handle the specifics, such as converting a URL into data to be indexed.  That’s where Protocol Handlers, IFilters, and Property Handlers come in.

Protocol Handlers

A Protocol Handler allows the indexer to crawl a specific kind of data store.  For example, the File System Protocol Handler allows the indexer to crawl files stored on your hard drive.  Windows Search includes a few Protocol Handlers including those for the File System, MAPI (ie. Outlook), and the Client-Side Cache for Offline Files (Vista only).  Other examples include the Protocol Handlers for Lotus Notes, the IE History / Cache, or Mozilla Thunderbird.

At a basic level, a Protocol Handler is just a piece of code that takes as input a URL (like “file://C:/Foo/” or “mapi://{USER-SID}/Brandon’s Mailbox/Inbox”) and performs two important tasks:

1. Enumeration of child URLs (such as “file://C:/Foo/Bar/” or “file://C:/Foo/Bar/Taxes.docx”)
2. Binding of URLs to either an IFilter, or a Stream (which can be bound to whatever IFilter is registered for its content type)

IFilters

An IFilter is responsible for taking an item such as a file (usually in the form of a Stream) and emitting the contents and properties of that item for indexing.

For example, the MS Word IFilter knows how to take the stream from a .DOC or .DOCX file and return both the contents and useful properties (like the author’s name or the date it was last modified) into the index.

Property Handlers

Property Handlers are similar to IFilters, except that they’re designed to simply return properties for items and not complex textual content.