In this small how-to we discuss different methods for iterating Outlook Folders and its subfolders.
In the Outlook Object Model (OOM) a folder is represented by a type called MAPIFolder.
There are different ways to retrieve a MAPIFolder instance:
- Using the Session.GetDefaultFolder(OlDefaultFolders) method to retrieve predefined folders like Inbox
- Using the Session.GetFolderById(folderId, [optional] storeId) method to retrieve a Folder by its EntryId
- Retrieving the Parent Property of an Item or another Folder
- Retrieving the Explorer.CurrentFolder Property
- Iterate over the Folders collection of a Store or a Folder instance.
- By creating a new Folder
Recursively iterating a folder tree and accessing Outlook Items can be a long running process. It depends on the amount of Items in a Folder and the levels of subfolders. When programming against the Outlook Object Model using .Net(managed) code you need to take care about memory management and how to use the COM Objects. Accessing and retrieving Outlook Objects like Emails, Appointments and Tasks could be very slow. This is by design and caused by the MAPI level where the Outlook Object Model is built on top. It also depends on the configuration, if you are using an Exchangeserver, if you are working connected(online) or offline and how fast the network connection is.
RPC Connections problem
This problem occures only if you are connected to an Exchangeserver and when you access multiple Outlook Items in a short time. For each Item that you access by code, a connection (Remote Procedure Call) is made to the server and the item is retrieved or synchronized. By default the size of RPC-connections made to an Exchangeserver by one Client is limited to 256. If an Outlook solution is developed offline – or just with a few amount of test data – the developer may never run into this problem. When the solution is deployed to the customer and under heavy load you may get unexpected behavior. So whenever you iterate over Outlook Items you need to take care of this problem. First we simply iterate over Outlook Items in the Inbox. Let's discuss different possibilities. For testing purposes we define a method to get the Inbox Folder Object.
Outlook.MAPIFolder GetInboxFolder() {
returnGlobals.ThisAddIn.Application.Session.GetDefaultFolder(Outlook.OlDefaultFolders.olFolderInbox);
}
The first test iterates over the Inbox folder Items and traces the Subject of the Item.
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
// Iterate over all Inbox items and get the Subject
foreach (object item in inboxFolder.Items) {
Trace.WriteLine (item.GetType ().InvokeMember ("Subject", BindingFlags.GetProperty, null, item, null));
}
What's wrong with this code? Theoretically this code is OK, but
- when there are more Items then available RPC-Connections?
- an Outlook Items collection can change the number of items?
- the process can't be interrupted until you iterated over all items
- the process is running in the main thread of the Outlook process – this means that the GUI is not responsible until the method has finished.
How can you solve it?
First we avoid the RPC-connections problem. We will define a variable that counts the processed items. While iterating over the folder content we increase the count of it and after a predefined value we force the .Net garbage collector to release the references from memory. The problem is: The .Net Garbage Collector is intelligent. It caches the data you accessed for a while – so if you access the same item again within a specific time you get this object again very fast. This is also true for Outlook Objects. When you access an Object a reference is kept in memory - even when you set this object to null by code. The GC decides when to finally release an object. Alternatively you can force the GC to do a Cleanup. Note: A GC cleanup is very expensive by consuming CPU load and some time – GC often will slow down all Applications – so you need to find a good mix. It is absolutely a must to do a GC cleanup after using and releasing an Inspector or an Explorer object.
Study the next snippet.
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
int watchDog = 0;
// Iterate over all Inbox items and get the Subject
foreach (object item in inboxFolder.Items) {
Trace.WriteLine (item.GetType ().InvokeMember ("Subject", BindingFlags.GetProperty, null, item, null));
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
The watchdog variable is used to force the Garbagecollector to release the COM objects from memory and close the RPC connections.
Note that the Folder.Items collection is not static – it can change its content at any time. Instead of iterating over the Items collection using the foreach statement, consider getting the collection count and process the items using a countdown loop.
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
int watchDog = 0;
// Iterate over all Inbox items and get the Subject
for (int index = inboxFolder.Items.Count; index > 0; index-- ) {
object item = inboxFolder.Items[index];
Trace.WriteLine(item.GetType().InvokeMember("Subject", BindingFlags.GetProperty, null, item, null));
item = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
The index variable is used for iterating over a number of items from last to first. This way you can delete or move items while iterating the collection. The iteration process still cannot be interrupted, because the process runs in the main thread of Outlook. To turn this into a stoppable and responsive iteration process, you need to use a thread for it. In the .Net Framework there is already a Framework that you can use called BackgroundWorker.
Check out the next code snippet.
BackgroundWorker _worker;
privatevoid ThisAddIn_Startup(object sender, System.EventArgs e)
{
// Setup the background thread
_worker = newBackgroundWorker();
_worker.DoWork += newDoWorkEventHandler(_worker_DoWork);
_worker.WorkerSupportsCancellation = true;
_worker.RunWorkerAsync();
}
void _worker_DoWork(object sender, DoWorkEventArgs e) {
BackgroundWorker worker = (BackgroundWorker)sender;
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
int watchDog = 0;
// Iterate over all Inbox items and get the Subject
for (int index = inboxFolder.Items.Count; index > 0; index--) {
// if someone interrupts the worker exit here
if (worker.CancellationPending) break;
object item = inboxFolder.Items[index];
Trace.WriteLine(item.GetType().InvokeMember("Subject", BindingFlags.GetProperty, null, item, null));
item = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
}
privatevoid ThisAddIn_Shutdown(object sender, System.EventArgs e)
{
if (_worker.IsBusy) {
_worker.CancelAsync();
}
}
Using a BackgroundWorker you can execute a long running process in a separate thread. The UI keeps responsive and you can interrupt the operation.
Recursive Operations
Next we are going to iterate recursively over a folder tree. We implement a function that calls itself recursively.
void IterateFolder(Outlook.MAPIFolder folder) {
int watchDog = 0;
// Iterate over all Inbox items and get the Subject
for (int index = folder.Items.Count; index > 0; index--) {
object item = folder.Items[index];
Trace.WriteLine(item.GetType().InvokeMember("Subject", BindingFlags.GetProperty, null, item, null));
item = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
for (int index = folder.Folders.Count; index > 0; index--) {
Outlook.MAPIFolder subFolder = folder.Folders[index];
// Call this function recursively
IterateFolder(subFolder);
subFolder = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
}
While this is working – it's not the best approach. Calling recursive functions can mess up the processor call stack. How can we solve this?
One easy solution is to use a Queue Object that is already part of the .Net Framework. A Queue has two essentials Methods called Enqueue() and Dequeue().
The Enqueue inserts an object at the end while Dequeue removes the first object of a Queue. In this scenario EntryID's of the folders that needs to be iterated recursively are stored in a queue. In the worker method the number of items in the queue is checked and if there is an item in the queue, it is processed. The EntryID of the folder is removed of the Queue and the folder is processed. While processing, each of the subfolders EntryID is added to the end of the Queue. This way there is no problem with the stack of the program and the process can be interrupted at any time.
BackgroundWorker _worker;
Queue<string> _folderIdsToProcess;
privatevoid ThisAddIn_Startup(object sender, System.EventArgs e) {
_folderIdsToProcess = newQueue<string>();
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
// and add it to the queue
_folderIdsToProcess.Enqueue(inboxFolder.EntryID);
// Setup the background thread
_worker = newBackgroundWorker();
_worker.DoWork += newDoWorkEventHandler(_worker_DoWork);
_worker.WorkerSupportsCancellation = true;
_worker.RunWorkerAsync();
}
void _worker_DoWork(object sender, DoWorkEventArgs e) {
BackgroundWorker worker = (BackgroundWorker)sender;
Outlook.MAPIFolder folder;
int watchDog = 0;
while (_folderIdsToProcess.Count > 0 && !worker.CancellationPending) {
// pick one folder
string entryId = _folderIdsToProcess.Dequeue();
folder = Globals.ThisAddIn.Application.Session.GetFolderFromID(entryId, Type.Missing);
// Iterate over all Inbox items and get the Subject
for (int index = folder.Items.Count; index > 0; index--) {
if (worker.CancellationPending) break;
object item = folder.Items [index];
Trace.WriteLine(item.GetType().InvokeMember("Subject", BindingFlags.GetProperty, null, item, null));
item = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
for (int index = folder.Folders.Count; index > 0; index--) {
if (worker.CancellationPending) break;
Outlook.MAPIFolder subFolder = folder.Folders[index];
// Call this function recursively
_folderIdsToProcess.Enqueue(subFolder.EntryID);
subFolder = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
folder = null;
}
}
privatevoid ThisAddIn_Shutdown(object sender, System.EventArgs e) {
if (_worker.IsBusy) {
_worker.CancelAsync();
}
}
There is no problem accessing the queue variable by different threads because the Queue is modified only by the worker thread.
Making things go faster
You may notice that iterating over a large number of Items costs a lot of time. There are a few Options that may help you to optimize your application. The first option is to use a Filter to restrict the number of items in a folder. Let's say you want to search for Items that are modified after a specific time.
BackgroundWorker _worker;
Queue<string> _folderIdsToProcess;
privatevoid ThisAddIn_Startup(object sender, System.EventArgs e) {
_folderIdsToProcess = newQueue<string>();
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
// and add it to the queue
_folderIdsToProcess.Enqueue(inboxFolder.EntryID);
// build a filter
// note that this filter is build a little bit complicated
// This works even if you are using an Outlook even with another language format, not only en-us
DateTime from = DateTime.Now.AddMonths(-1);
DateTime to = DateTime.Now;
string filter = string.Format("[LastModificationTime] > '{0}/{1}/{2}' AND [LastModificationTime] <='{3}/{4}/{5}'"
, from.Year, from.Month, from.Day
, to.Year, to.Month, to.Day);
// Setup the background thread
_worker = newBackgroundWorker();
_worker.DoWork += newDoWorkEventHandler(_worker_DoWork);
_worker.WorkerSupportsCancellation = true;
_worker.RunWorkerAsync(filter);
}
void _worker_DoWork(object sender, DoWorkEventArgs e) {
BackgroundWorker worker = (BackgroundWorker)sender;
string filter = (string)e.Argument;
Outlook.MAPIFolder folder;
int watchDog = 0;
while (_folderIdsToProcess.Count > 0 && !worker.CancellationPending) {
// pick one folder
string entryId = _folderIdsToProcess.Dequeue();
folder = Globals.ThisAddIn.Application.Session.GetFolderFromID(entryId, Type.Missing);
Outlook.Items items = folder.Items.Restrict(filter);
// Iterate over all Inbox items and get the Subject
for (int index = items.Count; index > 0; index--) {
if (worker.CancellationPending) break;
object item = items[index];
Trace.WriteLine(item.GetType().InvokeMember("Subject", BindingFlags.GetProperty, null, item, null));
item = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
for (int index = folder.Folders.Count; index > 0; index--) {
if (worker.CancellationPending) break;
Outlook.MAPIFolder subFolder = folder.Folders[index];
// Call this function recursively
_folderIdsToProcess.Enqueue(subFolder.EntryID);
subFolder = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
items = null;
folder = null;
}
}
privatevoid ThisAddIn_Shutdown(object sender, System.EventArgs e) {
if (_worker.IsBusy) {
_worker.CancelAsync();
}
}
Note that you cannot use every property to restrict the number of items in a folder.
The Table Object
Starting from Outlook 2007 there is a new Table object in the OOM that provides a fast access to the Folder table. It's fast because you can decide on what columns you want to retrieve on each row. You may get the best performance if you need just information that can be retrieved directly as column in the row of a table. In the code above we used only the properties LastModificationTime and Subject. Let's see if we can speed things a little bit. To measure success, the code is slightly modified to count the number of items processed and the duration of the whole process. The Test using Outlook items has the following result:
Processing 773 items took 11262ms
After modifying the code to use a table object instead, the test read the following values:
Processing 773 items took 15162ms
Funny. All of my tests using Outlook2010 showing that using the Folder.Table object is slower than iterating over the Outlook Items. Theoretically using GetTable() should be much faster than iterating over an Outlook Item collection. But it isn't.
Here is the code:
BackgroundWorker _worker;
Queue<string> _folderIdsToProcess;
privatevoid ThisAddIn_Startup(object sender, System.EventArgs e)
{
_folderIdsToProcess = newQueue<string>();
// Get the Inbox folder
Outlook.MAPIFolder inboxFolder = GetInboxFolder();
// and add it to the queue
_folderIdsToProcess.Enqueue(inboxFolder.EntryID);
// build a filter
DateTime from = DateTime.Now.AddMonths(-6);
DateTime to = DateTime.Now;
string filter = string.Format("[LastModificationTime] > '{0}/{1}/{2}' AND [LastModificationTime] <='{3}/{4}/{5}'"
, from.Year, from.Month, from.Day
, to.Year, to.Month, to.Day);
// Setup the background thread
_worker = newBackgroundWorker();
_worker.DoWork += newDoWorkEventHandler(_worker_DoWork);
_worker.WorkerSupportsCancellation = true;
_worker.RunWorkerAsync(filter);
}
void _worker_DoWork(object sender, DoWorkEventArgs e) {
BackgroundWorker worker = (BackgroundWorker)sender;
string filter = (string)e.Argument;
Outlook.MAPIFolder folder;
int itemsProcessed = 0;
Stopwatch watch = newStopwatch();
watch.Start();
int watchDog = 0;
while (_folderIdsToProcess.Count > 0 && !worker.CancellationPending) {
// pick one folder
string entryId = _folderIdsToProcess.Dequeue();
folder = Globals.ThisAddIn.Application.Session.GetFolderFromID(entryId, Type.Missing);
Outlook.Table table = folder.GetTable(filter, Outlook.OlTableContents.olUserItems);
// try to get the first row
Outlook.Row row = table.GetNextRow ();
// Iterate over all Inbox items and get the Subject
while (!table.EndOfTable ) {
if (worker.CancellationPending) break;
string subject = (string) row["Subject"];
Trace.WriteLine(subject);
itemsProcessed++;
row = table.GetNextRow();
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
for (int index = folder.Folders.Count; index > 0; index--) {
if (worker.CancellationPending) break;
Outlook.MAPIFolder subFolder = folder.Folders[index];
// Call this function recursively
_folderIdsToProcess.Enqueue(subFolder.EntryID);
subFolder = null;
// Ensures that not too many RPC connections are made to a Server
watchDog++;
if (watchDog > 100) {
watchDog = 0;
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
table = null;
folder = null;
}
Trace.WriteLine(string.Format("Processing {0} items took {1}ms", itemsProcessed, watch.ElapsedMilliseconds));
}
privatevoid ThisAddIn_Shutdown(object sender, System.EventArgs e)
{
if (_worker.IsBusy) {
_worker.CancelAsync();
}
}
The next enhancement would be to use two threads that work together. One thread enumerates all folders and adds the items into a Queue, a second thread processes all the Items.
Avoiding to collect the GC
While this is necessary to cleanly release COM objects – forcing the GC can be time consuming and affects all running .Net programs. In the loop we are calling the function to avoid a specific problem when connected online to Exchange. If Outlook is offline there's no need for the workaround.
Here is the optimized code:
privatevoid IncreaseWatchdogAndCallGCIfOnline(refint watchDog) {
watchDog++;
if (watchDog > 100) {
Outlook.OlExchangeConnectionMode mode = Globals.ThisAddIn.Application.Session.ExchangeConnectionMode;
if (mode == Outlook.OlExchangeConnectionMode.olCachedConnectedDrizzle
|| mode == Outlook.OlExchangeConnectionMode.olCachedConnectedFull
|| mode == Outlook.OlExchangeConnectionMode.olCachedConnectedHeaders
|| mode == Outlook.OlExchangeConnectionMode.olOnline) {
// Ensures that not too many RPC connections are made to a Server
GC.Collect();
GC.WaitForPendingFinalizers();
}
watchDog = 0;
}
}
References
Outlook Folder Object: http://msdn.microsoft.com/en-us/library/bb176362.aspx
Outlook Table Object: http://msdn.microsoft.com/en-us/library/bb176406.aspx
.Net Queue Object: http://msdn.microsoft.com/en-us/library/7977ey2c.aspx