Coder Factotum

In a recent development project I had the need to monitor when a specific class of USB devices (usb to serial converters) was plugged in or removed from the target system.  In order to accomplish this, I chose to use WMI in a C# application.  The key requirements of this application were:

1. Select a class of USB devices, identified by a particular vendor ID.  All USB devices have a company specific Vendor-ID, and a product-specific Product-ID.  These values are used to link the hardware device to a particular driver when the device is plugged in (known as the enumeration phase, where the host and the USB device exchange information).  When Windows pops up an "unrecognized device" when a new USB device it is essentially saying that it does not have a registered .inf (driver description file) for that vendor and product ID.
2. Maintain an active, dynamic list of all USB devices of that class.  Keep this list up to date by monitoring all connection/disconnection events for that class of device.
3. Look up meta-information about all connected devices of that class, and be able to produce a list containing the name, deviceId, and other useful information.  The key ability will be to link the USB device with its 'service' - in this case, a virtual serial port.

This article will cover the process I went through in developing this application, from determine where and how to access the WMI information and events, through building the application in VS.NET 2005.  The source code for this application may be downloaded here.

Getting Started - Using Wbemtest

In order to examine the WMI classes and events we need to incorporate into our C# application, we'll make use of the WMI test application wbemtest.exe (included with any Windows PC that has the WMI tools installed.  These tools can be downloaded from here). 

Click on the Connect button to connect to the target.

At this point wbemtest is now connected to the WMI store and ready to execute queries and capture events.

Creating the List -- Building a WMI Query (wbemtest)

Starting from an active wbemtest.exe window, click on Query to bring up the query editor.

In this window, type in the query "SELECT * FROM Win32_PnpDevice".  The core connectivity information for all USB devices is stored under this WMI class.  This generates a large number of results (all "pluggable devices on the system".

Note that the USB devices have a DeviceID of the form "USB\\VID_[vendor ID]&PID_[product ID]&[serial #] - the exact information we need to maintain our device list.  In terms of our C# application this means we need to look for Win32_PnpDevice objects with a DeviceID matching the pattern "USB%".

Creating the List -- Building a WMI Query (Visual Studio.NET)

After going through the process of using wbemtest, I thought to myself.. there's gotta be a better way to figure out / browse the WMI base than this.  It's a very powerful tool, but the user experience isn't very polished.  Visual Studio.NET provides the capability to browse the WMI metabase through the server explorer.  From the server explorer in Visual Studio.NET (View -> Server Explorer), click Add Classes.

From the list under root\CIMV2, add the Plug and Play Devices (Win32_PnpDevices) and the Serial Ports (Win32_SerialPorts) classes to the selected classes list.  Click OK.

These classes are now available for browsing from the server explorer.  We browse through this list until we find a device of interest (in this case the USB CDC serial port emulation device; these names may also be browsed through the Ports section of the Device Manager.).

Clicking on this device, then looking at the Properties window gives us a snapshot of the key information that will be required later on in the C# application.  In this view we can see the DeviceID containing the vendor, product and serial information, as well as the device display name (Caption).

The next step will be to find the "linking" information in the Win32_SerialPort class.  Opening up the serial port tree in the server explorer, we can see the matching record. 

Clicking on this record, then looking at the properties page provides us with the key linking information.  Browsing through the list of properties, we see that the COM port is listed under the DeviceID, and the PNP device ID is (ironically enough) PNPDeviceID. 

These last two pieces of information will allow us to manage a list of virtual COM ports that are plugged into and removed from the system in a C# application.

Maintaining the List -- WMI Notification Events (wbemtest)

Whenever a USB device is successfully plugged into a Windows PC, it generates a WMI __InstanceCreationEvent notification event.  A matching __InstanceDeletionEvent is generated when the device is removed from the system.  In order to maintain a dynamic list of all USB devices connected to the target system we need to receive these events from the WMI system and update our list accordingly.

Click the Connect button on the Connect form to establish a connection to WMI using your current credentials.

Click on the Notification Query button.  This will bring up the query editor window.  We want to capture __InstanceCreationEvent and __InstanceDeletionEvent events, but as the query only allows capturing from a single "table" (or event type) this will need to be done in two separate queries.  For now we'll start with the __InstanceCreationEvent with the query

SELECT * FROM __InstanceCreationEvent WITHIN .5 WHERE TargetInstance ISA 

'Win32_USBControllerdevice'

As seen in the query editor window below.  This query requests all events of type __InstanceCreationEvent, polling every 500 ms.  Click on the Apply button to bring up the query capture window. 

Note: if we try to select all __InstanceCreationEvent objects without targeting those of type Win32_USBControllerdevice, the WMI query will likely fail with a "quote execeeded" error (i.e. the query would attempt to capture too many types of events).  This ISA filter will limit the query to USB devices.

Nothing will appear in this window until an event of the appropriate type is raised.  Plug in a USB device (such as a memory stick) to raise an event to this window.  The window below shows the result of plugging in a USB device (in this case a USB-to-serial converter).

To inspect this event double-click on the __InstanceCreationEvent=<no key> line in the window.  This will bring up the event details window.  This is the real focus of this section - examining the event in the wbemtest application to determine which properties we need to look for in our C# application.

The information we're looking for will be in the TargetInstance property.  Double-click on the TargetInstance property to bring up details on the target instance.

Not too much exiting information here; click on the View Embedded button to bring up the real deal on this event.

Finally :)  All that work to get to this page of information.  By browsing through the various event properties, we see the key pieces of information that we need to tie this event back to the source USB device - the Dependent property.

Dependent = "\\\\MSIMMS-PC\\root\\cimv2:Win32_PnPEntity.DeviceID=\"USB\\\\VID_8765&PID_1111\\\\13245678\"";

This information shows us the USB device ID, vendor ID, product ID, and serial number.  These are the key pieces of information needed to link this event back against our device list.  Armed with knowledge of how to access the list of USB devices via WMI, and how to receive update events we may now proceed with implementing this in C#.

Maintaining the List -- WMI Notification Events (Visual Studio.NET)

Keeping in the same theme as the previous section, Visual Studio also provides a way to look up the management event information in a slightly more user-friendly way than wbemtest.  In the Server Explorer, right click on Management Events to bring up the context menu, and select Add Event Query.

1. From the Build Management Event Query dialog, select Plug and Play Devices under the ROOT\CIMV2 tree.
2. Select Object creation, deletion or modification from the Deliver events for these operations dropdown.
3. Select an Event polling interval of 1 second.
4. Click OK.

There should now be an event query visible under the Management Events node in the Server Explorer.  Click on this query, then look at the properties window.

Here in the query window we can see the WQL query statement (SELECT * FROM __InstanceOperationEvent WITHIN 1 WHERE TargetInstance ISA "Win32_PnPEntity") that we'll later use directly in our C# application to capture events.

With this query running:

1. Remove a USB device from our system
2. Right click the event query, and select Refresh.
3. The query node should now show child objects.  Click the + to open up the node.  There should be two child objects under the node (a deletion event and a modification event).

From this properties window we see one of the shortcomings of capturing events this way in Visual Studio.NET 2005 - there is limited information available (i.e. we'd have to go back to something like wbemtest to dig into the event details).

C# - Building the Device List

I'm not going to delve deeply into the plumbing of the application I developed to manage the device list, but rather focus on the WMI-specific aspects (the full source code may be downloaded here).  The two key aspects related to WMI are building the initial device list and updating the list when change notifications are received.

The code below is the method that builds the initial list of PnpDevices of type USB that have a matching vendor ID.  This is essentially a "plug in the query" exercise based on the WMI classes we investigated in previous sections.

/// <summary>
/// Build up the initial device list of USB devices.
/// </summary>
private void BuildDeviceList()
{
    try
    {
        // Look for class instances within the root\CIMV2 namespace of all types
        // of Win32_PnpEntity.
        ManagementScope scope = new ManagementScope(@"root\CIMV2");
        ObjectQuery query = new ObjectQuery(@"SELECT * FROM Win32_PnpEntity");

        // Build and execute the search synchronously
        ManagementObjectSearcher search = new ManagementObjectSearcher(scope, query);
        ManagementObjectCollection col = search.Get();

        lock (lockObjDevices)
        {
            // Reset the ListView and device list
            this.lvDevices.Clear();
            this.devices.Clear();

            // Hold off on GUI updates until our list update is complete
            this.lvDevices.BeginUpdate();

            // Iterate through the list of devices.  Any devices that match the USB device type
            // and current ID (as stored in a simple text box tbVendorId.Text) will be passed
            // to the UpdateDeviceInformation method.
            foreach (ManagementObject obj in col)
            {
                string deviceId = obj["DeviceID"].ToString();
                if (deviceId.StartsWith(String.Format("USB\\VID_{0}", this.tbVendorId.Text)))
                {
                    UpdateDeviceInformation(deviceId);
                    AppendTextToScreen(String.Format("Got device id {0}", deviceId),
                        EventArgs.Empty);
                }
            }

            this.lvDevices.EndUpdate();

        }

        // Dispose of the WMI search objects
        search.Dispose();
        col.Dispose();
    }
    catch (Exception ex0)
    {
        AppendTextToScreen(ex0.ToString(), EventArgs.Empty);
    }    
}

The real magic happens in the UpdateDeviceInformation and UpdatePortInformation methods, wherein we dig into the class information, and correlate the PNP device registration against any serial port information.

/// <summary>
/// Delegate object for invoking management events onto the foreground thread.
/// </summary>
/// <param name="deviceId"></param>
private delegate void UpdateDeviceHandler(string deviceId);

/// <summary>
/// Given a device Id update the device information dictionary
/// and display listview.
/// </summary>
/// <param name="deviceId"></param>
private void UpdateDeviceInformation(string deviceId)
{
    // Management events arrive on a background thread.  Since this method
    // updates a ListView, make sure it's invoked to a foreground thread.
    if (this.InvokeRequired)
    {
        this.Invoke(new UpdateDeviceHandler(UpdateDeviceInformation), deviceId);
    }
    else
    {
        // If this is a "known" device, update only the port status information.
        if (devices.ContainsKey(deviceId))
        {
            // Update information about "known" device.  
            UpdatePortInformation(devices[deviceId]);
        }
        else
        {
            // Add information about new device
            DeviceInformation di = new DeviceInformation();
            di.DeviceId = deviceId;

            // Device ID is always of
            // the form
            //
            // USB\VID_0000&PID_1234&[serial]
            di.VendorId = deviceId.Substring(8, 4);
            di.ProductId = deviceId.Substring(17, 4);
            di.Serial = deviceId.Substring(22);

            // This will be filled in by the UpdatePortInformation method
            di.Information = "TODO";

            // Fill in the list view information
            ListViewItem lvi = new ListViewItem();
            lvi.Text = di.Active.ToString();
            lvi.SubItems.Add(di.DeviceId);
            lvi.SubItems.Add(di.VendorId);
            lvi.SubItems.Add(di.ProductId);
            lvi.SubItems.Add(di.Serial);
            lvi.SubItems.Add(di.Information);
            lvi.Tag = di;

            di.display = lvi;
            this.lvDevices.Items.Add(di.display);
            this.devices.Add(deviceId, di);

            // Get any related serial port information
            UpdatePortInformation(di);

        }
    }
}

This is pretty standard stuff; simply converting the DeviceID into the vendor, product and serial information components.  The last piece of WMI information we need is linking this device information against any possible registered serial ports.  This is done in the UpdatePortInformation method:

/// <summary>
/// Given a device information object, check to see if it has any correlated serial
/// ports registered on the system.  The two items will be correlated by 
/// PNPDevice.DeviceId = SerialPort.PNPDeviceID.
/// </summary>
/// <param name="di"></param>
private void UpdatePortInformation(DeviceInformation di)
{            
    try
    {
        // Note: need to escape the "\" characters in the deviceId, or the query will fail.
        string queryStr = String.Format(@"SELECT * FROM Win32_SerialPort WHERE PNPDeviceID = '{0}'", 
            di.DeviceId.Replace(@"\", @"\\"));

        ManagementScope scope = new ManagementScope(@"root\CIMV2");
        ObjectQuery query = new ObjectQuery("WQL", queryStr);
        
        ManagementObjectSearcher search = new ManagementObjectSearcher(scope, query);
        ManagementObjectCollection col = search.Get();

        // If we don't find any items, we can assume that this deviceId has no registered serial ports.
        // Update the list view as appropriate.
        if (col.Count == 0)
        {
            AppendTextToScreen("Device with id " + di.DeviceId + " is not a registered serial port", EventArgs.Empty);
            di.Information = "";
            di.Active = false;

            di.display.SubItems[5].Text = di.Information;
            di.display.Text = "Inactive";
            this.lvDevices.Update();
            return;
        }

        // Otherwise, grab the COM port (the DeviceID).
        foreach (ManagementObject obj in col)
        {
            string deviceId = obj["DeviceID"].ToString();
            di.Information = deviceId;
            di.Active = true;

            di.Active = true;
            di.display.SubItems[5].Text = di.Information;
            di.display.Text = "Active";

            this.lvDevices.Update();
        }

        search.Dispose();
        col.Dispose();
    }
    catch (Exception ex0)
    {
        AppendTextToScreen(ex0.ToString(), EventArgs.Empty);
    }            
}

Running this application (where BuildDeviceList is called in the Load method) will produce (assuming that you have a device with a vendor ID of 0x8765:

C# - Capturing Events

Now that we have the initial list of devices built, it's necessary to keep it up to date when a device is plugged into or removed from the system.  Based on the WMI event information captured using the Server Explorer and wbemtest, we can go ahead and capture the necessary events in our C# application.

/// <summary>
/// Add a capture handler to the application to receive operation events 
/// (insert/delete) on USB controller devices (more specific than PNPDevice events).
/// </summary>
public void Add_UsbHandler()
{
    WqlEventQuery q;
    ManagementScope scope = new ManagementScope(@"root\CIMV2");
    scope.Options.EnablePrivileges = true;

    try
    {
        // Build up a query (essentially SELECT * FROM __InstanceOperationEvent WITHIN .5 WHERE 
        // TargetInstance ISA 'Win32_USBControllerdevice'
        q = new WqlEventQuery();
        q.EventClassName = "__InstanceOperationEvent";
        q.Condition = @"TargetInstance ISA 'Win32_USBControllerdevice'";
        q.WithinInterval = new TimeSpan(0, 0, 0, 0, 500);

        // Create a management event watcher based on this query, bound to the 
        // mw_eventArrived method
        mw_insert = new ManagementEventWatcher(scope, q);
        mw_insert.EventArrived += new EventArrivedEventHandler(mw_EventArrived);
        mw_insert.Start();
    }
    catch (Exception e)
    {
        this.rtbStatus.AppendText(e.ToString());
        if (mw_insert != null)
            mw_insert.Stop();
    }
}

This will allow us to receive management events into the mw_EventArrived method whenever a USB device is plugged into or removed from the system.

/// <summary>
/// Update the device list when a USB device is plugged into or removed from the
/// system.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
void mw_EventArrived(object sender, EventArrivedEventArgs e)
{
    AppendTextToScreen("USB device connectivity change", EventArgs.Empty);

    
    try
    {
        // Grab the target instance from the management event
        ManagementBaseObject obj = e.NewEvent;
        PropertyData o = obj.Properties["TargetInstance"] as PropertyData;
        if (o != null)
        {
            ManagementBaseObject mgtObj = o.Value as ManagementBaseObject;
            if (mgtObj != null)
            {
                // The key information is stored in the dependent field of the event
                string dependent = mgtObj.Properties["Dependent"].Value.ToString();
                string deviceId = dependent.Substring(dependent.IndexOf("USB"));    
                
                // Remove the double '\\' characters
                deviceId = deviceId.Replace(@"\\", @"\");
                deviceId = deviceId.Replace(@"\\", @"\");

                // Trim any trailing \ characters
                deviceId = deviceId.TrimEnd(new char[] { '\\', '\"' });
                UpdateDeviceInformation(deviceId);                        
            }
        }
    }
    catch (Exception ex0)
    {
        Debug.WriteLine(ex0.ToString());
    }
}

Since we already have a method (UpdateDeviceInformation) for updating the information associated with a device, our application is now essentially complete. 

Note: do NOT forget to dispose of the ManagementEventWatcher before closing out your application.

Conclusion

While WMI can be a little overwhelming to the novice (caveat: this is my first real usage of WMI events :), it's a very powerful tool for observing and managing virtually every aspect of a Windows-based computer.  With a minimal amount of code, I can have automatic updates of when USB devices from a particular vendor are changed on a target system, and map that to the specific capabilities of that device (in this case, serial ports).

This is a bit of a departure from the usual RFID centric madness; lately I've been working on improving our build automation and output publishing.  One aspect of that is introducing Sharepoint as the destination for published output from both a continuous and a "daily" (well, bi-weekly) build.

The source code project referenced in this article may be found here; this requires Visual Studio.NET 2008 and the CSharpOptsParse library.

Overview

The build system, as managed by CruiseControl.NET performs the following tasks:

1. When triggered (either manually, via a Subversion commit, or scheduled build) activate and produce deterministic output by:
2. Perform an SVN update to obtain the latest copy of the repository (note: this will eventually change to a full checkout to remove the chance of any leftover build artifacts creeping into the mix).
3. Call the appropriate build scripts (batch files, which call into either MSBuild or VBS scripts) to produce the output.
4. Each build script performs the clean, identify, make, verify and publish steps:
   1. Clean removes any left-over build artifacts from a previous build.
   2. The identify step updates all of the version information throughout the source tree with the master version and build number.
   3. Make invokes the requisite compilers and other build tools to produce the final output (a set of firmware images, software installers, documentation, etc)
   4. Verify validates certain aspects of the output (virus scanning, checksums, signatures, etc)
   5. Publish copies the build outputs to their staging areas.  This is the step this article will discuss in the context of using a sharepoint document library as the staging area.

Why Sharepoint?

As an organization, we make a lot of use of Sharepoint as a central hub of information.  It provides excellent support for document sharing, with the first built-in versioning system I've ever seen used by a non-developer.  In order to allow easy access to the build output, I wanted to be able to automatically publish build output to sharepoint document library along with the potential to store associated metadata (aka, build notes).

In order to do this, I had a number of different options:

1. Use the Sharepoint API in conjunction with a WCF web service to provide a service endpoint for pushing files into a specified Sharepoint library.  This would have worked well (the sharepoint developer libraries being very full featured, strongly typed and fairly well documented), but for two issues.  The first being that the development libs and tools only run on Windows Server 2003 (or 2008), and I wasn't keen on the logistical issues around getting the appropriate development environment set up for a one-off.  The other issue was that I wanted to have a minimal installation footprint (i.e. not having the build environment dependent on a custom piece of s/w installed remotely - I wanted it to work with a stock Sharepoint web).
2. Use the Sharepoint web services interface.  I initially looked at this approach, but decided against it due to the heavy lifting required.  The WSS web services are document-based and very loosely typed.  It's up to the developer to provide all of the necessary XML twiddling, and it didn't seem like a wise investment of time for a (somewhat optional) component of the build process.
3. Use the Front Page Server Extensions interface into Sharepoint.  This is the approach I ended up taking, primarily because it was well documented, had available samples, and didn't require any customization on the server (other than configuring permissions).

The Environment

1. Sharepoint Server Web
2. Document Library
3. Build Machine
4. Permissions

Postsp - The Post to Sharepoint Web Utility

In order to automate this posting process, I decided to whip up a quick command line utility in C#.  Its purpose would be to take a filename and destination sharepoint URL, and perform the necessary steps to copy the file to Sharepoint.  This consisted of three main pieces, options, directory creation and file copy as described in the following sections.

This application shamelessly borrowed from three sources, which deserve all credit here.  All I did was change the wiring a little bit.  The majority of the Sharepoint / FPSE code was borrowed from Geert Baeke, with modifications inspired by Hubkey for some of the command handling and metadata parsing.  For command line parsing I used the excellent, if lightly documented CSharpOptParselibrary.

Postsp - Using CSharpOpt to parse command line options

I wanted to make this as flexible a tool as possible (keeping in mind this is the first run through this tool, and I will likely make the options a little more coherent in future :), so the application needed to support the following options:

• Destination URL.  The full sharepoint URL to the destination file, such as http://sharepoint/web/doclib/build_number/outputfile.zip.  This had to account for both sub-webs and sub-directories.  Each automated build generates a unique build number (aka the Subversion repository number), which is used to generate a subdirectory in the destination library.  This allows all of the artifacts related to that build (firmware, software, test documents, etc) to be easily referenced from a common location.
• Input File.  The file to be uploaded to the sharepoint library.  I decided to keep things simple and scriptable (i.e. to copy multiple files, call the postsp application multiple times).
• Version Notes.  The destination web directory has a custom metadata column defined, to be used to write simple notes about each build.  As this is a required column, without this field defined the documents cannot be successfully checked in (and thus usable by others).
• General Metadata.  While I didn't have to deal with metadata beyond the Version Notes, I wanted to make it possible to extend additional metadata without modifying the application.
• Verbosity and Logging.  As with any application, the ability to turn on logging or verbose output for run-time diagnostics is invaluable.
• Subdirectory.  Not originally on my list, but due to some technical weirdness encountered with the FPSE calls (as described below), I needed to add an additional parameter to indicate the subdirectory to create (aka, too lazy to parse out the correct value.  This will likely change in a future version).

CSharpOptParse supports two methods of argument parsing: Dictionary and Object based.  In the former, the parser returns a Dictionary<string,object> type that contains the parsed command line arguments.  I chose to use the latter, wherein one defines an attributed class and the parser fills in the appropriate properties.  This class was written as:

class Properties 
    {
        [OptDef(OptValType.ValueReq)]
        [LongOptionName("url")]
        [Description("The destination sharepoint URL (including file name).")]
        [EditorBrowsable(EditorBrowsableState.Always)]
        public string DestUrl { get; set; }

        [OptDef(OptValType.ValueReq)]
        [LongOptionName("input")]
        [ShortOptionName('i')]
        [Description("The file to be uploaded to the sharepoint repository")]
        public string FileName { get; set; }

        [OptDef(OptValType.ValueOpt)]
        [DefaultValue("Version Notes:")]
        [LongOptionName("versionNotes")]        
        [Description("Version notes for the uploaded file")]
        public string VersionNotes { get; set; }

        [OptDef(OptValType.ValueOpt)]        
        [LongOptionName("metadata")]
        [Description("Pre-encoded metadata")]
        public string Metadata { get; set; }

        [OptDef(OptValType.Flag)]
        [ShortOptionName('v')]
        [LongOptionName("verbose")]
        [Description("Enable verbose output (debugging)")]
        [DefaultValue(false)]
        public bool Verbose { get; set; }

        [OptDef(OptValType.ValueOpt)]
        [LongOptionName("subdir")]
        [Description("Subdirectory for the file")]
        public string Subdirectory { get; set; }

    }

Essentially, the attributes inform the parser as to which attributes to look for, and where to put the stored values.  Couple of notes; there seems to be either an issue with the CSharpOptParse libs, or my usage of them, as the OptValType.ValueReq flag seems to have been ignored.  The parser also does not make use of the DefaultValue attribute (that was perhaps wishful thinking on my part :).  To see what this looks like from the end-user perspective, we make use of the really impressive built-in usage builder, via:

UsageBuilder usage = new UsageBuilder(); 
usage.GroupOptionsByCategory = false; 
usage.BeginSection("Name"); 
usage.AddParagraph(String.Format("postsp.exe\t[Version {0}]", 
    System.Reflection.Assembly.GetExecutingAssembly().GetName().Version.ToString(4))); 
usage.EndSection(); 

usage.BeginSection("Synopsis"); 
usage.AddParagraph("postsp.exe [options] [arguments]"); 
usage.EndSection(); 

usage.BeginSection("Description"); 
usage.AddParagraph("Application to post files to sharepoint libraries"); 

// Display options                       
usage.BeginSection("Options"); 
usage.AddOptions(parser);                         
usage.EndSection(); 
                      
usage.ToText(Console.Out, OptStyle.Unix, true);

Which produces the following output at the command line:

Usage:
Name

    postsp.exe  [Version 1.0.0.34017]

Synopsis

    postsp.exe [options] [arguments]

Description

    Application to post files to sharepoint libraries


    Options

        --url, --DestUrl
            (Type: Value required, Value Type:[String])
            The destination sharepoint URL (including file name).  The current
                user must have write privledges to this directory.

        --metadata, --Metadata
            (Type: Value optional, Value Type:[String])
            Pre-encoded metadata

        --versionNotes, --VersionNotes
            (Type: Value optional, Value Type:[String], Default Value:
                'Version Notes:')
            Version notes for the uploaded file

        --subdir, --Subdirectory
            (Type: Value optional, Value Type:[String])
            Subdirectory for the file

        -i, --input, --FileName
            (Type: Value required, Value Type:[String])
            The file to be uploaded to the sharepoint repository

        -v, --verbose, --Verbose
            (Type: Flag, Value Type:[Boolean], Default Value: 'False')
            Enable verbose output (debugging)

An example of using the application would:

 Example

       For example, to use this application to copy the local file test.doc
       to the sharepoint web http://sharepoint/myweb/, under the mydocs
       document libraryin the subdirectory 1234, use these options:

       postsp.exe --input="test.doc" -
       -url="http://sharepoint/myweb/mydocs/1234/test.doc"      -
       -subdir="mydocs/1234"

Once the properties class has been created and attributed, the heavy lifting for parsing the data is essentially non-existent; create a parser via the ParserFactory and call the Parse method as per:

// Parse the command line input
Properties myProps = new Properties();
PropertyFieldParserHelper pf = new PropertyFieldParserHelper(myProps);
Parser myParser = ParserFactory.BuildParser(pf);
myParser.OptionWarning += new WarningEventHandler(myParser_OptionWarning);

// Parse the input
string[] results = myParser.Parse(OptStyle.Unix, UnixShortOption.ShortSeparated, 
    DupOptHandleType.Error, UnknownOptHandleType.Error, false, 
    System.Environment.GetCommandLineArgs());

Note the use of Unix-style (-s, --short) versus Windows-style (/s /short:) argument formats both in the usage and in the call to the parsing method.  It's mostly a personal preference (I spent most of my early technical career on SunOS and Linux, so the preference has stuck.  My love of vi is now, however, far behind me.  Visual Studio.NET ftw :).

After the call to myParser.Parse(), the myProps object is populated with whatever command line options were passed through.  Since the required value attribute didn't seem to be throwing exceptions for missing values, I added some additional value checking, using the ever popular String.IsEmptyOrNull method. 

Since this application was meant to be called from shell.. er.. batch scripts I needed to remember to set the System.Environment.ExitCode before exiting the application.  Once the arguments have been processed, it's now time to create directories and copy documents to the sharepoint library.

Postsp - Accessing Sharepoint through FPSE

Note: this was definitely a first run through this type of application.  I don't do a whole lot of error checking, such as verifying the absence of directories before creating them, etc.  When I get time to polish this up a bit, I'll look at adding the extra FPSE functions.

These web methods are documented in MSDN on the FrontPage Server Extensions RPC Methods page.  The key web method for creating a directory is either the create url-directories method, or the deprecated create url-directory method, which I used.  The web method for copying a file is the put document method.  The sections below go into detail as to the specific parameters and result codes for each of these methods - this section will focus on the common bits required to invoke them.

Starting from the base destination web URL, the first step was to obtain a URI to the actual sharepoint web.  This was accomplished by invoking another FPSE method, url to web url.  By invoking this command against the root sharepoint web's /_vti_bin/shtml.dll library, we can obtain the command URI for the web in question (shamelessly cribbed from Geert Baeke excellent posting) :

public void UrlToWebUrl(string uri, out string webUrl, out string fileUrl)
{
    Uri myUri = new Uri(uri);

    string postBody = String.Format("method=url+to+web+url&url={0}&flags=0", myUri.AbsolutePath);
    string response = SendRequest(myUri.GetLeftPart(UriPartial.Authority) + "/_vti_bin/shtml.dll/_vti_rpc", postBody);

    webUrl = GetReturnValue(ref response, "webUrl");
    fileUrl = GetReturnValue(ref response, "fileUrl");
}

For example, starting with a final web URL of http://sharepoint/web/doclib/build_number/outputfile.zip, this method would produce a webUrl of TODO and a fileUrl of TODO.

The SendRequest method performs an HTTP POST against the URI in questino with the attached data (postBody), and returns the result.

private string SendRequest(string uri, byte[] postBody, long postLength)
{
    string responseText = null;

    try
    {
        WebRequest request = WebRequest.Create(uri);
        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        request.Headers.Add("X-Vermeer-Content-Type", "application/x-www-form-urlencoded");
        request.Credentials = CredentialCache.DefaultCredentials;
        Stream newStream = request.GetRequestStream();

        int offset = 0;
        while (postLength > 0)
        {
            if (postLength > 4096)
            {
                newStream.Write(postBody, offset, 4096);
                postLength -= 4096;
                offset += 4096;
            }
            else
            {
                newStream.Write(postBody, offset, Convert.ToInt32(postLength));
                break;
            }
        }

        newStream.Close();

        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        Stream receiveStream = response.GetResponseStream();
        StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);

        responseText = readStream.ReadToEnd();
        
        response.Close();
        readStream.Close();
    }
    catch (Exception e)
    {
        Console.WriteLine("ERROR: {0}", e.Message);
        throw;
    }

    return responseText;
}

Note the use of the default credentials; if I'd wanted to make this more configurable, I would have added  username and password options to the application and used them to build a credentials object.  

Once we've obtained the webUrl and fileUrl values, we're ready to invoke the create directory and put document methods. 

Postsp - Creating Directories in a Sharepoint Document Library

The put document method, described in the next section, is supposed to have an option called createdir which instructs FPSE to create any necessary directories.  A nice feature which did absolutely nothing for me :)  Not sure if I was calling things properly, but I couldn't get the put document to work without pre-creating the directory path.  When the documentation says one thing, empirical evidence says another, and it's a one-off, go with the empirical evidence.  In this case, pre-creating the directory structure.

public void CreateDirectory(string uri, string directoryName, bool verbose)
{
    Uri myUri = new Uri(uri);
    string webUrl, fileUrl;
    UrlToWebUrl(uri, out webUrl, out fileUrl);

    string postBody = String.Format(
      "method=create url-directory&service_name=/&url={0}\n",
      HttpUtility.UrlEncode(directoryName));

    if (verbose)
    {
        Console.WriteLine("Request:\r\n-----------------------------------------\r\n");
        Console.Write(postBody);
    }

    ASCIIEncoding encoding = new ASCIIEncoding();
    MemoryStream stream = new MemoryStream();
    stream.Write(encoding.GetBytes(postBody), 0, postBody.Length);

    string response = SendRequest(myUri.GetLeftPart(UriPartial.Authority) + 
        webUrl + "/_vti_bin/_vti_aut/author.dll",
        stream.GetBuffer(), stream.Length);
    stream.Close();

    if (verbose)
    {
        Console.WriteLine("Response:\r\n-----------------------------------\r\n");
        Console.WriteLine(response);
    }
}

In this method, we use the similar SendRequest method to send a crafted HTTP PUT request to the author.dll FPSE library using the create url-directory method.  The URL is the full path (from the web root) of the directory to be created, in this case doclib/build_number.  Given the request with a subdirectory of "doclib/4070" we send:

method=create url-directory&service_name=/&url=doclib%2f4070

We observe the successful response:

<html><head><title>vermeer RPC packet</title></head>
<body>
<p>method=create url-directory:12.0.0.6219
<p>message=successfully created URL-directory 'doclib/4070'
<p>urldir=
<ul>
<li>url=doclib/4070
<li>meta_info=
<ul>
<li>vti_isexecutable
<li>BR|false
<li>vti_isbrowsable
<li>BR|true
<li>vti_isscriptable
<li>BR|false
</ul>
</ul>
</body>
</html>

The next version of this application will add response parsing code to check for error conditions.

Postsp - Copying Files to a Sharepoint Document Library

With the directory structure created, we're now ready to copy files into our destination document library.  Using the put document we craft the request with this code snippet:

public void PutDocument(string uri, string fileName, string metaInfo, bool verbose)
{
    Uri myUri = new Uri(uri);
    string webUrl, fileUrl;
    UrlToWebUrl(uri, out webUrl, out fileUrl);

    if (null == metaInfo)
        metaInfo = "";

    if (!File.Exists(fileName))
        throw new Exception("Could not find file" + fileName);

    string postBody = String.Format(
        "method=put+document&service_name=&document=[document_name={0};meta_info=[{1}]]" +
        "&put_option=overwrite,createdir&comment=&keep_checked_out=false\n",
        HttpUtility.UrlEncode(fileUrl),
        metaInfo);

    if (verbose)
    {
        Console.WriteLine("Request:\r\n-----------------------------------------\r\n");
        Console.Write(postBody);
    }

    ASCIIEncoding encoding = new ASCIIEncoding();
    MemoryStream stream = new MemoryStream();
    stream.Write(encoding.GetBytes(postBody), 0, postBody.Length);

    FileStream fs = File.OpenRead(fileName);
    byte[] b = new byte[4096];
    while (fs.Read(b, 0, b.Length) > 0)
    {
        stream.Write(b, 0, b.Length);
    }
    fs.Close();

    string response = SendRequest(myUri.GetLeftPart(UriPartial.Authority) + webUrl + 
        "/_vti_bin/_vti_aut/author.dll", 
        stream.GetBuffer(), stream.Length);
    stream.Close();

    if (verbose)
    {
        Console.WriteLine("Response:\r\n-----------------------------------\r\n");
        Console.WriteLine(response);
    }
}

We invoke this command in a virtually identical fashion to that of the create directory method, save for the different method name and attached body data.  The important parameters in the create document method are:

• put_option; either edit (which does not allow an overwrite of an existing file, strangely enough) or overwrite with createdir (which didn't appear to work for me).
• document.  The document name in the destination repository.
• keep_checked_out.  Whether or not the keep the file checked out after copy.  This is set to false such that the file is visible by others immediately after this operation is complete.

Full documentation for the options is linked from the put document documentation.  Given a document name of doclib/4070/test.doc, we request:

method=put+document&service_name=&document=[document_name=doclib%2f4070%2ftest.doc;

meta_info=[]]&put_option=overwrite,createdir&comment=&keep_checked_out=false

and observe in response:

<html><head><title>vermeer RPC packet</title></head>
<body>
<p>method=put document:12.0.0.6219
<p>message=successfully put document doclib/4070/test.doc' as 'doclib/4070/test.doc'
<p>document=
<ul>
<li>document_name=doclib/4070/test.doc
<li>meta_info=
<ul>
<li>vti_rtag
<li>SW|rt:327A51C6-6734-41C5-A6B0-AFC6B475F196@00000000001
<li>vti_etag
<li>SW|&#34;&#123;327A51C6-6734-41C5-A6B0-AFC6B475F196&#125;,1&#34;
<li>vti_filesize
<li>IR|270336
<li>vti_parserversion
<li>SR|12.0.0.6219
<li>vti_modifiedby
<li>SR|TEST&#92;msimms
<li>vti_timecreated
<li>TR|07 Apr 2008 23:09:54 -0000
<li>ContentTypeId
<li>SW|0x010100D232599AAB180B4B8631F1B010EE1978
<li>vti_timelastmodified
<li>TR|07 Apr 2008 23:09:54 -0000
<li>vti_author
<li>SR|TEST&#92;msimms
<li>vti_sourcecontrolversion
<li>SR|V1.0
<li>vti_sourcecontrolcookie
<li>SR|fp_internal
</ul>
</ul>
</body>
</html>

As mentioned in the previous section, this sample isn't really complete without error checking on the response strings.  I'll look at adding that in the next week or so, depending on how my other writing project are going.

Summary

Sharepoint is cool.  Sharepoint is even cooler if you can automate posting content into it remotely without requiring lots of heavy lifting parsing XML, or needing to install Win2k3 and write service proxies.  While it's not the most elegant way of doing things, it works surprisingly well.  I've attached the source code for anybody that wants to give this a shot.

All of the source code for this article is available here as a Visual Studio.NET 2008 solution.  I highly recommend you have the solution open while you read through this article.  Things will likely make a lot more sense.

Continuing from the previous article, this article will address how to simulate tag events from a .NET application using the AddEventToProcessPipeline method from the ManagementWebProxies library.

This application will:

1. Perform a dynamic lookup to retrieve a list of available processes.
2. Use that process name to obtain the list of registered devices.
3. Allow the user to select from a set of pre-registered tags, and assign them to either a TagReadEvent or TagListEvent.
4. Generate either a single event, or sequence of events to be injected into the selected process.

Creating a Baseline Application

The first step will be to create an application baseline that references the necessary Biztalk RFID assemblies, and a simple UI.

We create a simple UI in VS.NET 2008, with a status bar, combo boxes for the process and reader names, and a DataGridView for visualizing the tag data.

This article won't talk much about the plumbing that went into the application (standard WinForms application, with some BackgroundWorker's for a better user experience.. UTSL).  The basics of the application are:

• Choose a process and logical device registered in the Biztalk RFID server (i.e. perform a live lookup against metadata from the server)
• Optionally select a subset of registered tags to be sent to Biztalk RFID.
• Send tag events either as individual TagReadEvent or grouped TagListEvent messages. 
• Control the timing of how tags are sent to the Biztalk RFID process.
• Optionally, simulate event attendance in the context of a tag only being scanned once (not overly realistic, but a useful starting approximation).

Next, we add references to the following assemblies; this will grant access to the management web proxies and data structures used to access the Biztalk RFID server.

1. Microsoft.Rfid.ManagementWebProxies
2. Microsoft.Rfid.Design
3. Microsoft.Rfid.SpiSdk

and use these namespaces:

1. Microsoft.SensorServices.Rfid.Management
2. Microsoft.SensorServices.Rfid.Design
3. Microsoft.SensorServices.Rfid
4. System.IO.SensorServices.Rfid.Client
5. Microsoft.SensorServices.Rfid.Utilities

Obtaining a list of Processes

To obtain a list of processes, create an instance of the ProcessManagerProxy object.  Use the GetAllProcesses method to retrieve a list of all of the registered processes in the system.  This list is then assigned to the combo box.

ProcessManagerProxy = new ProcessManagerProxy();

string[] procList = process.GetAllProcesses();

This code runs in a background worker component, triggered by Form_Load (to prevent slow startup times while instantiating the connect to Biztalk RFID).  Once the background worker is complete the combo box is filled in:

if (procList != null && procList.Length > 0)
{
    this.cbProcess.Items.AddRange(procList);
    this.cbProcess.SelectedIndex = 0;
}

Obtaining a list of Readers

A dictionary mapping process names to an array of reader names is also built in the background worker:

readerList = new Dictionary<string, List<string>>();

foreach (string proc in procList)
{
    List<string> list = new List<string>();

    RfidProcess processInfo = managerProxy.GetProcess(proc);
    ICollection<LogicalDevice> devices = processInfo.GetAllLogicalDevices();
    foreach (LogicalDevice d in devices)
        list.Add(d.Name);
    readerList.Add(proc, list);
}           

The SelectedIndexChanged event for the process combo box is bound to a method that automatically populates the reader dropdown with the appropriate information:

string processName = this.cbProcess.SelectedItem.ToString();
this.cbReader.Items.Clear();
if (readerList != null && readerList.ContainsKey(processName))
{
    this.cbReader.Items.AddRange(readerList[processName].ToArray());

    if (this.cbReader.Items.Count > 0)
        this.cbReader.SelectedIndex = 0;
}

Loading the tag metadata

Since our goal is to simulate a closed-loop system with registered/known tags, we make use of a preloaded metadata store (in this case a simple SQL Compact database).  The schema for the database is:

We also incorporate a "database initialization" function into the application, allowing the database to be loaded with a basic tag list.  This database initialization code (called from a menu item under the Tools menu) performs the following operations:

1. Checks for the presence of the Tags table in the database
2. Clears any extant tags from the table
3. Adds the desired number of tags using a selected format.

Loading the database is accomplished through the usual means (TableAdapter object into a strongly typed data set). 

Selecting the Event Parameters

Our simple UI provides a way for the end-user to shape the RFID events that will be injected into the selected process.  Upon clicking the Go button, these parameters will determine the behaviour of the BackgroundWorker process that posts the events to the Biztalk RFID server.  The options are:

• Selected or random tag IDs, as retrieved from the metadata store.
• TagReadEvent or TagListEvent
• Number of events
• Tags per event (only applicable to TagListEvent)
• Raise events every N seconds (how often to post events to Biztalk RFID).
• Simulate event attendance.

The last item deserves a little more explanation.  As our intent is to (partially) simulate event attendance, there needs to be some associated logic when selecting tags.  Essentially, if this option is enabled, a given tag ID will only be present in a single raised event. 

Injecting the Events

Once the run parameters have been selected, the user would hit the Go button.  This triggers our BackgroundWorker process to start posting events.

In a timed loop, the worker will post events to the Biztalk RFID server via the AddEventToProcessPipeline method.  Each event (be it TagListEvent or TagReadEvent) selects tag IDs and data from the metadata store, as well as the current local computer time.  All of the form parameters are passed to the Backgroundworker component via this structure:

struct RunOptions
{
    public bool useTagReadEvent;
    public bool useAttendanceMode;
    public int tagCountToSend;
    public int tagsPerEvent;

    public string process;
    public string logicalDevice;
    public int interval;

    public System.Collections.IList data;
}

The first primary action in the worker thread (aside from some basic input and parameter validation) is the conversion of the byte strings passed in from the dataset into a byte[] format:

// Create the set of "sendable" tags in binary format            
List<TagData> tags = new List<TagData>();
foreach (DataGridViewRow r in opts.data)
{
    SampleTagsDataSet.TagsRow row = (r.DataBoundItem as DataRowView).Row 
        as SampleTagsDataSet.TagsRow;

    TagData td = new TagData();
    td.Id = HexUtils.HexDecode(row.Id);
    td.Data = HexUtils.HexDecode(row.Data);
    tags.Add(td);
}

Note; in our application the data is passed to the worker thread as a DataGridViewRow IList.  Once this process is complete we have a List<TagData> object containing all of the available tag data for this run (if the user selected random the entire data set was passed in, otherwise, the selected rows were passed in.  It's a bit hackish, but as the focus of the article is passing tag data to Biztalk RFID, I felt it was "good enough").

We set up some basic state variables:

int tagsRemaining = opts.tagCountToSend;
int tagsAvailable = tags.Count;
int tagsPerRun = opts.tagsPerEvent;
double runCount = 1;
double totalRuns = tagsRemaining / tagsPerRun;

Which will be used to control the flow of data to Biztalk RFID. In a loop we until a user-triggered cancellation, or we're out of tags to send.  Depending on whether or not we're in attendance mode, tags selected for inclusion in the event are chosen slightly differently in the RetrieveTags method:

if (useAttendanceMode)
{
    for (int i = 0; i < count; i++)
    {
        int randomIndex = rand.Next(data.Count);
        tagsToReturn.Add(data[randomIndex]);
        data.RemoveAt(randomIndex);

        log.Info("Adding tag {0} data {1} to list (removing from run)",
          data[randomIndex].Id, data[randomIndex].Data);
    }
}

In attendance mode, as we randomly select a tag from the list, we remove it from the master list of tag data for this run.  When not in attendance mode we create a list of indices, and as each tag is randomly selected it is removed from the (locally scoped) list of indices:

List<int> availableIndices = new List<int>();
for (int i = 0; i < data.Count; i++)
    availableIndices.AddRange(i);

// Otherwise, iterate through the input list and randomly pick out a set
// of tags.  Note: this isn't close to optimized.
for (int i = 0; i < count; i++)
{
    randomIndex = rand.Next(availableIndices.Count);
    tagsToReturn.Add(data[randomIndex]);
    availableIndices.RemoveAt(randomIndex);
    
    log.Info("Adding tag {0} data {1} to list",
        data[randomIndex].Id, data[randomIndex].Data);
}

The first spin of this used a comparison against a growing list of used indices, but the performance would suffer greatly (in a non-deterministic fashion) when getting towards the end of a large data set (i.e. the statistical probability of getting the last tag when number of tags to retrieve = number of tags available would become very very small).

After the tags to send have been retrieved, we get to the real meat of the application, creating and sending the events.  A TagReadEvent is created as:

TagData myTag = data[0];
TagReadEvent tre = new TagReadEvent(myTag.Id, TagType.Iso15693, myTag.Data,
    opts.logicalDevice, DateTime.Now, null, TagDataSelector.All);
baseEvent = tre;

Where data is the array of TagData objects returned from the RetrieveTags method described above.  Crafting a TagListEvent is similar, with the caveat that the SOURCE property for the TagListEvent has to match the SOURCE property of the TagReadEvent which it encloses.

List<TagReadEvent> readEvents = new List<TagReadEvent>();

foreach (TagData d in data)
{
    // NOTE: the source of the tag read events has to be the same as the
    // source for the tag list event
    TagReadEvent tre = new TagReadEvent(d.Id, TagType.Iso15693, d.Data,
        "SOURCE", DateTime.Now, null, TagDataSelector.All);
    readEvents.Add(tre);
}
TagListEvent tle = new TagListEvent(readEvents, "SOURCE");
tle.DeviceName = opts.logicalDevice;
baseEvent = tle;

Finally, the easy part - posting the event to the Biztalk RFID process:

managerProxy.AddEventToProcessPipeline(opts.process, 
    baseEvent, opts.logicalDevice);

Summary

The AddEventToProcessPipeline feature in Biztalk RFID provides a rich method of testing your applications and end-to-end scenarios, either in the context of not having sufficient hardware (readers and tags) available for scale testing, or testing of specific scenarios without the physical and logistical overhead.  While it does not provide the ability to simulate the behaviour of readers (hence why I'm working on the RfidSimulator.NET app to do exactly that :), it is an excellent starting point for examining the behaviour of your Biztalk RFID system in a variety of scenarios.

All of the sample code referenced in this article may be downloaded here.

Continuing from the previous article, this posting will address how to simulate tag events using the RfidClientConsole application.  Using the AddEventToProcessPipeline command is the most commonly used simulation technique included with the sample handlers and code provided with Biztalk RFID.  The basic syntax of the command is

RfidClientConsole [-m[achine] machine name] [-p[ort] port number] AddEventToProcessPipeline <Process name> <Event XML file> <Logical device name>

In order to use this command, the following pieces of information are necessary:

Process Name. The list of process names can be obtained either by looking at the list of names in the RFID Manager, or via the GetAllProcessStatus command.  As seen below, the process name for this example is SampleProcess.

Logical Device Name.  The list of logical devices can be obtained either by looking at the list in the RFID Manager, or by using the GetAllDeviceStatus command.  As seen below, the logical device name for this example is SampleDevice.

Event XML File.  The RfidClientConsole interface uses XML files to pass complicated data structures to the back-end RFID service.  The "stock" sample XML file (as generate by RfidClientConsole help AddEventToProcessPipeline) is described in the next section, as there are a couple of elements in the file that can be a bit tricky the first time through.

Understanding the Event XML File

The stock Event XML file for the AddEventToProcessPipeline is shown above.  Whereas the next article in this series will discuss how to programatically construct events, the intent of this posting is to go through how to manually construct these event XML files.

<RfidEventBase xmlns:i="http://www.w3.org/2001/XMLSchema-instance" i:type="TagReadEvent" xmlns="http://schemas.datacontract.org/2004/07/Microsoft.SensorServices.Rfid">
  <m_lockObject />
  <vendorSpecificData xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.IO.SensorServices.Rfid.Client" i:nil="true" />
  <m_deviceName>Device Name</m_deviceName>
  <m_sourceName>Antenna 1</m_sourceName>
  <m_time>2008-01-04T03:02:52.3787395-03:30</m_time>
  <data i:nil="true" />
  <dataSelector xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.IO.SensorServices.Rfid.Client">
    <d2p1:dataSelector>123</d2p1:dataSelector>
  </dataSelector>
  <id>AQEBAQ==</id>
  <numberingSystemIdentifier i:nil="true" />
  <type xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.IO.SensorServices.Rfid.Client">
    <d2p1:description>EPC Class 0 tag</d2p1:description>
    <d2p1:enumValue>1</d2p1:enumValue>
  </type>
</RfidEventBase>

Some of the key aspects of this XML file to remember when developing a customized instance are:

Vendor Specific Data. No vendor specific data has been attached to the sample.

Device Name. The device name is largely irrelevant, as the events are directly injected through a logical device, as opposed to appearing to come directly from a physical device.

Source Name. As per the device name, the source name does not have to adhere to a specific convention -- unless your event processing pipeline happens to use the source name or device name to perform event enrichment.  In that case simply modify the source name to match the desired condition.

Time.  Time is a standard date time with the format noted above (full precision with time zone).  No special measures need be taken when customizing the time, provided that a parseable/deserializable DateTime format string is used.

Data/Id.  This is one of the more interesting aspects of the Event file.  Both the Data and the Id fields are base64 encoded binary strings.  A full description of how this file formats the Data and Id fields is given in the next section.

Type.  The type either has to match up with one of the standard tag types, or use a user defined type.  One useful technique when doing tag injections is the ability to code simulated tag events with a simulated tag type. 

Encoding and Decoding the Tag Id / Tag Data Fields

As mentioned in the previous section, the Tag Id and Tag Data fields in the event description XML file are base64 encoded.  In order to customize the ID and Data fields the appropriate conversions need to be made.  Looking at the sample file, we observe that the Id field contains the value

<id>AQEBAQ==</id>

Using a simple base64 encoding/decoding application written in .NET (download the project file here TODO), the heavy lifting of which is shown below, its easy to decode this ID string into the binary stream 01 01 01 01.

byte[] decbuff = Convert.FromBase64String(data);     

In order to add our own customized data values to the event file, we need to perform the opposite operation.  For example, our tag ID will be E0 07 01 02 03 04 05 06 (signifying an ISO15693 tag with a TI manufacturer code), and our tag data will represent the attendee code 1079 (binary 00 04 37 with a CRC of 33).

Running these values through our simple conversion application gives us a base64 encoded tag ID value of 4AcBAgMEBQY=, and a base64 encoded tag data value of AAQ3Mw==.

Adding the Tag Event to a Running Process

Now that we have the necessary information to customize a tag event, we are ready to add our simulated tag event into the event processing pipeline.  Using the new tag event file as shown below, execute the following command:

RfidClientConsole AddEventToProcessPipeline SampleProcess CustomTagEvent.xml SampleDevice

<RfidEventBase xmlns:i="http://www.w3.org/2001/XMLSchema-instance" i:type="TagReadEvent" xmlns="http://schemas.datacontract.org/2004/07/Microsoft.SensorServices.Rfid">
  <m_lockObject />
  <vendorSpecificData xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.IO.SensorServices.Rfid.Client" i:nil="true" />
  <m_deviceName>Simulated Device</m_deviceName>
  <m_sourceName>Antenna 1</m_sourceName>
  <m_time>2008-01-04T03:02:52.3787395-03:30</m_time>
  <data>AAQ3Mw==</data>
  <dataSelector xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.IO.SensorServices.Rfid.Client">
    <d2p1:dataSelector>123</d2p1:dataSelector>
  </dataSelector>
  <id>4AcBAgMEBQY=</id>
  <numberingSystemIdentifier i:nil="true" />
  <type xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.IO.SensorServices.Rfid.Client">
    <d2p1:description>Simulated Tag</d2p1:description>
    <d2p1:enumValue>1024</d2p1:enumValue>
  </type>
</RfidEventBase>

Looking at the event logs the following event log report should appear:

13|Verbose|010408 13:35:41|Enriching tag with attendee code 1079 (000437 CRC 51)|[SampleEnrichment]

Looking to the TagEvents table in the rfidsink database:

SELECT TOP 1 DeviceName, TagId, TagTypeDescription, TagTime, TagData FROM TagEvents ORDER BY TagTime DESC

Simulated Device    0xE007010203040506    Simulated Tag    2008-01-04 03:02:52.380    0x00043733

Finally, looking at the vendor extension data for this tag event, we see our enriched data:

<VendorSpecificInformation>
  <dictionary >
    <d2p1:KeyValueOfstringanyType>
      <d2p1:Key>AttendeeCode</d2p1:Key>
      <d2p1:Value i:type="d4p1:string">1079</d2p1:Value>
    </d2p1:KeyValueOfstringanyType>
  </dictionary>
</VendorSpecificInformation>

Note: I've removed the xmlns portions of the file for clarity.

Summary

This technique is excellent for static tests, but inflexible when it comes to simulating large numbers of tag reads, or specific logic chains.  The same interface provided by the RfidClientConsole can be accessed through a .NET application to enable more flexible and dynamic scenarios.  Using the management API to inject tag events into running processes will be described in the next article in this series.

One of the key tasks during implementation of an RFID-centric information system is pushing tag events through the system.  The purpose of the RFID Simulator.NET project is to provide this capability in modeling information flows through distributed systems by simulating a network of readers, each with assigned behaviour models.

Whereas the RFID Simulator.NET system will supply tag events by simulating a network of readers, Biztalk RFID offers another method of simulating tag events, by directly injecting them into a running process through the management API.  This is typically done either through the RfidClientConsole, or directly through a .NET application.  Note that most of the provided sample applications scenarios provided with Biztalk RFID use this technique to simulate tag reads.

This series of articles will demonstrate both of these methods, and common scenarios for using this technique. This first article will cover the sample system used to demonstrate the technique in context, as well as the development of a simple event enrichment component.

All of the sample code referenced in this article may be downloaded here.

Baseline System Configuration

The rest of this article will reference the sample system described below.  It consists of a Biztalk RFID process chain connected to the Contoso reader simulator.  The scenario is assumed to be that of a trade show booth with a wireless reader (an IDBlue device connected via Bluetooth) scanning attendee tags for a door prize.

The tags are standard ISO15693 tags.  The readers are assumed to be configured to read the first block (4 bytes) of data and include that data with the tag read event.  This 4 byte data block consists of a 24-bit integer (MSB), and a simple 8-bit CRC.

Note that this technique is not reader-dependent, as tag events are injected against a logical reader, not against a physical reader.

The process chain consists of two components:

1. The enrichment component that validates the tag data, and enriches the raw binary data into an Attendee code.  This attendee code is attached to the event and passed down the chain.  If the tag data is invalid (i.e. the CRC code doesn't match up) the tag is instead assigned an error code.

   Note: the filtering component doesn't actually "filter" out the tags that do not contain the correct application code due to the anticipated user experience (i.e. if a user sees a tag scan, the user will expect to see some reflection of that tag scan through a visual interface other than the reader.  If there is no system reaction to a scanned tag, even a small "unrecognized tag" message, the end-user assumption is that something is broken).

2. The standard SqlSink component stores the tag events (with the associated enriched tag information).

Developing the Enrichment Component

For the purposes of this sample system, we wish to develop an event processing component with the following features:

1. Handles TagRead and TagList events, looking for tags with a specific tag data pattern (the 24-bit identifier + CRC as described in the previous section).
2. If a valid tag pattern is present, attach a vendor specific extension data element containing the attendee identifier.
3. If a valid tag pattern is not present, have the configurable ability to either:
   1. Filter the tag from the stream
   2. Attach a vendor specific extension containing a warning
   3. Pass the event through the component

All of the sample code referenced in this article may be downloaded here.

Step 1: Prepare the Project, References and Using Statements