Archive for September, 2009

Using IronPython as a scripting engine – continued

3 Comments »

In my last post, I gave an example of how to execute a Python script from within a C# application and I tried to justify why you might want to do something like that. In this post I will elaborate on the very simple example from the previous post by passing data to and from the Python script being executed. If you haven’t read the previous post, you should at least skim it.

The data we will be passing to IronPython will represent an order in an order processing application. To this end I have created two very simple classes:

public class Order
{
    public Order()
    {
        this.OrderLines = new List<OrderLine>();
    }

    public string Headline { get; set; }
    public List<OrderLine> OrderLines { get; private set; }

    public override string ToString()
    {
        StringBuilder sb = new StringBuilder();
        sb.Append("\t");
        sb.AppendLine(this.Headline);
        sb.AppendLine(string.Join("\n", OrderLines.Select(l => l.ToString()).ToArray()));

        return sb.ToString();
    }
}

public class OrderLine
{
    public OrderLine(string productNumber, int quantity, decimal unitPrice)
    {
        this.ProductNumber = productNumber;
        this.Quantity = quantity;
        this.UnitPrice = unitPrice;
    }

    public string ProductNumber { get; set; }
    public int Quantity { get; set; }
    public decimal UnitPrice { get; set; }
    public decimal Price { get { return Quantity * UnitPrice; } }

    public override string ToString()
    {
        return string.Format("{0} pcs. of {1} at {2} each: {3}", Quantity, ProductNumber, UnitPrice, Price);
    }
}

I have also created a simple method for creating a dummy order:

public static Order GetOrder()
{
    Order order = new Order();
    order.Headline = "For expo 2009";
    order.OrderLines.Add(new OrderLine("prod1", 1, 24m));
    order.OrderLines.Add(new OrderLine("prod2", 2, 10m));
    order.OrderLines.Add(new OrderLine("prod3", 1, 26m));

    return order;
}

In our Python script we want to call ToString on the order, so make your Main method look like this:

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode = @"print order.ToString()";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();

    source.Execute(scope);

    System.Console.ReadKey();
}

Now, try running the application. The call to source.Execute(scope) will throw an IronPython.Runtime.UnboundNameException saying that “name ‘order’ is not defined”. What this means is obviously that the IronPython runtime does not recognize the symbol ‘order’. So we will have to add it to the symbol table before executing the script. Since we already have an object representing the scope of the script, it is natural to look to this object for a way to inject the ‘order’ symbol. Indeed, the ScriptScope class comes with a method

public void SetVariable(string name, object value);

which we can use. Thus, we change our Main method to

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode = @"print order.ToString()";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();
    scope.SetVariable("order", order); //order injected here
    source.Execute(scope);

    System.Console.ReadKey();
}

Now, running the application we see that the Python script is able to reference the order:

image

Of course, we can also change properties of the Order object inside the Python script: changing the Python code and calling the order.ToString() method after the execution of the Python script

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode = //New Python script calculating the order sum
@"sum = 0
for l in order.OrderLines:
    sum += l.Price

order.Headline = 'Total for expo 2009: ' + sum.ToString()
";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();
    scope.SetVariable("order", order);
    source.Execute(scope);

    System.Console.WriteLine(order.ToString());

    System.Console.ReadKey();
}

results in

image

On the other hand, you might want to create an object inside the Python script and afterwards reference it from the C# application.

Variables may be retrieved from the Python script scope using the ScriptScope’s methods

public object GetVariable(string name);
public T GetVariable<T>(string name);

Thus, to create an Order inside the Python script and reference it afterwards, you can do something like

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode =
@"order2 = Order()
order2.Headline = 'This is the second order'
";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();

    source.Execute(scope);
    Order order2 = scope.GetVariable<Order>("order2");
    System.Console.WriteLine(order2.ToString());

    System.Console.ReadKey();
}

However, executing this code will throw an UnboundNameException because the IronPython runtime cannot find the ‘Order’ symbol. Fortunately, you can inject the Order class just like you did for the variable earlier. Just add the line

scope.SetVariable("Order", typeof(Order));

before executing the script. Then running the application will result in

image

Using the techniques described above, it should be pretty easy to extend almost any application with support for IronPython plug-ins.

The technical details of this post are based on the material in Chapter 15 of Michael Foord’s “IronPython in Action”


Using IronPython as a scripting engine

2 Comments »

In this post I will look into the subject of extending an application written in a traditional .NET language like C# using IronPython. I am not talking about authoring assemblies by writing them in IronPython, though: I’m talking about letting some third party create plug-ins in Python which my application will load and execute at runtime.

Let’s say we are C# programmers developing some LOB application for processing orders from a fixed set of customers. Since these are trusted customers, they all receive special treatment: Some receive a discount on all their orders, for others we want to always add a “URGENT”-headline to their order, and yet others receive other kinds of special treatment.

Even if no third party will ever be creating plug-ins for our application, the support for plug-ins will greatly increase the application’s flexibility and reduce our maintenance costs.

To enable our order processing system to accomodate these and future cases, we have considered various options:

  • Implement all cases directly in the system using a lot of if-then-else and switches. When requirements are added or changed we implement, test and redeploy the application.
  • Implement a set of (parameterized) templates for handling cases like those described above. When requirements are added or changed, pray that they can be covered by one or more of the existing templates. Otherwise, implement a new template, test and redeploy the application.
  • Allow the application to execute plug-ins at certain stages in the process. Make the plug-ins execute in a sandboxed environment, thereby restricting the amount of havoc they can cause. When requirements are added or change, write a new snippet of code, test just the snippet (as opposed to the whole application) and deploy it.

Obviously, this post will explore the third option, using IronPython as an embedded scripting engine. I will not be presenting an implementation of the imagined order processing application but only showing code snippets illustrating the points of interacting with IronPython from C#.

So, the first thing to do is to be able to execute Python code from a C# application. This isn’t as difficult as it might sound:

  1. Download and install IronPython (cf. http://ironpython.codeplex.com/)
  2. Create a new C# console application in Visual Studio
  3. Add references to the DLR and IronPython assemblies (all found in the IronPython install directory, which is “C:\Program Files\IronPython 2.6” on my machine):
    • IronPython.dll
    • IronPython.Modules.dll
    • Microsoft.Scripting.dll
    • Microsoft.Scripting.Core.dll
    • Microsoft.Scripting.ExtensionAttribute.dll
  4. Add the following code to your application:
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Microsoft.Scripting.Hosting;
    using IronPython.Hosting;
    
    namespace Console
    {
        class Program
        {
            static void Main(string[] args)
            {
                ScriptEngine engine = Python.CreateEngine();
                ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
                ScriptScope scope = engine.CreateScope();
                source.Execute(scope);
    
                System.Console.ReadKey();
            }
        }
    }

As you have probably guessed, the Python code we will be executing comes from the pythonCode variable, which is undeclared at the moment. In a real application we might read the plug-in code from a database or from a file on the disk, but for now we’ll just embed it in the application: add the following as the first statement in the Main method:

string pythonCode =
@"g = lambda x: 3*x
print '3*4 equals', g(4)
";

When we run the Python code this calculates and prints the result of multiplying 3 by 4.

Now compile and run the C# application. You should something similar to this:

image

So, that was it. We were able to execute a piece of Python code from within a C# application.

Of course, if the technique above is to be used for anything useful, we will have to be able to pass data to and from the plug-in being executed. I will look into this in a later posting.

The technical details of this post are based on the material in Chapter 15 of Michael Foord’s “IronPython in Action”.


Writing an Add-in for Visual Studio 2008

4 Comments »

I’ve been thinking about writing a test generation add-in for Visual Studio. The idea is to have an add-in that will enable the user to auto-generate a bunch of tests for each controller in an ASP.NET MVC project.

I haven’t done any programming for Visual Studio before, so I decided that the first step would be to be able to place a menu item in the context menu of the files in Solution Explorer containing controllers.

 vsaddin_enabled vsaddin_disabled

To be able to query the Solution Explorer for things like ‘give me the files the user currently has selected’, ‘does the file the user has selected contain a controller’ etc., I created a wrapper:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using EnvDTE;
using EnvDTE80;
using TestGenerator.CodeGeneration;

namespace TestGenerator
{
    public class SolutionExplorerWrapper
    {
        private DTE2 _applicationObject;

        public SolutionExplorerWrapper(DTE2 applicationObject)
        {
            this._applicationObject = applicationObject;
        }

        public bool IsFolder(ProjectItem item)
        {
            foreach (string guid in folderGUIDs)
            {
                if (item.Kind.ToUpper() == guid)
                    return true;
            }
            return false;
        }

        public IEnumerable<ProjectItem> GetSelectedProjectItems()
        {
            UIHierarchy uih = (UIHierarchy)_applicationObject.Windows.Item(Constants.vsWindowKindSolutionExplorer).Object;
            Array selectedItems = (Array)uih.SelectedItems;
            foreach (UIHierarchyItem item in selectedItems)
                yield return (ProjectItem)item.Object;
        }

        public IEnumerable<ProjectItem> GetSelectedItemsContainingControllers()
        {
            return (from pi in GetSelectedProjectItems()
                    where
                        pi.IsPhysicalFile() &&
                        ContainsControllers((FileCodeModel2)pi.FileCodeModel)
                    select pi);
        }



        private bool ContainsControllers(FileCodeModel2 file)
        {
            FileCodeModel2QueryHelper helper = new FileCodeModel2QueryHelper(file);
            return helper.FileContainsControllers();
        }

        private readonly string[] folderGUIDs = new string[3] { "{6BB5F8EF-4483-11D3-8BCF-00C04F8EC28C}", "{6BB5F8F0-4483-11D3-8BCF-00C04F8EC28C}", "{66A26722-8FB5-11D2-AA7E-00C04F688DDE}" };
    }
}

The elements in the Solution Explorer are represented as ProjectItem objects and because of all the COM goo (I guess) you can’t just do something like

if(projectItem is IFolder)

to determine if a given ProjectItem instance is a folder. Thus, to ease the interaction with these objects, I created two extension methods:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using EnvDTE;

namespace TestGenerator
{
    public static class ProjectItemExtensions
    {
        public static bool IsPhysicalFile(this ProjectItem item)
        {
            return item.Kind == EnvDTE.Constants.vsProjectItemKindPhysicalFile;
        }

        public static bool IsPhysicalFolder(this ProjectItem item)
        {
            return item.Kind == EnvDTE.Constants.vsProjectItemKindPhysicalFolder;
        }
    }
}

Given a COM object like a ProjectItem, you can use a class in the Microsoft.VisualBasic namespace to get a string representation of what type of object the COM object is wrapping. Hence, I created the following extension method, which was very helpful during development:

public static class ComObjectExtentions
{
    public static string GetTypeName(this object o)
    {
        return Microsoft.VisualBasic.Information.TypeName(o);
    }
}

The actual implementation of the add-in looks complex at first glance, but it is really quite simple. Much of the code is just boilerplate code generated by the Visual Studio Add-In wizard.

public class Connect : IDTExtensibility2, IDTCommandTarget
    {
        /// <summary>Implements the constructor for the Add-in object. Place your initialization code within this method.</summary>
        public Connect()
        {
        }

        /// <summary>Implements the OnConnection method of the IDTExtensibility2 interface. Receives notification that the Add-in is being loaded.</summary>
        /// <param term='application'>Root object of the host application.</param>
        /// <param term='connectMode'>Describes how the Add-in is being loaded.</param>
        /// <param term='addInInst'>Object representing this Add-in.</param>
        /// <seealso class='IDTExtensibility2' />
        public void OnConnection(object application, ext_ConnectMode connectMode, object addInInst, ref Array custom)
        {
            try
            {
                _applicationObject = (DTE2)application;
                _addInInstance = (AddIn)addInInst;

                switch (connectMode)
                {
                    case ext_ConnectMode.ext_cm_UISetup:

                        // Create commands in the UI Setup phase. This phase is called only once when the add-in is deployed.
                        CreateCommands();
                        break;

                    case ext_ConnectMode.ext_cm_AfterStartup:

                        InitializeAddIn();
                        break;

                    case ext_ConnectMode.ext_cm_Startup:

                        // Do nothing yet, wait until the IDE is fully initialized (OnStartupComplete will be called)
                        break;
                }
            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.ToString(), "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
            }
        }

        private void InitializeAddIn()
        {
            CommandBarControl myCommandBarControl;
            CommandBar codeWindowCommandBar;
            Command myCommand1;
            CommandBars commandBars;

            // Retrieve commands created in the ext_cm_UISetup phase of the OnConnection method
            myCommand1 = _applicationObject.Commands.Item(_addInInstance.ProgID + "." + m_COMMAND_GENERATETESTS_NAME, -1);

            // Retrieve the context menu of an item in solution explorer
            commandBars = (CommandBars)_applicationObject.CommandBars;
            codeWindowCommandBar = commandBars["Item"];

            // Add a popup command bar
            myCommandBarControl = codeWindowCommandBar.Controls.Add(MsoControlType.msoControlPopup,
               System.Type.Missing, System.Type.Missing, System.Type.Missing, System.Type.Missing);

            m_commandBarPopup = (CommandBarPopup)myCommandBarControl;

            // Change its caption
            m_commandBarPopup.Caption = "RUI Test Generator";

            // Add controls to the popup command bar
            m_commandBarControl1 = (CommandBarControl)myCommand1.AddControl(m_commandBarPopup.CommandBar,
               m_commandBarPopup.Controls.Count + 1);

            m_commandBarControl1.Caption = "Generate Tests";
        }

        /// <summary>Implements the OnDisconnection method of the IDTExtensibility2 interface. Receives notification that the Add-in is being unloaded.</summary>
        /// <param term='disconnectMode'>Describes how the Add-in is being unloaded.</param>
        /// <param term='custom'>Array of parameters that are host application specific.</param>
        /// <seealso class='IDTExtensibility2' />
        public void OnDisconnection(ext_DisconnectMode disconnectMode, ref Array custom)
        {
            try
            {
                if (m_commandBarPopup != null)
                {
                    m_commandBarPopup.Delete(true);
                }
            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.ToString(), "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
            }
        }

        /// <summary>Implements the OnAddInsUpdate method of the IDTExtensibility2 interface. Receives notification when the collection of Add-ins has changed.</summary>
        /// <param term='custom'>Array of parameters that are host application specific.</param>
        /// <seealso class='IDTExtensibility2' />
        public void OnAddInsUpdate(ref Array custom)
        {
        }

        /// <summary>Implements the OnStartupComplete method of the IDTExtensibility2 interface. Receives notification that the host application has completed loading.</summary>
        /// <param term='custom'>Array of parameters that are host application specific.</param>
        /// <seealso class='IDTExtensibility2' />
        public void OnStartupComplete(ref Array custom)
        {
            InitializeAddIn();
        }

        /// <summary>Implements the OnBeginShutdown method of the IDTExtensibility2 interface. Receives notification that the host application is being unloaded.</summary>
        /// <param term='custom'>Array of parameters that are host application specific.</param>
        /// <seealso class='IDTExtensibility2' />
        public void OnBeginShutdown(ref Array custom)
        {
        }

        /// <summary>Implements the QueryStatus method of the IDTCommandTarget interface. This is called when the command's availability is updated</summary>
        /// <param term='commandName'>The name of the command to determine state for.</param>
        /// <param term='neededText'>Text that is needed for the command.</param>
        /// <param term='status'>The state of the command in the user interface.</param>
        /// <param term='commandText'>Text requested by the neededText parameter.</param>
        /// <seealso class='Exec' />
        public void QueryStatus(string commandName, vsCommandStatusTextWanted neededText, ref vsCommandStatus status, ref object commandText)
        {
            if (commandName == _addInInstance.ProgID + "." + m_COMMAND_GENERATETESTS_NAME)
            {
                SolutionExplorerWrapper solutionExplorer = new SolutionExplorerWrapper(_applicationObject);
                if (solutionExplorer.GetSelectedItemsContainingControllers().Any())
                    status = vsCommandStatus.vsCommandStatusSupported | vsCommandStatus.vsCommandStatusEnabled;
                else
                    status = vsCommandStatus.vsCommandStatusSupported;
            }
        }

        /// <summary>Implements the Exec method of the IDTCommandTarget interface. This is called when the command is invoked.</summary>
        /// <param term='commandName'>The name of the command to execute.</param>
        /// <param term='executeOption'>Describes how the command should be run.</param>
        /// <param term='varIn'>Parameters passed from the caller to the command handler.</param>
        /// <param term='varOut'>Parameters passed from the command handler to the caller.</param>
        /// <param term='handled'>Informs the caller if the command was handled or not.</param>
        /// <seealso class='Exec' />
        public void Exec(string commandName, vsCommandExecOption executeOption, ref object varIn, ref object varOut, ref bool handled)
        {
            if (commandName == _addInInstance.ProgID + "." + m_COMMAND_GENERATETESTS_NAME)
            {
                GenerateTestsForSelectedFiles();
            }
        }

        #region Menu and command setup

        private const string m_COMMAND_GENERATETESTS_NAME = "GenerateTests";
        private const string m_COMMAND_GENERATETESTS_TEXT = "Generate Tests";
        //private const string m_NAME_COMMAND2 = "MyCommand2";

        private CommandBarPopup m_commandBarPopup;
        private CommandBarControl m_commandBarControl1;
        //private CommandBarControl m_commandBarControl2;

        private void CreateCommands()
        {
            object[] contextUIGuids = new object[] { };

            _applicationObject.Commands.AddNamedCommand(_addInInstance, m_COMMAND_GENERATETESTS_NAME, m_COMMAND_GENERATETESTS_TEXT, m_COMMAND_GENERATETESTS_TEXT, true, 59,
               ref contextUIGuids, (int)vsCommandStatus.vsCommandStatusSupported);
        }

        #endregion

        public bool GenerateTestsForSelectedFiles()
        {
            SolutionExplorerWrapper solutionExplorer = new SolutionExplorerWrapper(_applicationObject);
            foreach (ProjectItem projectItem in solutionExplorer.GetSelectedProjectItems())
            {
                if (!projectItem.IsPhysicalFile())
                {
                    MessageBox.Show(string.Format("Cannot generate code for {0}", projectItem.Name), "Project item not supported");
                    return true;
                }

                CodeFileGenerator generator = new CodeFileGenerator();
                generator.GenerateTests(projectItem, _applicationObject.Solution);
            }
            return true;
        }

        private DTE2 _applicationObject;
        private AddIn _addInInstance;
    }

Of particular interest are the methods QueryStatus(…) and Exec(…).

QueryStatus is called whenever Visual Studio wants to know if the add-in should be available for use. As you can see, I just use the SolutionExplorerWrapper to make the add-in available, whenever the user has selected a file containing one or more controllers.

Exec is called whenever a command associated to the add-in is invoked. If the command is the command for generating tests (the add-in might contain other buttons or menu items invoking other commands) I proceed to create tests.

That’s it. My add-in now shows up whenever the user right-clicks a controller in Solution Explorer.

For completeness sake I have made all of the source code available here, including the CodeFileGenerator that I am using for generating (at the moment, very rudimentary) test code.