Posts Tagged ‘IronPython’

Using IronPython as a scripting engine – continued

3 Comments »

In my last post, I gave an example of how to execute a Python script from within a C# application and I tried to justify why you might want to do something like that. In this post I will elaborate on the very simple example from the previous post by passing data to and from the Python script being executed. If you haven’t read the previous post, you should at least skim it.

The data we will be passing to IronPython will represent an order in an order processing application. To this end I have created two very simple classes:

public class Order
{
    public Order()
    {
        this.OrderLines = new List<OrderLine>();
    }

    public string Headline { get; set; }
    public List<OrderLine> OrderLines { get; private set; }

    public override string ToString()
    {
        StringBuilder sb = new StringBuilder();
        sb.Append("\t");
        sb.AppendLine(this.Headline);
        sb.AppendLine(string.Join("\n", OrderLines.Select(l => l.ToString()).ToArray()));

        return sb.ToString();
    }
}

public class OrderLine
{
    public OrderLine(string productNumber, int quantity, decimal unitPrice)
    {
        this.ProductNumber = productNumber;
        this.Quantity = quantity;
        this.UnitPrice = unitPrice;
    }

    public string ProductNumber { get; set; }
    public int Quantity { get; set; }
    public decimal UnitPrice { get; set; }
    public decimal Price { get { return Quantity * UnitPrice; } }

    public override string ToString()
    {
        return string.Format("{0} pcs. of {1} at {2} each: {3}", Quantity, ProductNumber, UnitPrice, Price);
    }
}

I have also created a simple method for creating a dummy order:

public static Order GetOrder()
{
    Order order = new Order();
    order.Headline = "For expo 2009";
    order.OrderLines.Add(new OrderLine("prod1", 1, 24m));
    order.OrderLines.Add(new OrderLine("prod2", 2, 10m));
    order.OrderLines.Add(new OrderLine("prod3", 1, 26m));

    return order;
}

In our Python script we want to call ToString on the order, so make your Main method look like this:

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode = @"print order.ToString()";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();

    source.Execute(scope);

    System.Console.ReadKey();
}

Now, try running the application. The call to source.Execute(scope) will throw an IronPython.Runtime.UnboundNameException saying that “name ‘order’ is not defined”. What this means is obviously that the IronPython runtime does not recognize the symbol ‘order’. So we will have to add it to the symbol table before executing the script. Since we already have an object representing the scope of the script, it is natural to look to this object for a way to inject the ‘order’ symbol. Indeed, the ScriptScope class comes with a method

public void SetVariable(string name, object value);

which we can use. Thus, we change our Main method to

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode = @"print order.ToString()";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();
    scope.SetVariable("order", order); //order injected here
    source.Execute(scope);

    System.Console.ReadKey();
}

Now, running the application we see that the Python script is able to reference the order:

image

Of course, we can also change properties of the Order object inside the Python script: changing the Python code and calling the order.ToString() method after the execution of the Python script

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode = //New Python script calculating the order sum
@"sum = 0
for l in order.OrderLines:
    sum += l.Price

order.Headline = 'Total for expo 2009: ' + sum.ToString()
";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();
    scope.SetVariable("order", order);
    source.Execute(scope);

    System.Console.WriteLine(order.ToString());

    System.Console.ReadKey();
}

results in

image

On the other hand, you might want to create an object inside the Python script and afterwards reference it from the C# application.

Variables may be retrieved from the Python script scope using the ScriptScope’s methods

public object GetVariable(string name);
public T GetVariable<T>(string name);

Thus, to create an Order inside the Python script and reference it afterwards, you can do something like

static void Main(string[] args)
{
    var order = GetOrder();
    string pythonCode =
@"order2 = Order()
order2.Headline = 'This is the second order'
";

    ScriptEngine engine = Python.CreateEngine();
    ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
    ScriptScope scope = engine.CreateScope();

    source.Execute(scope);
    Order order2 = scope.GetVariable<Order>("order2");
    System.Console.WriteLine(order2.ToString());

    System.Console.ReadKey();
}

However, executing this code will throw an UnboundNameException because the IronPython runtime cannot find the ‘Order’ symbol. Fortunately, you can inject the Order class just like you did for the variable earlier. Just add the line

scope.SetVariable("Order", typeof(Order));

before executing the script. Then running the application will result in

image

Using the techniques described above, it should be pretty easy to extend almost any application with support for IronPython plug-ins.

The technical details of this post are based on the material in Chapter 15 of Michael Foord’s “IronPython in Action”


Using IronPython as a scripting engine

2 Comments »

In this post I will look into the subject of extending an application written in a traditional .NET language like C# using IronPython. I am not talking about authoring assemblies by writing them in IronPython, though: I’m talking about letting some third party create plug-ins in Python which my application will load and execute at runtime.

Let’s say we are C# programmers developing some LOB application for processing orders from a fixed set of customers. Since these are trusted customers, they all receive special treatment: Some receive a discount on all their orders, for others we want to always add a “URGENT”-headline to their order, and yet others receive other kinds of special treatment.

Even if no third party will ever be creating plug-ins for our application, the support for plug-ins will greatly increase the application’s flexibility and reduce our maintenance costs.

To enable our order processing system to accomodate these and future cases, we have considered various options:

  • Implement all cases directly in the system using a lot of if-then-else and switches. When requirements are added or changed we implement, test and redeploy the application.
  • Implement a set of (parameterized) templates for handling cases like those described above. When requirements are added or changed, pray that they can be covered by one or more of the existing templates. Otherwise, implement a new template, test and redeploy the application.
  • Allow the application to execute plug-ins at certain stages in the process. Make the plug-ins execute in a sandboxed environment, thereby restricting the amount of havoc they can cause. When requirements are added or change, write a new snippet of code, test just the snippet (as opposed to the whole application) and deploy it.

Obviously, this post will explore the third option, using IronPython as an embedded scripting engine. I will not be presenting an implementation of the imagined order processing application but only showing code snippets illustrating the points of interacting with IronPython from C#.

So, the first thing to do is to be able to execute Python code from a C# application. This isn’t as difficult as it might sound:

  1. Download and install IronPython (cf. http://ironpython.codeplex.com/)
  2. Create a new C# console application in Visual Studio
  3. Add references to the DLR and IronPython assemblies (all found in the IronPython install directory, which is “C:\Program Files\IronPython 2.6” on my machine):
    • IronPython.dll
    • IronPython.Modules.dll
    • Microsoft.Scripting.dll
    • Microsoft.Scripting.Core.dll
    • Microsoft.Scripting.ExtensionAttribute.dll
  4. Add the following code to your application:
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Microsoft.Scripting.Hosting;
    using IronPython.Hosting;
    
    namespace Console
    {
        class Program
        {
            static void Main(string[] args)
            {
                ScriptEngine engine = Python.CreateEngine();
                ScriptSource source = engine.CreateScriptSourceFromString(pythonCode);
                ScriptScope scope = engine.CreateScope();
                source.Execute(scope);
    
                System.Console.ReadKey();
            }
        }
    }

As you have probably guessed, the Python code we will be executing comes from the pythonCode variable, which is undeclared at the moment. In a real application we might read the plug-in code from a database or from a file on the disk, but for now we’ll just embed it in the application: add the following as the first statement in the Main method:

string pythonCode =
@"g = lambda x: 3*x
print '3*4 equals', g(4)
";

When we run the Python code this calculates and prints the result of multiplying 3 by 4.

Now compile and run the C# application. You should something similar to this:

image

So, that was it. We were able to execute a piece of Python code from within a C# application.

Of course, if the technique above is to be used for anything useful, we will have to be able to pass data to and from the plug-in being executed. I will look into this in a later posting.

The technical details of this post are based on the material in Chapter 15 of Michael Foord’s “IronPython in Action”.


Parsing EDI messages with IronPython

2 Comments »

I have recently been wondering about how to build a generic system for parsing and processing lots of EDI messages in such a way that a minimum of work is needed when a new message type is to be processed by the system. The syntactic format of EDI messages is fairly consistent, but the semantics of particular fields are open for interpretation. Thus, I thought it should be possibly to write a general parser for turning a message into a tree structure (much like an abstract syntax tree for “the EDI language”) which would be appropriate as input to the processing phase of the system.

Having recently picked up “IronPython in Action” I decided to apply IronPython to the task. Since this was only an experiment, I decided to somewhat give up the idea of a tree structure and instead leverage the dynamic programming capabilities of IronPython. Given an EDI message at runtime I wanted to generate an object with properties corresponding to the segments, subsegments and elements of the message.

To represent the document elements I created the following very simple classes:

class Document(object):
    pass

class Segment(object):
    pass

class Element(object):
    pass

class DataElement(object):
    pass

As you can see these are all identical classes, simply derived from object. They are created as seperate classes to allow code that reflect on their type to be able to make sense of their intended use.

To describe the various delimiters and escape characters of an EDI message I created an EDIDelimiterContext:

class EDIDelimiterContext(object):
    def __init__(self, segmentSeparator='\'', elementSeparator='+', dataElementSeparator=':', escapeCharacter='?'):
        self.SegmentSeparator = segmentSeparator
        self.ElementSeparator = elementSeparator
        self.DataElementSeparator = dataElementSeparator
        self.EscapeCharacter = escapeCharacter


The code for the parser looks like this:

from Document import Document, Segment, Element, DataElement
import clr

class Parser(object):
    def __init__(self, ediDelimiterContext):
        self.EDIDelimiterContext = ediDelimiterContext

    "Splits the string input according to the given separator, taking the given escape character into consideration"
    def splitter(self, input, escape, separator):
        subStrings = []
        i = 0
        while (i < input.Length):
            if (input[i] == separator and (i == 0 or input[i - 1] != escape)):
                subStrings.append(input.Substring(0, i))
                input = input.Remove(0, i + 1)
                i = 0
            else:
                i += 1

        if(input.Length > 0):
            subStrings.append(input)

        return subStrings

    def getSegmentsStrings(self, input):
        return self.splitter(input, self.EDIDelimiterContext.EscapeCharacter, self.EDIDelimiterContext.SegmentSeparator)

    def getElementsStrings(self, input):
        return self.splitter(input, self.EDIDelimiterContext.EscapeCharacter, self.EDIDelimiterContext.ElementSeparator)

    def getDataElementsStrings(self, input):
        return self.splitter(input, self.EDIDelimiterContext.EscapeCharacter, self.EDIDelimiterContext.DataElementSeparator)

    "Attaches a property named propertyName with value value to the object targetObj. If a property with that name already exists, an index is added to the end of the name to construct a unique property name"
    def attachProperty(self, targetObj, propertyName, value):
        postfix = ""
        if hasattr(targetObj, propertyName):
            i = 2
            while hasattr(targetObj, propertyName + str(i)):
                i += 1
            setattr(targetObj, propertyName + str(i), value)
        else:
            setattr(targetObj, propertyName, value)

    "Builds a Document from a string (input)"
    def buildDocumentObject(self, input):
        doc = Document()
        segmentStrings = self.getSegmentsStrings(input)
        for segmentstr in segmentStrings:
            name, value = self.buildSegmentObject(segmentstr)
            self.attachProperty(doc, name, value)
        return doc

    "Builds a Segment from a string (input)"
    def buildSegmentObject(self, input):
        seg = Segment()
        elementsStrings = self.getElementsStrings(input)
        name, elements = self.splitNameFromSegment(elementsStrings)
        for elementstr in elements:
            self.attachProperty(seg, 'Element', self.buildElementObject(elementstr))
        return name, seg

    "Builds an Element from a string (input)"
    def buildElementObject(self, input):
        element = Element()
        dataElements = self.getDataElementsStrings(input)
        for dataelementstr in dataElements:
            self.attachProperty(element, 'DataElement', dataelementstr)
        return element

    "Determines the name of a segment. If the segment contains multiple elements, the first element will form the name of the segment. That is, the segment ABC+DEF:def+GH:gef will be named ABC"
    def splitNameFromSegment(self, elements):
        if len(elements) == 0:
            return ('Segment', elements)

        name = elements.pop(0)
        return (name, elements)

The method of primary interest in the Parser is attachProperty(self, targetObj, propertyName, value) which will attach a new property to the object given by targetObj. The property will have the value given by the value parameter. The property will be named according to the propertyName parameter, unless a property with that name already exists. In that case we will try to go with <propertyName>2, <propertyname>3, etc. Coming from a background in C# I think it is really nice to see how easy it is to attach a property using the setattr method. Dynamically typed, late bound languages like IronPython and Javascript really make you see the world in a different light once tricks like this become part of your arsenal.

Using the parser above, I can now create an object from an EDI message as follows:

>>> context = Parser.EDIDelimiterContext()

>>> parser = Parser.Parser(context)

>>> doc = parser.buildDocumentObject(‘ABC+DEF:def+GH:gef\’IJK+LM:nop:q21′)

>>> doc.ABC

<Segment object at 0x000000000000002B>

>>> doc.ABC.Element.DataElement

‘DEF’

>>> doc.ABC.Element.DataElement2

‘def’

>>>

Notice that when I write doc.ABC.El<TAB> the tab completion of the python console will allow me to easily cycle through the elements of the ABC segment. This will be immensely valuable, since given a sample EDI message of some new type, the tab completion features will guide me when I’m implementing the logic for processing such messages.