2012-08-29

C# scripting in .NET

Based on a couple of days of experimenting it seems that I now have C# scripting working in my application to a level that I am satisfied.  Because I couldn’t have achieved this without looking at other people’s examples I thought it was only fair that I share what I now have.  Firstly I want to say that my purpose was to give the user the ability to write procedural code rather than object code.  Although it is still possible for the user to write OOP source code my requirement was to let the user define a function which returned a specific return type.

public interface ICompiledFunction<T>
{
T Execute(Dictionary
<string, object> variables);
}


A typical script might look something like this




public decimal string Main()
{
return "Bob Monkhouse";
}



To create an instance of the ICompiledFunction<T> I use ICompilerService, which is defined like so



public interface ICompilerService
{
bool Compile<T>(
string[] scripts,
Dictionary
<string, Type> variableDefinitions,
IEnumerable
<Assembly> referencedAssemblies,
out ICompiledFunction<T> function,
out string sourceCode,
out IEnumerable<string> compilerErrors);
}




    1. Scripts: An array of methods which should be combined to make the full script.  Each may include its own “using” statements.


    2. VariableDefinitions: A dictionary of variable names with their types.  These will appear to be global variables available to the combined scripts, but in fact they will be properties defined as part of the wrapper class which is generated automatically.


    3. ReferencedAssemblies: A collection of Assembly, used to ensure that the script has access to any types declared.


    4. Function: The compiled function


    5. SourceCode: The complete source code which is compiled after it has been merged; useful for working out what is wrong with your source code.


    6. CompilerErrors: A collection of compiler errors.




The importance of CompilerService is that the implementation will create a new AppDomain each time you call Compile.  When the CompilerService is dispose it will unload all assemblies.  I will post the code with comments inline



using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Reflection;

namespace PeterMorris.Scripting
{
public class CompilerService : ICompilerService, IDisposable
{
Stack
<AppDomain> SandboxAppDomains = new Stack<AppDomain>();
bool Disposed;

public bool Compile<T>(
string[] scripts,
Dictionary
<string, Type> variableDefinitions,
IEnumerable
<Assembly> referencedAssemblies,
out ICompiledFunction<T> function,
out string sourceCode,
out IEnumerable<string> compilerErrors)
{
if (scripts == null)
throw new ArgumentNullException("Scripts");

//Define app domain settings such as the code base
var appDomainSetup = new AppDomainSetup();
string appBase = typeof(CompilerService).Assembly.CodeBase;
appBase
= new Uri(appBase).LocalPath;
appDomainSetup.ApplicationBase
= Path.GetDirectoryName(appBase);
var evidence
= AppDomain.CurrentDomain.Evidence;

//Create a uniquely named app domain and push it onto a stack for disposing later
string sandboxAppDomainName = "Sandbox" + Guid.NewGuid().ToString().Replace("-", "");
var sandboxAppDomain
= AppDomain.CreateDomain(sandboxAppDomainName, evidence, appDomainSetup);
SandboxAppDomains.Push(sandboxAppDomain);

//Clone the original scripts, we don't want to alter the originals
scripts = scripts.ToArray();

//Strip out the using clauses from each script so that they can be combined
var usingClauses = StripOutUsingClauses(scripts).OrderBy(x => x).ToList();

//Build the complete source code to be compiled
sourceCode = BuildSourceCode(usingClauses, scripts, variableDefinitions);

CompiledFunction
<T> result;
try
{
//Create an instance of CompiledFunction<T> in the new app domain
result = (CompiledFunction<T>)sandboxAppDomain.CreateInstanceAndUnwrap(
assemblyName:
typeof(CompiledFunction<T>).Assembly.GetName().FullName,
typeName:
typeof(CompiledFunction<T>).FullName);
}
catch (Exception unexpectedException)
{
function
= null;
compilerErrors
= new string[] { unexpectedException.Message };
return false;
}

//Compile the function
//Variable definitions and source code are passed so that we can translate line numbers
//in the error message back to the line number of the snippet of source the user
//has typed in
function = (ICompiledFunction<T>)result;
return result.Compile(
sourceCode: sourceCode,
variableDefinitions: variableDefinitions,
referencedAssemblies: referencedAssemblies,
compilerErrors:
out compilerErrors);
}

public static HashSet<string> StripOutUsingClauses(string[] scripts)
{
//Make a hashset of using clauses, to ensure we have no duplicates
var usingClauses = new HashSet<string>();
//Ensure that the standard using clauses are present
usingClauses.Add("System");
usingClauses.Add(
"System.Linq");
usingClauses.Add(
"System.Collections.Generic");

//Create a regex which matches a using statement
var regex = new Regex(@"^[\s]*using[\s]+(\S*?)[\s]*;[\s]*$", RegexOptions.IgnoreCase);

//Loop through each script
for (int scriptIndex = 0; scriptIndex < scripts.Length; scriptIndex++)
{
string script = scripts[scriptIndex];
if (string.IsNullOrEmpty(script))
continue;

var scriptReader
= new StringReader(script);
var scriptBuilder
= new StringBuilder();
while (true)
{
//Read the next line of the current script, null == end of script
string line = scriptReader.ReadLine();
if (line == null)
break;

//Ignore blank lines
if (line.Trim() == "")
continue;

var match
= regex.Match(line);
if (match.Captures.Count > 0)
{
//If there is a match then add the name to the using clause hash set
string nameSpace = match.Groups[1].Value;
usingClauses.Add(nameSpace);
}
else
{
//If no match then the using clauses are finished, add the rest
//of the text from the reader into the writer
scriptBuilder.AppendLine(line);
scriptBuilder.Append(scriptReader.ReadToEnd());
break;
}
}
//Loop through each line of script

//Replace the script with text without the using clauses
scripts[scriptIndex] = scriptBuilder.ToString();
}
//foreach script
return usingClauses;
}

//Builds the full source code with wrapping class
string BuildSourceCode(List<string> usingClauses, string[] scripts, IEnumerable<KeyValuePair<string, Type>> variableDefinitions)
{
if (usingClauses == null)
throw new ArgumentNullException("usingClauses");

var scriptBuilder
= new StringBuilder();
//Add using clauses
foreach (string nameSpace in usingClauses.OrderBy(x => x))
scriptBuilder.AppendFormat(
"using {0};\r\n", nameSpace);
if (usingClauses.Any())
scriptBuilder.AppendLine();

//namespace
scriptBuilder.AppendLine("namespace Sandbox");
scriptBuilder.AppendLine(
"{");
{
//Wrapper class
scriptBuilder.AppendLine("public class ScriptHolder");
scriptBuilder.AppendLine(
"{");
{
//Add variable definitions as properties within the class
BuildProperties(scriptBuilder, variableDefinitions);

//Combine all scripts as a single string
foreach (string script in scripts)
{
scriptBuilder.AppendLine(script);
scriptBuilder.AppendLine();
}
}
scriptBuilder.AppendLine(
"}"); //class
}
scriptBuilder.AppendLine(
"}"); //namespace
return scriptBuilder.ToString();
}

//Add variable definitions as properties within the script
void BuildProperties(StringBuilder scriptBuilder, IEnumerable<KeyValuePair<string, Type>> variableDefinitions)
{
if (variableDefinitions == null)
return;

foreach (var kvp in variableDefinitions.OrderBy(x => x.Key))
{
//ToScriptableString is required to expand generic types back into
//proper looking source code
string scriptableTypeName = kvp.Value.ToScriptableString();
string variableName = kvp.Key;
scriptBuilder.AppendFormat(
"\tpublic {0} {1};\r\n", scriptableTypeName, variableName);
}
if (variableDefinitions.Any())
scriptBuilder.AppendLine();
}

void Dispose(bool isDisposing)
{
if (Disposed)
return;
Disposed
= true;
if (isDisposing)
//Unload all sandbox app domains,
//in reverse order - just because I have OCD :-)
while (SandboxAppDomains.Count > 0)
AppDomain.Unload(SandboxAppDomains.Pop());
}

public void Dispose()
{
Dispose(
true);
}
}
}


One thing to note here is the use of Type.ToScriptableString() – this is implemented via a helper class.  This is because when you use GetType().FullName on a generic type you get a mangled name which would not compile in a script, so it must be unmangled.



using System;
using System.Text;

namespace PeterMorris.Scripting
{
public static class TypeHelper
{
public static string ToScriptableString(this Type type)
{
if (type == null)
return "Void";

//If not a generic type then just return the full name
if (!type.IsGenericType)
return type.FullName;

//Convert a generic type back to a compilable representation
var builder = new StringBuilder();
Type genericType
= type.GetGenericTypeDefinition();
//Unmangle the name
builder.Append(genericType.Name.Remove(genericType.Name.IndexOf("`")) + "<");
string separator = "";
//Include the inner types of the generic type
foreach (var genericArgument in type.GetGenericArguments())
{
//Recursively call ToScriptableString for the typename, as the inner type
//itself may also be a generic type
builder.AppendFormat("{0}{1}", separator, genericArgument.ToScriptableString());
separator
= ",";
}
builder.Append(
">");
return builder.ToString();
}
}
}


And now the implementing code for CompiledFunction<T>



using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using Microsoft.CSharp;

namespace PeterMorris.Scripting
{
public class CompiledFunction<T> : MarshalByRefObject, ICompiledFunction<T>
{
bool IsCompiled;
Dictionary
<string, Type> VariableDefinitions;
Dictionary
<string, FieldInfo> FieldReferences;
object Instance;
MethodInfo Method;

public CompiledFunction()
{
VariableDefinitions
= new Dictionary<string, Type>();
FieldReferences
= new Dictionary<string, FieldInfo>();
}

public bool Compile(
string sourceCode,
Dictionary
<string, Type> variableDefinitions,
IEnumerable
<Assembly> referencedAssemblies,
out IEnumerable<string> compilerErrors
)
{
if (IsCompiled)
throw new InvalidOperationException("Already compiled");
IsCompiled
= true;

if (variableDefinitions != null)
this.VariableDefinitions = variableDefinitions;

//Get the full path to the executing binary, this is used to find assembly
//references which are not in the GAC. Use Uri to convert to a local path.
string binaryPath = new Uri(AppDomain.CurrentDomain.BaseDirectory).LocalPath;
var fullReferencedAssemblyList
= new List<Assembly>(referencedAssemblies);
//System.dll
fullReferencedAssemblyList.Add(typeof(string).Assembly);
//System.Core.dll
fullReferencedAssemblyList.Add(typeof(IQueryable).Assembly);
//Get assembly names
string[] referencedAssemblyNames =
fullReferencedAssemblyList
.Select(x
=> x.GetName().Name + ".dll")
.Select
(
//If the file exists in the binary folder then add it by full path,
//otherwise add it without a path so that it is assumed to be in the GAC
assemblyName =>
File.Exists(Path.Combine(binaryPath, assemblyName))
?
Path.Combine(binaryPath, assemblyName) :
assemblyName
)
.Distinct()
.ToArray();

//Set the compiler parameters to compile in memory
var compilerparameters = new CompilerParameters
{
GenerateExecutable
= false,
GenerateInMemory
= true,
IncludeDebugInformation
= true,
TreatWarningsAsErrors
= true
};
//Add the list of referenced assemblies
compilerparameters.ReferencedAssemblies.AddRange(referencedAssemblyNames);

//Use C# 3.5 so that we have access to nice features such as LINQ
var options = new Dictionary<string, string>
{
{
"CompilerVersion", "v3.5" }
};

//Create the compiler
var compiler = new CSharpCodeProvider(options);
//Compile the source code
CompilerResults compilerResults;
try
{
compilerResults
= compiler.CompileAssemblyFromSource(
options: compilerparameters,
sources:
new string[] { sourceCode });
}
catch (Exception unexpectedException)
{
compilerErrors
= new string[] { unexpectedException.Message };
return false;
}

var errors
= new List<string>();
if (compilerResults.Errors.Count == 0)
{
//Create an instance of the new class
Instance = compilerResults.CompiledAssembly.CreateInstance("Sandbox.ScriptHolder");
//Find the Main() method
Method = Instance.GetType().GetMethod("Main");
//Report an error if it is not decalred
if (Method == null)
errors.Add(
string.Format("public {0} Main() has not been defined", typeof(T).Name));
else if (!typeof(T).IsAssignableFrom(Method.ReturnType))
//If the Main() method's return type is not assignable to the expected return type
//then report an error
errors.Add(string.Format("Expected return type {0} but found {1}",
typeof(T).FullName, Method.ReturnType.FullName));
}
//Add any compiler errors. FormatError uses the source code and variable definitions
//in order to work out the line number relative to the script entered by the user
//rather than the position in the composite source code
if (compilerResults.Errors != null)
compilerResults.Errors.Cast
<CompilerError>().ToList()
.ForEach(x
=> errors.Add(x.FormatError(sourceCode, variableDefinitions)));

//Create field references (i.e. global variables)
if (!errors.Any())
CreateFieldReferences();
compilerErrors
= errors.ToArray();
return !errors.Any();
}

//Execute the compiled function
public T Execute(Dictionary<string, object> variables)
{
if (!IsCompiled)
throw new InvalidOperationException("Function has not been compiled");

//If we have any variable definitions, set the field references to their default values
if (VariableDefinitions != null)
foreach (var kvp in VariableDefinitions)
{
string fieldName = kvp.Key;
Type fieldType
= kvp.Value;
object defaultValue = fieldType.IsValueType ? Activator.CreateInstance(fieldType) : null;
SetVariable(fieldName, defaultValue);
}

//Set the variables passed by the caller
if (variables != null)
foreach (var kvp in variables)
SetVariable(kvp.Key, kvp.Value);
//Invoke the compiled scripted method
return (T)Method.Invoke(Instance, null);
}

//Populate the FieldReferences list so that it can be built into the composite source code
void CreateFieldReferences()
{
if (VariableDefinitions == null)
return;

foreach (var kvp in VariableDefinitions)
{
string fieldName = kvp.Key;
Type fieldType
= kvp.Value;
FieldInfo fieldInfo
= Instance.GetType().GetField(fieldName);
FieldReferences.Add(fieldName, fieldInfo);
}
}

//Set a field (global variable) value
void SetVariable(string variableName, object value)
{
FieldInfo fieldInfo;
if (!FieldReferences.TryGetValue(variableName, out fieldInfo))
throw new ArgumentException(string.Format("Variable {0} has not been defined", variableName));
fieldInfo.SetValue(Instance, value);
}
}
}


And finally the helper class to reverse the line number from the composite script back to the individual script.



using (var compilerService = new CompilerService())
{
string script = @"
public string Main()
{
return ""Hello "" + Name;
}
";

var scripts
= new string[] { script };
var variableDefinitions
= new Dictionary<string, Type>
{
{
"Name" typeof(string) }
};

string fullSourceCode;
IFunction
<string> function;
IEnumerable
<string> compilerErrors;

if (!compilerService.Compile(
scripts: scripts,
variableDefinitions: variableDefinitions,
referencedAssemblies :
null,
function:
out function,
sourceCode:
out fullSourceCode,
compilerErrors:
out compilerErrors))
{
compilerErrors.ToList()
.ForEach(x
=> Console.WriteLine(x));
return;
}

//Now execute it as many times as you like
var variableValues = new Dictionary<string, object>
{
{
"Name", "Bob Monkhouse" }
};
Console.WriteLine(function(variableValues));

variableValues[
"Name"] = "Peter Poppov";
Console.WriteLine(function(variableValues));
}
//Calls CompilerService.Dispose, which unloads all temporary app domains

No comments: