Code generation with Roslyn

I recently wrote an answer on StackOverflow.com regarding what I've learned doing code generation with Roslyn.

My conclusion after a few weeks playing around is that using a combination of inline code snippets (parsed using CSharpSyntaxTree.ParseText) and manually generated SyntaxNodes, works well. But I felt a strong preference for the former. I have also used T4 in the past but am moving away from them due to general lack of integration & capability. I compare the three approaches below and conclude with some tips about doing code gen with Roslyn:

Advantages/disadvantages of each:

Roslyn ParseText

• Generates arguably more readable code-generator code.
• Allows 'text templating' approach e.g. using C# 6 string interpolation.
• Less verbose.
• Guarantees valid syntax trees.
• Can be more performant.
• Easier to get started.
• Text can become harder to read than SyntaxNodes if majority is procedural.

Roslyn SyntaxNode building

• Better for transforming existing syntax trees - no need to start from scratch.
  • But existing trivia can make this confusing/complex.
• More verbose. Arguably harder to read and build.
  • Syntax trees are often more complex than you imagine
• SyntaxFactory API provides guidance on valid syntax.
• Roslyn Quoter helps you transform textual code to factory code.
• Syntax trees are not necessarily valid.
• Code is perhaps more robust once written.

T4 templates

• Good if majority of code to be generated is boiler plate.
• No proper CI support.
• No syntax highlighting or intellisense without 3rd party extensions.
• One to one mapping between input and output files.
  • Not ideal if you are doing more complex generation e.g. entire class hierarchy based on single input.
• Still probably want to use Roslyn to "reflect" on input types, otherwise you will get into trouble with System.Reflection and file locks etc.
• Less discoverable API. T4 includes, parameters etc. can be confusing to learn.

Roslyn code-gen tips

• If you are only parsing snippets of code e.g. method statements, then you will need to use CSharpParseOptions.Default.WithKind(SourceCodeKind.Script) to get the right syntax nodes back.
• If you are parsing a whole block of code for a method body then you will want to parse it as a GlobalStatementSyntax and then access the Statement property as a BlockSyntax.
• Use a helper method to parse single SyntaxNodes:

        private static TSyntax ParseText<TSyntax>(string code, bool asScript = false)
        {
            var options = asScript
                ? CSharpParseOptions.Default.WithKind(SourceCodeKind.Script)
                : CSharpParseOptions.Default;
    
            var syntaxNodes =
                CSharpSyntaxTree.ParseText(code, options)
                    .GetRoot()
                    .ChildNodes();
    
            return syntaxNodes.OfType<TSyntax>().First();
        }

• When building SyntaxNodes by hand you will typically want to make a final call to SyntaxTree.NormalizeWhitespace(elasticTrivia: true) to make the code "round-trippable".
• Typically you will want to use SyntaxNode.ToFullString() to get the actual code text including trivia.
• Use SyntaxTree.WithFilePath() as a convenient place to store the eventual file name for when you come to write out the code.
• If your goal is to output source files, the end game is to end up with valid CompilationUnitSyntaxs.
• Don't forget to pretty-print using Formatter.Format as one of the final steps.

Comments

Diagnosing ASP.NET page compilation errors

Page compilation errors are a common cause of 'yellow screens of death' (YSOD) encountered by ASP.NET developers. During the heat of development it's very easy to ignore the details of these error messages, and instead resolve them with a bit of intuition. But there are limits to intuition. Today I had to drill into one of these errors and I learnt quite a bit along the way.

Are you missing an assembly reference?

I'm going to tackle some of the most common page compilation errors, which relate to missing references and/or incorrect using statements. The screenshot below shows a typical 'CS0234' error:

The compiler error message reads:

The type or namespace name 'Helpers' does not exist in the namespace 'System.Web' (are you missing an assembly reference?).

Page compilation errors

To understand what is going on it's useful to have a quick reminder of how ASP.NET works under the hood.

Every page or view in an ASP.NET site ultimately gets converted into executable code. By default this happens at runtime.

It's a two-step process: Using a Razor view as an example, first the Razor view engine uses code generation to translate the .cshtml file into a .cs file. Then ASP.NET compiles that .cs source into an executable binary.

It's errors occuring during this second step which reveal themselves as YSODs like the one shown above. YSODs *do *actually include useful debug information...

YSOD examined

Clicking 'Show Complete Compilation Source' shows me the full C# source file which has been generated by the view engine for the requested page. Here you can clearly see all the using statements generated by the view engine code generator.

Clicking on 'Show Detailed Compiler Output' shows you exactly how csc.exe (the C# compiler) was invoked by ASP.NET in order to compile the page's generated C# code. Importantly the /R: parameters are the referenced assemblies.

Every C# developer knows that in order to succesfully build an assembly from a module of code you need to make sure all the types you use are made available to the compiler via references.

With normal C# projects you add your own references and write your own using statements. But what determines the references and using statements when code is generated for pages and compiled by ASP.NET?

References

The references used by ASP.NET during page compilation are specified entirely in configuration files using the assemblies element for compilation.

"But I haven't specified an assemblies element!", I hear you say.

The reason you still get some by default is that the root-level web.config file (typically found in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config), that comes distributed with .NET, contains something akin to the following:

    <compilation>
        <assemblies>
            <remove assembly="Microsoft.VisualStudio.Web.PageInspector.Loader, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="Microsoft.VisualStudio.Web.PageInspector.Loader, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="mscorlib" />
            <add assembly="Microsoft.CSharp, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.Web.Services, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
            <add assembly="System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.Runtime.Serialization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.ServiceModel.Activation, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.ServiceModel.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.Activities, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.ServiceModel.Activities, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.WorkflowServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.Data.DataSetExtensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.Xml.Linq, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
            <add assembly="System.ComponentModel.DataAnnotations, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.Web.DynamicData, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="System.Web.ApplicationServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
            <add assembly="*" />
            ...
        </assemblies>
    </compilation>

Unless you make configuration changes this list will match the referenced assemblies you observe 'Show Detailed Compiler Output' section of the YSOD. Excluding one important exception...

Note that the final entry is special. It uses a star '*' instead of providing an assembly name. The remarks on MSDN explain it's purpose:

Optionally, you can specify the asterisk (*) wildcard character to add every assembly within the private assembly cache for the application, which is located either in the \bin subdirectory of an application or in the.NET Framework installation directory (%systemroot%\Microsoft.NET\Framework\version).

The implications of this are that you can easily ensure additional assemblies get referenced during page compilation by placing them in your private assembly cache. And you do that simply by changing the "Copy Local" property of an assembly to true (correlated with the Private reference attribute within an MSBUILD file).

But what if you don't want to add the assembly to your private assembly cache? In that case you just explicity add it to your application-level web.config using the same assemblies.add element shown above.

Namespaces

Where do the default using statements come from for the generated code? Well this actually depends on the view engine you are using. The default view engine (Web Forms) takes its defaults from the root-level web.config under the namespaces element:

    <pages>
        <namespaces>
            <add namespace="System" />
            <add namespace="System.Collections" />
            <add namespace="System.Collections.Generic" />
            <add namespace="System.Collections.Specialized" />
            <add namespace="System.ComponentModel.DataAnnotations" />
            <add namespace="System.Configuration" />
            <add namespace="System.Linq" />
            <add namespace="System.Text" />
            <add namespace="System.Text.RegularExpressions" />
            <add namespace="System.Web" />
            <add namespace="System.Web.Caching" />
            <add namespace="System.Web.DynamicData" />
            <add namespace="System.Web.SessionState" />
            <add namespace="System.Web.Security" />
            <add namespace="System.Web.Profile" />
            <add namespace="System.Web.UI" />
            <add namespace="System.Web.UI.WebControls" />
            <add namespace="System.Web.UI.WebControls.WebParts" />
            <add namespace="System.Web.UI.HtmlControls" />
            <add namespace="System.Xml.Linq" />
        </namespaces>
        ...
    </page>

Namespaces with Razor

If you are using Razor the default namespaces are hardcoded!

You can easily add to the list of default namespaces by adding a razor-specific namespaces element to your web.config (sample shown below), but you cannot easily remove default namespaces. Even attempting to use <clear/> does not work. If you really need to get rid of default namespaces you will have to do it in code.

<system.web.webPages.razor>
  <host factoryType="System.Web.Mvc.MvcWebRazorHostFactory, System.Web.Mvc, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
  <pages pageBaseType="System.Web.Mvc.WebViewPage">
   <namespaces>
    <add namespace="System.Web.Mvc" />
    <add namespace="System.Web.Mvc.Ajax" />
    <add namespace="System.Web.Mvc.Html" />
    <add namespace="System.Web.Optimization"/>
    <add namespace="System.Web.Routing" />
   </namespaces>
  </pages>
</system.web.webPages.razor>

A note about Web.configs

Bare in mind that projects often have a web.config specifically for the Views folder (e.g. MVC) and the page compiler will pay attention to the config heirarchy. Typically you will want to makes changes specific to your Views web.config.

Conclusion

Using the information in this blog post you should be able to understand and resolve any page compilation errors relating to missing references and/or incorrect using statements. Good luck!

Comments

The Economics of Microsoft Surface

This is a post I wrote August 2009 but never got around to publishing. Whilst spring cleaning my drafts, I re-read it and decided it wasn't that bad, so here it is. I wonder if the issues are still the same in 2011....

---

It’s almost a year ago to the month that Amnesia Razorfish smuggled the first Microsoft Surface tables into Australia with the help of parent company Razorfish, much to the excitement of local geeks. There was also a fair bit of uproar from surprised parties who hadn’t expected to be usurped in such spectacular fashion.

I was hired by Amnesia shortly before Surface arrived, partly on speculation that they would soon need a Surface developer. I had experience with the technologies it uses, and a great interest in the product, so was really excited to get such an opportunity. It was also a bit of a gamble on my part, as it was by no means guaranteed at the time they would even be getting a table, let alone any paying clients.

The first few weeks, maybe months, there was a great buzz and curiosity from clients and media about this new device. The phones were ringing off the hook and there were regular coffee table tours.

Despite this, and the amazing stock of talent Amnesia had at its disposal, I think it’s safe to say that during my year there, they never needed more than this one, solitary, Surface developer.

How can that be? “Surely it would be as easy as selling antifreeze to Eskimos!”, I hear you mutter…

Here’s why: In many scenarios, the economics of Surface development simply don’t add up.

For background, let’s put a few ballpark figures on the coffee table:

• Cost of a MS Surface table: $10k
• Cost of  development/design time: $200/hour

Let’s tackle the most obvious thing first: This Is Software Development.

It’s been many years since the Adobe Flash experiences that draw crowds on the web have been built entirely with a visual tool. Most are chock full of ActionScript. In fact, dare I say, they probably contain more programming hours than design hours.

Suffice to say there are very few digital experiences of the richness that people expect these days, that can be built without significant software development capabilities, and that goes for Microsoft Surface too.

Microsoft Surface carries the extra burden of not only having slightly less mature tools than industry mainstay Adobe Flash, but more importantly having a much smaller experience pool, particularly in the crucial “devigner” niche. And as with any new technology, the early days often mean higher costs and higher risks.

So, creating a quality Surface experience means building software. But what do you get for your money? Well it could easily cost you $6000 per man week, and even after many weeks development the chances are you won’t get that award winning experience you were hoping for. My guess is that most commercial Surface applications you see being touted cost in the region of $300,000. Definitely in the hundreds of thousands. Let your imagination go wild and you could easily spend half a million.

The next most obvious thing is: Penetration.

Unlike building a user experience for the web, prospective clients will be footing the bill for each and every device that will end up in front of users. If you are going to pay a hefty sum for bespoke software development you sure as hell want to leverage that investment by deploying it to the maximum number of locations.

You can keep the costs low by only having a tiny number of tables (and maybe even spin that to your boss by claiming the experience would then be truly unique/exclusive!), but ultimately you are talking about investing in a piece of custom-written software that will only ever be seen by a tiny audience. Yes, that might be tenable for a small number of luxury brands, but for the majority who actually need to justify there expenditure? I think not.

So, to get “good value” from your investment in a Surface experience you need to also purchase a reasonably large number of tables. I would say at least in the tens if not hundreds (e.g. retail locations). There’s $500,000 right there.

Lastly: Cost/benefit.

At the end of the day, clients have to estimate how much a Surface experience is going to be worth to them. It’s obvious that many applications of Surface can be categorized as marketing in some sense; whether its experiential branding, attractive gimmick, or 9 o’clock news hype.

I can only imagine it’s very hard to estimate the financial benefits such a Surface experience will bring, because in the majority of cases it’s going to be an indirect effect at best. And such financial benefits are almost certainly going to be impossible to determine a-priori.

Which is also the case if Surface is used as Point-of-Sale platform – it’s all pretty much undiscovered country. With such little data on natural user interfaces, not to mention Surface specifically, it’s very hard at this stage to make confident guesses about how conversion rates or stickiness can be improved. The only way to find out for sure is by putting down the cash. And that’s a risk many firms can’t or wont take.

Comments