Sunny Ahuwanya's Blog

Mostly notes on .NET and C#

Introducing C# Pad

I’m excited to present C# Pad, an interactive web based C# REPL.

Have you ever wanted to quickly evaluate an expression or test some code, like say try out different DateTime string formats or test a method or clear up some confusion (like what’s the difference between Uri.EscapeDataString and Uri.EscapeUriString or what new Random().Next(0) returns), or decode some string in Base64 or some other format?

C# Pad lets you easily do all those things and a whole lot more.

Interactive REPL

Do you see the embedded code pad below? Go ahead and hit the Go button.

var greeting = "こんにちは世界";
Console.WriteLine("{0} in Japanese is {1}", "Hello World", greeting);

After the submission is processed, you’ll see the result of the code evaluation.
Now, in the same code pad, type Console.WriteLine(greeting.Length) and hit Go.

As you can see, the greeting variable from the previous submission is accessible in the code editor.
That’s because C# Pad is a REPL. Objects in previous submissions are visible and accessible from the current one.
( Did you also notice the cool code completion? :) )

You don’t need to call Console.WriteLine to display results. Simply type the variable name (without a semicolon) in the last line of a code submission and the string representation of the variable’s value will be displayed. 
For example, type greeting in the same code pad and hit Go to see its value.

The following code pad contains a method that encodes an input string in Base64 format.

using System.IO;
using System.Text;

string Base64Encode(string input)
{
    if(input == null) throw new ArgumentNullException("input");
    
    var bytes = Encoding.UTF8.GetBytes(input);
    return Convert.ToBase64String(bytes);
}

Hit Go to submit the method and then type Base64Encode("My string") (do not include a semicolon) and hit Go.

You’ll observe the result is a string in Base64 format, encoded by the method that was defined in an earlier submission. Because the semicolon was left out, the result of the evaluation was displayed.

You can use C# Pad to write complex code, define classes and methods and evaluate all kinds of expressions ranging from simple mathematical expressions like 60 * 60 * 24 or Math.Sin((30 * Math.PI)/ 180) to LINQ expressions.

(from c in "The quick brown fox jumps over the lazy dog".ToLower()
group c by c into grp
select new { Letter = grp.Key, Count = grp.Count() }).ToArray()

C# Everywhere

You can embed interactive code pads in your blog, website or any site you can edit, just like I have in this blog post. You can even select a theme that matches your site.
Yes, this means you can now create a C# playground anywhere. Simply visit csharppad.com, compose the code you’d like to embed, click the embed button and follow the instructions.  

Here’s an example JSBin with an embedded code pad.

You can also load and save Github Gists.
To load a Gist, open csharppad.com/gist/_gist_owner_/_gist_id or simply csharppad.com/gist/_gist_id

As examples, the links below open up Gists in C# Pad.

http://csharppad.com/gist/9220821
http://csharppad.com/gist/octocat/1169852 ( Did you know The Octocat codes in C#? )

To open a single file in a Gist, append ?file_name at the end of the url, like so
http://csharppad.com/gist/9220821?random.cs

Numerical Analysis

The awesome Math.Net Numerics library is bundled with C# Pad, which makes working on complex math and statistics problems a breeze.

The following program integrates the function f(x) = exp(-x/5) (2 + sin(2 * x)) on a closed interval of [0, 100].

using MathNet.Numerics;
Integrate.OnClosedInterval(x => Math.Exp(-x / 5) * (2 + Math.Sin(2 * x)), 0, 100)


The following program generates ten samples of a Poisson distribution with a lambda parameter of 1.

using MathNet.Numerics.Distributions;
var poisson = new Poisson(1);
for(var i = 0; i < 10; i++)
{
    Console.WriteLine("{0:N05}", poisson.Sample());
}


The following program calculates the 75th percentile of an array of numbers.

using MathNet.Numerics;

var percentile = 75;
var array = new double[]{ 89.6, 33.5, 11.6, 44.3, 66.78, 34.78, 97.1, 68.0, 25.7, 48.7};

ExcelFunctions.Percentile(array, percentile / 100d)


The following program multiplies the transpose of a matrix with another matrix.

using MathNet.Numerics.LinearAlgebra.Double;

var matrixA = DenseMatrix.OfArray(new[,] { { 1.0, 2.0, 3.0 }, { 4.0, 5.0, 6.0 }, { 7.0, 8.0, 9.0 } 
});
var matrixB = DenseMatrix.OfArray(new[,] { { 1.0, 3.0, 5.0 }, { 2.0, 4.0, 6.0 }, { 3.0, 5.0, 7.0 } });

matrixA.Transpose() * matrixB

//Same as matrixA.TransposeThisAndMultiply(matrixB)

 

C# Pad is the first and only (as of this writing) web based interactive C# REPL, with code completion, diagnostics, themes, embeddability, timing information, Gist support and more features to come.

I hope you find C# Pad useful and delightful. Drop me a line or use the help link on the site to provide feedback on feature suggestions, bug reports and other kinds of feedback.

Have fun!

Enumerating AppDomains from a CLR Host

If you’ve ever tried to enumerate the appdomains in your application, you would have discovered that it’s not exactly an easy task. It’s even more difficult if you try to enumerate the appdomains from an unmanaged CLR host.
I was faced with this task recently and decided to blog about how circuitous it can be.

The obvious solution is to use the ICorRuntimeHost::EnumDomains method, however, the ICorRuntimeHost hosting interface was deprecated and superceded by the CLR v2 ICLRRuntimeHost interface, which does not provide a means to enumerate appdomains.

That’s okay, there are other options available, one of which is to use the ICorPublishProcess::EnumAppDomains method.
To do so, first get an ICorPublishProcess instance that represents the current process, like so:

ICorPublish* pPub = NULL;
ICorPublishProcess* pCurrProcess = NULL;

HRESULT hr = CoCreateInstance (CLSID_CorpubPublish, NULL, CLSCTX_INPROC_SERVER, IID_ICorPublish,(LPVOID *)&pPub);

DWORD cpid = GetCurrentProcessId();
hr = pPub->GetProcess(cpid, &pCurrProcess);

And then pass that instance to the following function.

void EnumAppDomains (ICorPublishProcess *pProcess)
{
    #define NAME_LEN 261

    // Enumerate the application domains.
    ICorPublishAppDomainEnum* pEnumDomains;
    HRESULT hr = pProcess->EnumAppDomains(&pEnumDomains);
    if (SUCCEEDED(hr))
    {
        ICorPublishAppDomain* appDomains[1];
        ULONG aFetched = 1;
		ULONG32 nId;
        while (aFetched > 0 && pEnumDomains->Next(1,appDomains, &aFetched) == S_OK)
        {
			// Display information about each domain.
			if (aFetched > 0)
			{
				appDomains[0]->GetID(&nId);

				WCHAR name[NAME_LEN];
				ULONG32 size=0;
				appDomains[0]->GetName(NAME_LEN, &size, name);
				if (size > 0)
				{
					wprintf(L"name = %s , id = %Iu\n", name, nId);
				} 
				appDomains[0]->Release();
			}

        }
        pEnumDomains->Release();
    }
}

So far, so good. Looks like we’re done here except that there are a few lingering issues:

You have to make sure you are instantiating the right version of ICorPublish for the version of the CLR you’re running, or else you’ll run into some issues as discussed here and here.

The more significant drawback is that you have to call CoCreateInstance as seen in the first code listing. This means you are creating a COM object and you really should call CoInitializeEx(NULL, COINIT_MULTITHREADED) when your host starts, to properly initialize the the COM library.
You’re now knee deep in COM land and have to deal with COM related issues if you choose this route.

An approach that avoids initializing COM involves using the ICLRDebugging interface which can be instantiated with the CLRCreateInstance function. Once instantiated, make a call to OpenVirtualProcess method to obtain an ICorDebugProcess interface and then call ICorDebugProcess::EnumerateAppDomains to enumerate the appdomains in the process.

I don’t like this approach because it’s a *debugging* API and seems excessive for the objective.
So what other options do we have left? Let’s revisit the ICorRuntimeHost interface. It’s deprecated but that doesn’t mean it can no longer be used.
Using this interface is pretty straightforward and there are good examples (.NET 2 version) available on how to use it. The only snag is that my application already makes use of the newer ICLRRuntimeHost interface and I don’t want to replace it with the ICorRuntimeHost interface. I need the older interface to enumerate appdomains and the newer interface for all other hosting related tasks.

Fortunately, you can obtain the two interfaces from the same runtime as shown in this MSDN forum discussion. Here’s an excerpt of the relevant code.

#include"stdafx.h"
#include<mscoree.h>
#include<metahost.h>
#pragmacomment(lib, "mscoree.lib")

int _tmain(int argc, _TCHAR* argv[])
{
	HRESULT hr;
	ICLRMetaHost *pMetaHost = NULL;
	hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, (LPVOID*)&pMetaHost);
	ICLRRuntimeInfo * lpRuntimeInfo = NULL;
	hr = pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, (LPVOID*)&lpRuntimeInfo);
	ICLRRuntimeHost * lpRuntimeHost = NULL;
	hr = lpRuntimeInfo->GetInterface(CLSID_CLRRuntimeHost, IID_ICLRRuntimeHost, (LPVOID *)&lpRuntimeHost);
	hr = lpRuntimeHost->Start();
	ICorRuntimeHost* lpCorRuntimeHost = NULL;
	hr = lpRuntimeInfo->GetInterface(CLSID_CorRuntimeHost, IID_ICorRuntimeHost, (LPVOID *)&lpCorRuntimeHost);
	// To do.
	return 0;
}

One thing I discovered while trying out the code above is that you have to start the runtime first ( lpRuntimeHost->Start() ) before obtaining the second interface. All my attempts to obtain the second interface before starting the runtime failed.

If you don’t have a pointer to the runtime, you can inspect the process using the ICLRMetaHost interface to get the loaded runtimes as shown below.

HRESULT GetV1CorRuntimeHost(ICorRuntimeHost** ppCorRuntimeHost){

    IEnumUnknown* pRuntimeEnum = NULL;
    ICLRMetaHost* pMetaHost = NULL;
    IUnknown* pClrRunTimeInfoThunk = NULL;
    ICLRRuntimeInfo* pClrRunTimeInfo = NULL;

    //Get the ICLRMetaHost interface
    HRESULT hr = CLRCreateInstance(CLSID_CLRMetaHost, IID_PPV_ARGS(&pMetaHost));
    if (FAILED(hr))
    {
        wprintf(L"GetV1CorRuntimeHost: CLRCreateInstance CLRMetaHost failed w/hr 0x%08lx\n", hr);
		goto Cleanup;
    }

    //Get the runtime enumerator
    hr = pMetaHost->EnumerateLoadedRuntimes(GetCurrentProcess(),&pRuntimeEnum);
	if (FAILED(hr))
    {
        wprintf(L"GetV1CorRuntimeHost: EnumerateLoadedRuntimes failed w/hr 0x%08lx\n", hr);
		goto Cleanup;
    }

    //Get the first loaded runtime
    ULONG fetched = 0;
    hr = pRuntimeEnum->Next(1, &pClrRunTimeInfoThunk, &fetched);
    if (FAILED(hr))
    {
        wprintf(L"GetV1CorRuntimeHost: pRuntimeEnum->Next failed w/hr 0x%08lx\n", hr);
		goto Cleanup;
    }

    if(fetched != 1)
    {
	wprintf(L"GetV1CorRuntimeHost: Runtime could not be fetched");
	hr = E_FAIL;
	goto Cleanup;
    }

    //Get the ICLRRuntimeInfo object
    hr = pClrRunTimeInfoThunk->QueryInterface(IID_PPV_ARGS(&pClrRunTimeInfo));
    if (FAILED(hr))
    {
        wprintf(L"GetV1CorRuntimeHost: Failed to get the ICLRRuntimeInfo object w/hr 0x%08lx\n", hr);
		goto Cleanup;
    }

    //Get the ICorRuntimeHost interface
    *ppCorRuntimeHost = NULL;
    hr = pClrRunTimeInfo->GetInterface(CLSID_CorRuntimeHost, 
    IID_PPV_ARGS(ppCorRuntimeHost));
    if (FAILED(hr))
    {
        wprintf(L"GetV1CorRuntimeHost: GetInterface(CLSID_CorRuntimeHost) failed w/hr 0x%08lx\n", hr);
	goto Cleanup;
    } 

    hr = S_OK;

Cleanup:

    if(pClrRunTimeInfo) pClrRunTimeInfo->Release();
    if(pClrRunTimeInfoThunk) pClrRunTimeInfoThunk->Release();
    if(pRuntimeEnum) pRuntimeEnum->Release();
    if(pMetaHost) pMetaHost->Release();

	return hr;
	
}

Obtaining the ICorRuntimeHost interface is now as easy as

ICorRuntimeHost *pCorRuntimeHost = NULL;
HRESULT hr = GetV1CorRuntimeHost(&pCorRuntimeHost);

Henceforth, it’s easy to enumerate the appdomains as shown in this blog post.
Here’s an excerpt of the relevant code (substituting the ICorRuntimeHost variable name to match the name used in this post)

// Enumerate the AppDomains
HDOMAINENUM adEnum;
hr = pCorRuntimeHost->EnumDomains(&adEnum);
EXITONERROR(hr, "Unable to enumerate AppDomains");

// Loop thru the domains
IUnknown * pDomainUnk = NULL;
hr = pCorRuntimeHost->NextDomain(adEnum, &pDomainUnk);
while(SUCCESSFUL(hr))
{
    // Got the IUnknown* to the AppDomain - convert it to AppDomain pointer
    _AppDomain * pCurDomain = NULL;
    hr = pDomainUnk->QueryInterface(__uuidof(_AppDomain), (VOID**)&pCurDomain);
    if (SUCCESSFUL(hr))
    {
        // Display the name of the AppDomain
        BSTR str;
        if (SUCCESSFUL(pCurDomain->get_FriendlyName(&str)))
        {
        wprintf(L"AppDomain: %s\n",str);
        }
        else
        {
            printf("AppDomain: unable to get the name!\n");
        }
    } 

    // Loop onto the next Domain
    hr = pCorRuntimeHost->NextDomain(adEnum, &pDomainUnk);
}

StackOverflowException and the x64 platform

The StackOverflowException is one of the three deadly exceptions. The other two being the OutOfMemoryException and the ThreadAbortException. These exceptions are deadly because they can theoretically occur anywhere in an application and terminate the process. Also, developers hardly ever prepare for them and are caught unawares when they occur.

Of these three, the StackOverflowException is arguably the deadliest because it cannot be caught when it is thrown by the runtime. 
That's right. When a StackOverflowException is thrown by the runtime (not when it is thrown by the application), try blocks are not processed and the program is promptly terminated. This is the default behavior and cannot be changed unless there is a custom CLR host involved.

This is conventional .NET wisdom that has been time tested and proven, so you can imagine my surprise when a condition that should have caused a StackOverflowException and subsequent process termination, did not.

If you compile following code, target the x86 platform in release mode and run it from your desktop:

class Program
{
    static void Main(string[] args)
    {
        CauseStackOverflow();
    }

    static void CauseStackOverflow()
    {
        CauseStackOverflow();	
    }
}

The program will terminate after encountering a StackOverflowException.

image

However, if you compile the same program targeting the x64 platform in release mode, and run it from your desktop (not from Visual Studio which attaches a debugger), the program keeps running. The StackOverflowException is not thrown and the program never terminates.

After some investigation, it turns out that the JIT compiler on the x64 will perform tail call optimization for such code and so will never overflow the stack!

These blog posts by David Broman go into detail on tail call optimization in the CLR.

If I create a condition that prevents the optimization like modifying the recursed method to return a value type:

class Program
{
    static void Main(string[] args)
    {
        CauseStackOverflow();
    }

    static int CauseStackOverflow()
    {
        return CauseStackOverflow();
	
    }
}

The StackOverflowException occurs on the x64 platform and the process terminates as expected.

Testing the BufferPool Project

One of the challenges I faced while working on the BufferPool project was verifying that it worked perfectly. Although I wrote a bunch of unit tests to assert the right behavior, I couldn’t conclude that it actually ensured that buffers were pinned within a pre-defined contiguous memory block.

To confirm this, I manually compared memory dumps from a process that used the library with dumps from another one that did not.

Two programs were used to perform this test. The first program is a variant of the asynchronous socket listener sample on MSDN, modified to call GC.Collect upon accepting a fifth connection. It also creates a long empty byte array on each new connection to simulate other memory allocations that typically take place during the read callback (which isn’t triggered during the test).

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;

namespace BufferPinningTester
{

    public class AsynchronousSocketListener
    {
        public static ManualResetEvent allDone = new ManualResetEvent(false);
        static int connectCount = 0;

        public static void StartListening()
        {
            // Establish the local endpoint for the socket.
            // The DNS name of the computer
            // running the listener is "host.contoso.com".
            IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
            IPAddress ipAddress = ipHostInfo.AddressList[0];
            IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);

            // Create a TCP/IP socket.
            Socket listener = new Socket(AddressFamily.InterNetwork,
                SocketType.Stream, ProtocolType.Tcp);

            // Bind the socket to the local endpoint and listen for incoming connections.
            try
            {
                listener.Bind(localEndPoint);
                listener.Listen(100);

                while (true)
                {
                    // Set the event to nonsignaled state.
                    allDone.Reset();

                    // Start an asynchronous socket to listen for connections.
                    Console.WriteLine("Waiting for a connection...");
                    listener.BeginAccept(
                        new AsyncCallback(AcceptCallback),
                        listener);

                    // Wait until a connection is made before continuing.
                    allDone.WaitOne();
                    connectCount++;

                    if (connectCount == 5)
                    {
                        GC.Collect();
                        Console.WriteLine("GC Collected");
                    }
                    
                }

            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }

            Console.WriteLine("\nPress ENTER to continue...");
            Console.Read();

        }

        public static void AcceptCallback(IAsyncResult ar)
        {

            Console.WriteLine("In AcceptCallback");
            // Signal the main thread to continue.
            allDone.Set();

            // Get the socket that handles the client request.
            Socket listener = (Socket)ar.AsyncState;
            Socket handler = listener.EndAccept(ar);

            // Create the state object.
            StateObject state = new StateObject();
            //create some random array
            byte[] b = new byte[StateObject.BufferSize];
            state.workSocket = handler;
            handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                new AsyncCallback(ReadCallback), state);
        }

        public static void ReadCallback(IAsyncResult ar)
        {
            Console.WriteLine("In ReadCallback");

            String content = String.Empty;

            // Retrieve the state object and the handler socket
            // from the asynchronous state object.
            StateObject state = (StateObject)ar.AsyncState;
            Socket handler = state.workSocket;

            
            // Read data from the client socket. 
            int bytesRead = handler.EndReceive(ar);

            return;

        }

        public static int Main(String[] args)
        {
            StartListening();
            return 0;
        }
    }


// State object for receiving data from remote device.
public class StateObject {
    // Client socket.
    public Socket workSocket = null;
    // Size of receive buffer.
    public const int BufferSize = 16000;
    // Receive buffer.
    public byte[] buffer = new byte[BufferSize];
    // Received data string.
    public StringBuilder sb = new StringBuilder();
}



}

The second program is a version of the first program modified to use a buffer pool. In this program, The 16KB buffer allocations are managed by the pool.

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using ServerToolkit.BufferManagement;

namespace BufferPinningTester
{

    public class AsynchronousSocketListener
    {
        public static ManualResetEvent allDone = new ManualResetEvent(false);
        static int connectCount = 0;
        static BufferPool pooledBuffers = new BufferPool(80000, 1, 1);
        

        public static void StartListening()
        {
            // Establish the local endpoint for the socket.
            // The DNS name of the computer
            // running the listener is "host.contoso.com".
            IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
            IPAddress ipAddress = ipHostInfo.AddressList[0];
            IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);

            // Create a TCP/IP socket.
            Socket listener = new Socket(AddressFamily.InterNetwork,
                SocketType.Stream, ProtocolType.Tcp);

            // Bind the socket to the local endpoint and listen for incoming connections.
            try
            {
                listener.Bind(localEndPoint);
                listener.Listen(100);

                while (true)
                {
                    // Set the event to nonsignaled state.
                    allDone.Reset();

                    // Start an asynchronous socket to listen for connections.
                    Console.WriteLine("Waiting for a connection...");
                    listener.BeginAccept(
                        new AsyncCallback(AcceptCallback),
                        listener);

                    // Wait until a connection is made before continuing.
                    allDone.WaitOne();
                    connectCount++;

                    if (connectCount == 5)
                    {
                        GC.Collect();
                        Console.WriteLine("GC Collected");
                    }
                    
                }

            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }

            Console.WriteLine("\nPress ENTER to continue...");
            Console.Read();

        }

        public static void AcceptCallback(IAsyncResult ar)
        {

            Console.WriteLine("In AcceptCallback");
            // Signal the main thread to continue.
            allDone.Set();

            // Get the socket that handles the client request.
            Socket listener = (Socket)ar.AsyncState;
            Socket handler = listener.EndAccept(ar);

            // Create the state object.
            StateObject state = new StateObject();
            //create some random array
            byte[] b = new byte[StateObject.BufferSize];
            state.workSocket = handler;
            IBuffer readBuffer = pooledBuffers.GetBuffer(StateObject.BufferSize);
            handler.BeginReceive(readBuffer.GetArraySegment(), 0,
                new AsyncCallback(ReadCallback), state);
        }

        public static void ReadCallback(IAsyncResult ar)
        {
            Console.WriteLine("In ReadCallback");

            String content = String.Empty;

            // Retrieve the state object and the handler socket
            // from the asynchronous state object.
            StateObject state = (StateObject)ar.AsyncState;
            Socket handler = state.workSocket;

            
            // Read data from the client socket. 
            int bytesRead = handler.EndReceive(ar);

            return;

        }

        public static int Main(String[] args)
        {
            StartListening();
            return 0;
        }
    }


// State object for receiving data from remote device.
public class StateObject {
    // Client socket.
    public Socket workSocket = null;
    // Size of receive buffer.
    public const int BufferSize = 16000;
    // Receive buffer.
    //public byte[] buffer = new byte[BufferSize];
    // Received data string.
    public StringBuilder sb = new StringBuilder();
}



}

The test procedure consists of running the socket listener program and connecting to it from five telnet clients simultaneously.
A memory snapshot is taken before the fifth connection is made and another snapshot is taken after it is made. Both snapshots are then compared.
The test is conducted twice, once for each socket listener program.
To take these snapshots, you’ll need to use WinDbg debugger tool coupled with the Son of Strike (SOS) debugging extension.

In Detail:

  • Run the first socket listener program.
  • Run WinDbg and attach the socket listener process.
  • Load Son of Strike (SOS) from WinDbg by executing .loadby sos clr (if you are working with .NET 2.0/3.5, execute .loadby sos mscorwks instead).
  • Execute g to continue running the program.
  • Connect four telnet clients to port 11000 (the socket listener port).
    While running telnet, it’s important not to hit any key after telnet makes the connection, so as not to transmit any data and keep the buffer pinned. If you see a "In ReadCallback" message in the socket listener console window, it means data was transmitted – we don’t want that.
  • Press ctrl-break in WinDbg to halt execution of the program.
  • Execute the !gchandles command to get information about asynchronous pinned handles.
  • Execute the !dumpheap command to get a memory snapshot of objects in the heap.
  • Execute g to continue running the program.
  • Connect the fifth telnet client.
  • You should see "GC Collected" in the socket listener program console window.
  • Press ctrl-break in WinDbg to halt execution of the program.
  • Execute the !gchandles command to get information about asynchronous pinned handles.
  • Execute the !dumpheap command to get a memory snapshot of objects in the heap.
  • Run the test again, this time for the second socket listener program.

A sample output from !gchandles is shown below. The number of asynchronous pins are highlighted.

0:005> !gchandles
*********************************************************************
* Symbols can not be loaded because symbol path is not initialized. *
*                                                                   *
* The Symbol Path can be set by:                                    *
*   using the _NT_SYMBOL_PATH environment variable.                 *
*   using the -y <symbol_path> argument when starting the debugger. *
*   using .sympath and .sympath+                                    *
*********************************************************************
PDB symbol for mscorwks.dll not loaded
GC Handle Statistics:
Strong Handles: 37
Pinned Handles: 6
Async Pinned Handles: 4
Ref Count Handles: 0
Weak Long Handles: 35
Weak Short Handles: 10
Other Handles: 0
Statistics:
              MT    Count    TotalSize Class Name
000007fef8ac7370        1           24 System.Object
000007fef92a2f88        1           32 System.Threading.RegisteredWaitHandle
000007fef8ac87c0        1           48 System.SharedStatics
000007fef8ac8078        1          136 System.ExecutionEngineException
000007fef8ac7f68        1          136 System.StackOverflowException
000007fef8ac7e58        1          136 System.OutOfMemoryException
000007fef8ac8980        1          192 System.AppDomain
000007fef92a3040        5          200 System.Threading._ThreadPoolWaitOrTimerCallback
000007fef8aca540        5          240 System.Reflection.Assembly
000007fef8ab5d68        5          240 System.Threading.ManualResetEvent
000007fef76be9b0        4          256 System.Net.Logging+NclTraceSource
000007fef8ac8188        2          272 System.Threading.ThreadAbortException
000007fef76d59a8        4          288 System.Diagnostics.SourceSwitch
000007fef8ac43f8        4          384 System.Reflection.Module
000007fef8ac8520        4          416 System.Threading.Thread
000007fef8ace270        7          448 System.Security.PermissionSet
000007fef8699058        4          480 System.Threading.OverlappedData
000007fef8ac4558       35         5040 System.RuntimeType+RuntimeTypeCache
000007fef8ab5870        6        33856 System.Object[]
Total 92 objects
  

You now have useful output from the !dumpheap and !gchandles commands for both programs.
You can download the !dumpheap and !gchandles output from my tests here.
You’ll notice that the output from dumpheap after garbage collection is much smaller than the output before garbage collection. That’s because garbage collection got rid of stale objects in memory.

If you compare the two !dumpheap output listings from the first (unpooled) test, you can easily verify that all memory addresses that hold byte arrays that are at least 16,000 bytes long were collected by the GC except for four addresses. You can tell that a memory address holds a byte array if its MT (MethodTable) address is the one for System.Byte[] in the statistics section at the bottom of the !dumpheap listings. On my system, the MT for System.Byte[] is 000007fef8acfac0.
The output from !gchandles shows that there were four asynchronous pins before garbage collection and five after garbage collection. This observation strongly suggests that those empty byte arrays that were created were collected while the pinned buffer byte arrays were not.

Now let’s take a look at the memory behavior of the pooled test by comparing its two !dumpheap output listings before and after garbage collection. The desired behavior with pooled buffers is that all pinned addresses are within one long byte array.
You’ll observe that all byte arrays in the heap are collected except one, and that one is 92 KB long.
The pooler creates a minimum pool size of 92 KB so as to force .NET to allocate the array in the large object heap.
The output from !gchandles shows that there were four asynchronous pins before garbage collection and five after garbage collection. Since only byte arrays were pinned, it means they all occurred within the 92k byte array block!
We have verified that the pooler is working correctly.

To see details of the 92KB block, run the !dumpobj command on the block’s memory address:

0:005> !dumpobj 00000000127b9060
Name: System.Byte[]
MethodTable: 000007fef8acfac0
EEClass: 000007fef86d2680
Size: 92184(0x16818) bytes
Array: Rank 1, Number of elements 92160, Type Byte
Element Type: System.Byte
Fields:
None
  

Exploring System.Void Part II

In my previous post, I described the role of System.Void in the .NET framework and demonstrated some of the restrictions placed on the type.
In this post, I’ll postulate why those restrictions are in place and I’ll try to bypass them in an attempt to create an instance of the type.

The restrictions on the System.Void type are documented in the Common Language Infrastructure (CLI) specification (ECMA-335)PDF Document.
I find two of these restrictions particularly interesting:

“The type System.Void is never boxable.” (Section 8.2.4)
“No location or value shall have System.Void” (Section 8.7)

Why is instantiation of the System.Void type forbidden? After all, as long as methods are not allowed to declare a return type of System.Void, the purpose of the type as discussed in my previous post remains intact, regardless of whether instances are allowed to exist or not.

So, what’s the harm in permitting instances of this type to exist? I proffer the following explanations.

Firstly, from a philosophical standpoint, it doesn’t make much sense to have the ability to instantiate a type that methods cannot declare as a return type, because that is effectively saying that you can create this object but your methods cannot pass its value (unless in a boxed representation or masqueraded as an inherited type) around.

Secondly, some unexpected issues can arise when calling methods declared in generic types. For example, if developers can instantiate System.Void, they’ll expect to be able to compile and run the following code, but by so doing produce a method that has a System.Void return type.

class Program
{
	static void Main(string[] args)
	{
		var instance = new MyClass<System.Void>().GetInstance();			
	}
	
}

class MyClass<T> where T:new()
{
	public T GetInstance()
	{
		return new T();
	}
}

The compiler/runtime can be updated to prevent such code from compiling or running, however it’s cleaner to have a rule that states that System.Void cannot be instantiated and that the expression “System.Void” is illegal anywhere in C#.

The last explanation I proffer has to do with pointers in C#, and I think it is the most functional explanation.
You can get the System.Type object of a value type pointer with the following expression:

typeof(MyStruct*)
For example:
typeof(int*)
evaluates to a System.Type object called System.Int32*

Similarly, typeof(bool*), typeof(DateTime*), typeof(char*) and typeof(System.Guid*) will evaluate to System.Type objects called System.Boolean*, System.DateTime*, System.Char* and System.Guid* respectively.

Now, there is a special kind of pointer called the void pointer which can point to any pointer type.
The type declaration for a void pointer is void*, and the following expression gets the System.Type object of a void pointer:

typeof(void*)
It evaluates to a System.Type object called System.Void*

You can already see how an ambiguity is introduced if instances of System.Void were allowed to exist.
In that scenario, typeof(System.Void*) would evaluate to a System.Type object also called System.Void*.
It would be impossible (or at least error-prone) to tell if the object is for a pointer to System.Void or for the special void pointer.

Nevertheless, and in spite of my failed attempts to instantiate System.Void in the previous post, it is possible to create an instance of System.Void -- by making it a static field.

// Metadata version: v2.0.50727
.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
  .ver 2:0:0:0
}
.assembly ConsoleApplication1
{
  .hash algorithm 0x00008004
  .ver 1:0:0:0
}
.module ConsoleApplication1.exe
// MVID: {60F1E74C-3123-4EF0-8269-A0C3B8B85ADF}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x00670000


// =============== CLASS MEMBERS DECLARATION ===================

.class private auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
  .field public static valuetype [mscorlib]System.Void o
  .method private hidebysig static void  Main() cil managed
  {
    .entrypoint
    // Code size       1 (0x1)
    .maxstack  8
    IL_0000:  ret
  } // end of method Program::Main

  .method public hidebysig specialname rtspecialname 
          instance void  .ctor() cil managed
  {
    // Code size       7 (0x7)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
    IL_0006:  ret
  } // end of method Program::.ctor

} // end of class Program

The IL code listing above is equivalent to the following C# program (if it were possible to compile)

class Program
{
	public static System.Void o;
	static void Main()
	{
	}
}

If I compile the IL code using ilasm and run it, I don’t observe any errors, unlike in my previous attempts.
What does this mean? Did it create an instance of the type? Is the runtime skipping over that System.Void field creation instruction? Is the program dying silently?

This is a good example of the "If a tree falls in a forest and no one is around to hear it, does it make a sound?" philosophical riddle.

What I need is some observability. I need to be absolutely sure that the type was instantiated (or not).

I’ll just modify the code a little bit to get some output:

class Program
{
	public static System.Void o;
	static void Main()
	{
		Console.WriteLine(o.ToString());

	}
}

which translates to the following IL code:

// Metadata version: v2.0.50727
.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
  .ver 2:0:0:0
}
.assembly ConsoleApplication1
{
  .hash algorithm 0x00008004
  .ver 1:0:0:0
}
.module ConsoleApplication1.exe
// MVID: {EE60F7FD-BD3D-46C8-B353-A459F5119496}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x00260000


// =============== CLASS MEMBERS DECLARATION ===================

.class private auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
  .field public static valuetype [mscorlib]System.Void o
  .method private hidebysig static void  Main() cil managed
  {
    .entrypoint
    // Code size       22 (0x16)
    .maxstack  8
    IL_0000:  ldsflda    valuetype [mscorlib]System.Void Program::o
    IL_0005:  constrained. [mscorlib]System.Void
    IL_000b:  callvirt   instance string [mscorlib]System.Object::ToString()
    IL_0010:  call       void [mscorlib]System.Console::WriteLine(string)
    IL_0015:  ret
  } // end of method Program::Main

  .method public hidebysig specialname rtspecialname 
          instance void  .ctor() cil managed
  {
    // Code size       7 (0x7)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
    IL_0006:  ret
  } // end of method Program::.ctor

} // end of class Program

If I run the program, I get a fatal InvalidProgramException error message. The issue is that ToString() is defined on the base System.Object class. When I call System.Void.ToString(), the runtime boxes the value type to an object before calling the ToString() method.
Remember that rule which states that System.Void cannot be boxed. It is enforced by the runtime. There is a check that the value is not System.Void during boxing, and if the check fails, the program is halted.

On a side note, this is another good reason to avoid boxing whenever possible. There is some overhead, besides copying the value to the boxed representation, due to many type checks.

System.Void does not define any members that we can access. The other members (GetType(), ToString(), GetHashCode(), etc.) are all inherited. Calling the inherited methods will lead to boxing which is prohibited. How can we observe its existence?

There’s one way.

// Metadata version: v2.0.50727
.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
  .ver 2:0:0:0
}
.assembly MultiVoid
{
  .permissionset reqmin
             = {[mscorlib]System.Security.Permissions.SecurityPermissionAttribute = {property bool 'SkipVerification' = bool(true)}}
  .hash algorithm 0x00008004
  .ver 1:0:0:0
}
.module MultiVoid.exe
// MVID: {AC411515-B789-4594-B52B-1EC2668D16C9}
.custom instance void [mscorlib]System.Security.UnverifiableCodeAttribute::.ctor() = ( 01 00 00 00 ) 
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x00BC0000


// =============== CLASS MEMBERS DECLARATION ===================

.class private auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
  .field public static valuetype [mscorlib]System.Void o
  .field public static valuetype [mscorlib]System.Void p
  .method private hidebysig static void  Main() cil managed
  {
    .entrypoint
    // Code size       35 (0x23)
    .maxstack  1
    .locals init (valuetype [mscorlib]System.Void& pinned V_0,
             valuetype [mscorlib]System.Void& pinned V_1)
    IL_0000:  ldsflda    valuetype [mscorlib]System.Void Program::o
    IL_0005:  stloc.0
    IL_0006:  ldloc.0
    IL_0007:  conv.i
    IL_0008:  conv.i4
    IL_0009:  call       void [mscorlib]System.Console::WriteLine(int32)
    IL_000e:  ldc.i4.0
    IL_000f:  conv.u
    IL_0010:  stloc.0
    IL_0011:  ldsflda    valuetype [mscorlib]System.Void Program::p
    IL_0016:  stloc.1
    IL_0017:  ldloc.1
    IL_0018:  conv.i
    IL_0019:  conv.i4
    IL_001a:  call       void [mscorlib]System.Console::WriteLine(int32)
    IL_001f:  ldc.i4.0
    IL_0020:  conv.u
    IL_0021:  stloc.1
    IL_0022:  ret
  } // end of method Program::Main

  .method public hidebysig specialname rtspecialname 
          instance void  .ctor() cil managed
  {
    // Code size       7 (0x7)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
    IL_0006:  ret
  } // end of method Program::.ctor

} // end of class Program

The IL code above translates to the following uncompilable C# code:

class Program
{
	public static System.Void o;
	public static System.Void p;
	static void Main()
	{
		unsafe
		{

			fixed (System.Void* i = &o)
			{
				Console.WriteLine((int)i);
			}

			fixed (System.Void* j = &p)
			{
				Console.WriteLine((int)j);
			}

		}

	}
}

It displays the addresses of the System.Void instances in memory: If you run the IL code, you’ll see an output with the memory addresses of the two instances like as shown below:

Congratulations! You just created two instances of the forbidden System.Void type!

That may be as far as you may go. The runtime will detect attempts to place the instance on the evaluation stack, which is necessary to return the instance value from a method.

Exploring System.Void Part I

When calling a method in a dynamically created instance of a type that is discovered at runtime, it's useful to discover the types of the method parameters and the method return type. This information is needed to successfully call the method and retrieve its return value.

It's also useful to discover if the method doesn't have a return type, so that we know not to expect a return value.
One way to find out is to check if the MethodBase.Invoke call returns a null, since a dynamically invoked method that has no return type will return a null. However, this is not the best approach for the following reasons.

Firstly, any method that returns a reference type can return a null, rendering such null checks unreliable.

Secondly, this approach cannot be used to determine the return type before invoking the method.

Fortunately, MethodInfo objects have a ReturnType property that developers can inspect to determine the return type before invoking the method. For instance, if the method returns a string type, MethodInfo.ReturnType will evaluate to typeof(string). It would seem that the natural way to determine if a method does not define a return type would be to check if MethodInfo.ReturnType == typeof(null), however, that won't work because the expression typeof(null) is illegal in C#. Null is not a type.

What's needed is a specially marked type that can be used in the typeof() evaluation to determine that the method does not have a return type. Think about that for a few seconds to get a feel of how counter-intuitive that sounds.
The issue with this approach is that if a method returns an instance of the proposed special type, then the test becomes inconclusive and unreliable. To prevent this from happening, the runtime must bar creation of methods that return this special type.

Enter the System.Void structure. This type is specially designated by .NET for just this purpose. In C#, the void keyword is an alias for the System.Void structure, thus MethodInfo.ReturnType == typeof(void) is a reliable means of discovering if a method has a return type. As expected, no method is allowed to define System.Void as its return type.

This solution looks good and seems like a perfect answer to the "no return type detection" problem. However, there may be a lot more involved. C# and the CLR strangely takes it one step further by disallowing instantiation of the System.Void type by any means. Allow me to demonstrate:

Attempt 1: Try to instantiate System.Void normally

static void Main()
{
    object o = new System.Void();
    //object o = new void() is illegal in C# lingo
}
You'll get a "System.Void cannot be used from C# -- use typeof(void) to get the void type object" (CS0673) error message if you try to compile the code.

Attempt 2: Try to instantiate System.Void dynamically

static void Main(string[] args)
{

    object o = Activator.CreateInstance(typeof(void));
            
}	
It compiles, but when you try to run it, you get a "Cannot dynamically create an instance of System.Void." error message.

Attempt 3: Bypass the C# compiler, Try to create it directly from IL

// Metadata version: v2.0.50727
.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
  .ver 2:0:0:0
}
.assembly ConsoleApplication1
{

  .hash algorithm 0x00008004
  .ver 1:0:0:0
}
.module ConsoleApplication1.exe
// MVID: {47CAF74F-C24A-400E-A0F9-26EB27500120}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x00D10000


// =============== CLASS MEMBERS DECLARATION ===================

.class private auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
  .field public static valuetype [mscorlib]System.Void o
  .method private hidebysig static void  Main() cil managed
  {
    .entrypoint
    // Code size       12 (0xc)
    .maxstack  8
    IL_0000:  ldsflda    valuetype [mscorlib]System.Void Program::o
    IL_0005:  initobj    [mscorlib]System.Void
    IL_000b:  ret
  } // end of method Program::Main

  .method public hidebysig specialname rtspecialname 
          instance void  .ctor() cil managed
  {
    // Code size       7 (0x7)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  call       instance void [mscorlib]System.Object::.ctor()
    IL_0006:  ret
  } // end of method Program::.ctor

} // end of class Program

The IL code above is equivalent to the following C# code
using System;

class Program
{
	public static System.Void o;
	static void Main()
	{
		o = new System.Void();
	}
}
If you compile the IL code with ilasm, It compiles successfully, however it fails verification when you run Peverify against the compiled assembly. If you try to run the assembly, it runs into a fatal InvalidProgramException exception.
If you replace occurrences of System.Void in the IL code to any other struct in the System namespace (besides the ones that map to simple types -- they have a different syntax in IL), the IL code will compile and run smoothly.

From the examples shown above, we have established that the C# compiler and runtime prevents instantiation of the System.Void type. In my next post, We'll explore the System.Void type some more and try to figure out why the runtime doesn't want it to exist.

Happy new year!!

Tamper proof and Obfuscate your Configuration Files

Introduction

The Signature Protected Configuration Provider is a configuration protection provider which can be used to protect configuration file sections from being tampered with. It can optionally obfuscate (scramble) those sections to improve privacy and discourage unauthorized modification.

You can download the source code at www.codeproject.com/KB/security/signedconfigprovider.aspx

Background

I always run into a tight corner whenever I need to encrypt sections of a configuration file because it seems I can’t find an easy, secure way to do it. The .NET provided RSAProtectedConfigurationProvider and DpapiProtectedConfigurationProvider providers tie configuration files to the machine and so, are unsuitable for XCopy deployment.

I started investigating how I could implement a secure, universal and portable configuration encryption/decryption scheme, and I found out it wasn’t possible – because of the nature of .NET applications.

Any kind of encryption scheme requires that the application use a decryption key to decrypt the encrypted information. .NET applications are easy to decompile, and the decompiled source can be examined to discover where the decryption key is read from. Even if it were possible to magically hide the key source, it’s not hard to read the decrypted information while the application is running, using a memory reader.

My point is, if the application can decrypt the information, so can an attacker.

The only reasonable thing that can be done is to obfuscate sections of the configuration file to make it much harder for the attacker. Additionally, it’s possible to securely prevent the attacker from modifying the configuration section. This can be quite useful in enterprise applications where you want only an administrator to be able to modify certain sections of a configuration file and end users to modify others.

Design

At its core, the Signature Protected Configuration Provider uses RSA asymmetric keys. The private key is used to sign the configuration section, which is optionally scrambled (obfuscated) by encrypting it using a symmetric key that is derived from the public key. The configuration section and the signature are enclosed in a new protected section and stored in the configuration file.

The provider has access to the public key and uses it to decrypt the configuration section (if it was encrypted) and to verify the signature with the configuration section to make sure it was not modified.

The provider can implicitly read the protected configuration section because it has access to the public key; however the private key is stored in a secure location inaccessible to the provider. Thus, the provider is implicitly read-only. Consequently, only someone who has access to the private key can modify the protected section.

The Code

The code is stored in the SignatureProtectedConfigurationProvider folder and the main class is the SignatureProtectedConfigurationProvider class.

You can download the source code at www.codeproject.com/KB/security/signedconfigprovider.aspx

The SignatureProtectedConfigurationProvider class inherits the ProtectedConfigurationProvider base class. The beauty of deriving from this class is that the .NET framework automagically decrypts information as needed from the configuration section if the section references the provider. For instance if you protected the appSettings section, you don’t need any special code to decrypt it, all you need to do is access the ConfigurationManager.AppSettings property like as usual. The Framework takes care of the decryption behind the scenes.

Normally, with other protected configuration providers, you can protect sections of your configuration file with the SectionInformation.ProtectSection method, however, the Signature Protected Configuration provider is a read-only provider and cannot implicitly protect a section. To explicitly protect a section, call the SignatureProctectedConfigurationProvider.Write method.

The Utils class contains utility methods called by the Provider class. Housing these methods in a separate class makes it easy to change the internal implementation without touching the provider code.

An important method is the Utils.GetPublicKey method. The public key (the RSA modulus and the exponent components) is also stored in this method. It is stored as either a byte array or a base-64 encoded string, depending on the setting of the StorePublicKeyAsBytes symbol.

The program.cs file is a console application that shows examples of using the provider to explicitly read a protected section, protect a section and generate new keys. You can also use the bundled Configuration File Editor to perform these tasks. (See below)

The Configuration File Editor

The editor was written to facilitate easy protection, unprotection and modification of configuration file sections. It works with existing configuration providers – so if you are tired of dropping to the command line to run aspnet_regiis.exe (with all its parameters), this is the tool you have been looking for.

The editor enables editing configuration files in a hierarchical manner. In fact, it can edit any xml file hierarchically. The editor can also generate new keys and supports other features necessary to configure the Signature Protected Configuration Provider.

The source code for the editor is in the ConfigFileEditor folder.

Using The Provider To Protect A New Configuration File

Run the Configuration File Editor, (you can perform these tasks by following the code samples in the provider program.cs file, but it’s a lot easier to use the configuration editor)

1. Open the Configuration File (you can open the bundled sample.config file)

2. Click the tools menu, then select ‘Generate configProtectedData Element’ under the ‘Signature Protected provider’ sub-menu.

The Generate configProtectedData Element window appears.

3. Click the ‘Add to Configuration’ button.

This action adds XML elements required for the .NET framework to use the provider from your application.

4. Click the tools menu, and then select ‘Generate New Key Pair’ under the ‘Signature Protected provider’ sub-menu.

The Generate New RSA Key Pair window appears.

This window contains the private/public key information. It is important that you store this information in a secure location that is not accessible from your application. Without this information, it is not possible to modify protected sections.

5. Click ‘Copy To Clipboard’ to copy the information to the Windows Clipboard.

6. Click the Close button.

7. Save the key information (in the clipboard) to your secure location.

8. As an example, right click the appSettings node on the side-bar on the left side of the editor to open a context menu.

9. Select ‘Protect’

The Select Provider window appears.

10. You can choose which protection provider you want to use (you will see the RsaProtectedConfigurationProvider and the DataProtectionConfigurationProvider options)

11. Select the SignatureProtectedConfigurationProvider option.

12. You can uncheck the ‘Obfuscate section’ checkbox if you want your users to be able to read the protected information -- don't worry, they still won't be able to modify it.

13. Click the OK button.

14. You will be prompted for the key – you can just paste the entire XML you saved in step 7 and click OK.

The section will be protected. You can protect other sections as you wish by repeating steps 8 to 13.

15. Save the file by selecting ‘Save’ under the File menu.

Now you need to set up the provider to work from your application.

16. Copy SignedConfigProvider.cs and SignedConfigUtils.cs files from the SignatureProtectedConfigurationProvider folder to your application’s project folder.

17. Add the files to your project so that they appear in Visual Studio’s solution explorer.

18. Open up SignedConfigUtils.cs from within your project and navigate to the GetPublicKey method

19. Open the key information file you stored securely earlier (at step 7), look at the section titled PUBLIC KEY INFORMATION. You have to replace the public key in the GetPublicKey method with the one in that section.

You can do this either by replacing the byte arrays in the method with the ones in the file or by replacing the modulus and exponent strings in the method. You have to change the StorePublicKeyAsBytes symbol to use the latter method. I prefer to use byte arrays because they are easier to manipulate.

Now, your application will transparently read the protected sections.

Using The Provider To Modify A Protected Configuration File

1. Open the configuration file with the Configuration Editor.

2. Click to select the protected node on the left side-bar of the editor.

3. You will be prompted for the private key.

4. Paste the private key (you can paste the entire xml) you previously securely stored.

5. Make the changes you want to make.

6. Click 'Apply Changes'.

7. Save the Configuration File.

Security Notes

To check if a section was protected with the provider from your application, include code that examines the SectionInformation.ProtectionProvider property to make sure it’s the same type as the SignatureProtectedConfigurationProvider class.

The public key is embedded in the provider code and not stored in an external file because if it is read from a file or read from a library; an attacker can generate his own private/public key pair, modify the configuration section and protect it using his keys and replace the public key in the file or library with his.
As a consequence, make sure you compile the provider (with the embedded public key) into your application. Do not compile the provider as a dynamically linked library.

It is important to note that the obfuscation aspect of this provider is performed using the public key which is accessible to anyone who has access to your application. Do not depend on this provider to secure sensitive information like database connection strings. This provider is more suited for protecting important information like web service urls from unauthorized modification.
You can make it harder for an attacker to figure out what the public key is by making the ReadPublicKey method harder to understand and by obfuscating the application after compilation, but it’s safer to treat the public key for what it is – public.

Always use the same key to protect all sections in a configuration file. It is possible to accidentally protect different sections with different keys while using the configuration file editor. Protecting different sections using different keys will render some sections of the configuration file undecipherable by the provider because it has access to only one public key.
You can enter comments in your configuration file and in the public/private key XML to help you remember which key protects which configuration file.

Points of Interest

The proper way to package signed XML is to use the XML Signature standard format . The System.Security.Cryptography.Xml.SignedXml class implements this standard, however for the sake of brevity; the provider simply encloses the plain (or obfuscated) configuration section in the SignedInfo element, and the base-64 encoded signature is enclosed in the SignatureValue element.
Both elements are enclosed inside an EncryptedData element which replaces the contents of the original unprotected element.

The proper way to encrypt data with asymmetric keys is to encrypt the data using a symmetric encryption algorithm and then encrypt the symmetric key using the asymmetric keys. In this case, I needed to encrypt the symmetric key using the private key and decrypt it using the public key.
I couldn’t do this because the .NET implementation of the RSA algorithm only lets you encrypt with the public key and decrypt with the private key; which makes sense, because, data that can be decrypted with the public key is in reality, plain-text; since everybody has access to the public key.
However, I’m more interested in obfuscation than secure encryption so I simply used portions of the public key to create the symmetric key used to perform the encryption and decryption. The Utils.GetAESFromRSAParameters method instantiates a RijnDaelManaged object using the public key parameters.
This approach doesn’t improve or reduce security because either way, you only need access to the public key to read the encrypted information.

Conclusion

This is a truly portable configuration protection provider. It works on both desktop and web applications.

It can obfuscate the configuration section – this feature, when combined with obfuscation of the compiled application can make it very difficult for an attacker to read sections of the configuration file.

It also securely prevents modification to critical sections of your configuration file. Furthermore, it can be extended to facilitate secure messaging in a client/server environment because the application can use the embedded public key to verify that the transmission is from the right source.

References

How To: Encrypt Configuration Sections in ASP.NET 2.0 using RSA
Implementing a Protected Configuration Provider
XML Signature Syntax and Processing

Want To Pass 70-562? Here’s How.

After studying for the past few months I took the Microsoft .NET Framework 3.5, ASP.NET Application Development exam today and passed with a score of 873.
This is certainly the broadest certification exam I’ve taken so far. I almost spent the entire allotted two and a half hours on the test.
I'm now MCTS .NET Framework 3.5, ASP.NET Applications. This is really cool considering that as at today there are only about 271 certified people worldwide. Cool
I guess everyone else is waiting for the training kit to be released in April next year.

If you can't wait till April, here's how I studied:

Prerequisites:

Study Plan:

  • If you are not comfortable with ASP.NET AJAX, read Introducing Microsoft ASP.NET AJAX by Dino Esposito.
  • Read Programming Microsoft ASP.NET 3.5, also by Dino Esposito. This is a very detailed book on ASP.NET. It covers a broad and practical range of ASP.NET topics.
  • Get a copy of MCTS Self-Paced Training Kit (Exam 70-528). This is the training kit for the .NET 2.0 version of this exam. You’ll need it to study topics not covered in the books I mentioned earlier.
    • Study Chapters 4, 5, 10, 12 and 13.
    • Practice the “Practice tests” on the CD that comes with this kit especially questions from the mentioned chapters.
  • Purchase and practice the Measure-up practice tests, available at http://www.measureup.com/catalog/exam.aspx?vid=5&cid=MCTS&tid=139 . I cannot over-emphasize this step. It is important that you score a 100% twice on ALL the questions before attempting this exam. Do not take the exam if you have not seen and understood all questions and answers in the Measure-up practice tests. It’s a great indicator of your preparedness level.
  • Take a look at the 70-562 Preparation Guide and if there is any topic you do not instantly recognize, look it up in MSDN and learn it.

Tips:

  • Practice every topic from each book you study.
  • If there is anything you do not understand, Google it and master it.
  • Pay particular attention to features that are new with ASP.NET 3.5 such as the Listview control, IIS 7, WCF, Aspnet_merge.exe, Datapager control, etc.
  • Practice a lot of web.config configuration settings and command line tools like Aspnet_Compiler.exe, Aspnet_regiis.exe, etc.


Do not under-estimate this exam; it is difficult because it covers a very wide range of topics. The exam basically tests your ASP.NET 3.5 experience-level.
However, by the time you study and practice all the topics I’ve mentioned, you’ll be quite experienced.
Good luck!!

 

Generate Stored Procedure Wrapper Methods and Associated Wrapper Classes

Introduction

It is generally a good idea to create a wrapper method for every stored procedure that an application needs to call. Such methods can then be grouped into a single data access utility class. This approach improves type safety and portability.

These methods generally call the Command.ExecuteScalar, Command.ExecuteNonQuery and Command.ExecuteReader ADO.NET methods. They also perform tasks like checking if the Connection object is still connected, adding stored procedure parameters to the Command object, making sure the DataReader object is properly disposed, etc.

On the average, there are about fifty lines of code for a properly written method that wraps an ExecuteReader call! Writing these methods easily eats into overall development time on data-intensive projects that access many stored procedures.

Developers usually resort to copying and pasting code from other wrapper methods and modifying the code to suit the stored procedure call. This process often leads to bugs due to human error.

I figured that since most of these methods share a common programming pattern; it should be possible to describe what your stored procedure looks like to a code generation tool and have the tool generate these methods. Nothing as complex as AutoSproc, just a light tool that lets a developer specify details of the stored procedure, and then generate the wrapper method code.

Implementation

This tool was developed with ASP.NET 2.0. It makes code generation decisions based on information provided by the user – pretty much the same way a human developer would make coding decisions.
It supports .NET 1.1 and .NET 2.0 features, for instance it would create nullable variables for nullable fields if .NET 2.0 is selected. It supports the SQL Server and ODBC data providers.

The actual code generation code (no pun intended) is in APP_Code\MethodGen.cs while the user interface code is in the sprocmethodgen.aspx.
The code generation code can easily be used by another application with a different user interface (for instance, a desktop-application that supplies most of the input from the actual database schema). It can also be easily modified to follow a different programming pattern or support more ADO.NET features.

The meat of the code generation code lies in the GenerateMethod, GenerateTryClause and GenerateResultsWrapperClass methods of the MethodGenerator class. The GenerateMethod method generates the non-varying parts of the method such as sections that add parameters to a command object. It also calls the GenerateTryClause method and optionally the GenerateResultsWrapperClass method.

The GenerateTryClause method generates the big try clause in the method which varies greatly, depending on what type of execution was selected.

The GenerateResultsWrapperClass method generates a class which stores results returned by a DataReader. (It’s better to return a list of strongly typed objects than returning a DataTable.)

Using the Tool

This example uses the ‘Sales by Year’ stored procedure in the Northwind database.

1) Run the ASP.NET solution, and navigate to the web page.

This tool is also available at www.ahuwanya.net/tools/sprocmethodgen/

2) Specify the .NET Version, Data Provider, Stored Procedure name, and Type of Execution.

3) The Sales by Year stored procedure has two input parameters, so specify the two input parameters:

4) After running the stored procedure, it is discovered that it has four columns in the result set, so specify the four result columns:


5) Specify the name of the class that will store the results and the name of the generated method.

6) Click the Generate Code! button to generate the code. You may need to scroll down on the page.
(To view the generated code, click the expand source button below.)

internal static List<SalesInfo> GetSalesByYear(DateTime StartDate, DateTime EndDate, SqlConnection DBConn )
{
	//TODO: Insert method into your data access utility class
	
	//Check if connection is null
	if(DBConn == null)
	{
		throw new ArgumentNullException("DBConn");
	}

	//Open connection if it's closed
	bool connectionOpened = false;
	if(DBConn.State == ConnectionState.Closed)
	{
		DBConn.Open();
		connectionOpened = true;
	}

	//TODO: Move constant declaration below directly into containing class
	const string sprocGetSalesByYear = "[Sales by Year]";

	string sproc = sprocGetSalesByYear;

	SqlCommand cmd = new SqlCommand(sproc,DBConn);
	cmd.CommandType = CommandType.StoredProcedure;
	cmd.Parameters.Add("@Beginning_Date",SqlDbType.DateTime ).Value = StartDate;
	cmd.Parameters.Add("@Ending_Date",SqlDbType.DateTime ).Value = EndDate;
	
	List<SalesInfo> result = new List<SalesInfo>();
	SqlDataReader rdr;
	
	try
	{
		rdr = cmd.ExecuteReader();

		try
		{
			if (rdr.HasRows)
			{
				int shippeddateOrdinal = rdr.GetOrdinal("ShippedDate");
				int orderidOrdinal = rdr.GetOrdinal("OrderID");
				int subtotalOrdinal = rdr.GetOrdinal("Subtotal");
				int yearOrdinal = rdr.GetOrdinal("Year");
				
				while (rdr.Read()) 
				{
					// declare variables to store retrieved row data
					DateTime? shippeddateParam;
					int orderidParam;
					decimal subtotalParam;
					string yearParam;
					
					// get row data
					if (rdr.IsDBNull(shippeddateOrdinal))
					{
						shippeddateParam = null;
					}
					else
					{
						shippeddateParam = rdr.GetDateTime(shippeddateOrdinal);
					}
					orderidParam = rdr.GetInt32(orderidOrdinal);
					subtotalParam = rdr.GetDecimal(subtotalOrdinal);
					if (rdr.IsDBNull(yearOrdinal))
					{
						yearParam = null;
					}
					else
					{
						yearParam = rdr.GetString(yearOrdinal);
					}
					
					// add new SalesInfo object to result list
					result.Add(new SalesInfo(shippeddateParam,orderidParam,subtotalParam,yearParam));
					
				}
			}
		}
		finally
		{
			rdr.Close();
		}
		
	}
	catch(Exception ex)
	{
		//TODO: Handle Exception
		throw ex;
	}
	finally
	{
		cmd.Dispose();

		if(connectionOpened ) // close connection if this method opened it.
		{
			DBConn.Close();
		}
	}

	return result;
	

}

[Serializable]
public class SalesInfo
{
	//TODO: Integrate this class with any existing data object class

	private DateTime? shippeddate;
	private int orderid;
	private decimal subtotal;
	private string year;
	
	public SalesInfo(DateTime? ShippedDate, int OrderID, decimal SubTotal, string Year)
	{
		shippeddate = ShippedDate;
		orderid = OrderID;
		subtotal = SubTotal;
		year = Year;
		
	}

	public SalesInfo()
	{
		shippeddate = null;
		orderid = 0;
		subtotal = 0;
		year = null;
		
	}

	public DateTime? ShippedDate
	{
		get { return shippeddate; }
		set { shippeddate = value; }
	}

	public int OrderID
	{
		get { return orderid; }
		set { orderid = value; }
	}

	public decimal SubTotal
	{
		get { return subtotal; }
		set { subtotal = value; }
	}

	public string Year
	{
		get { return year; }
		set { year = value; }
	}

	
}

7) Copy the generated code into your project.

8) Add the following namespaces to your project.

using System.Data; 
using System.Data.SqlClient; //if using SQL Server Data Provider
using System.Data.Odbc; //if using ODBC Provider
using System.Collections.Generic; //if using .NET 2.0 or later
using System.Collections //if using .NET 1.1

9) Look for //TODO: comments in the generated code and act accordingly. The code will still work, even if the //TODO: comments are ignored.

10) Now you can simply access the sales data from your project with the following statements:

 //Create Sql connection
SqlConnection conn = new SqlConnection(@"Data Source=.\SQLEXPRESS;Initial Catalog=Northwind;Integrated Security=True");
//Get Sales Information
List<SalesInfo> sales = GetSalesByYear(new DateTime(1992,1,1),new DateTime(2008,1,1),conn);

Considerations and Limitations

Obviously, this tool cannot generate code for every conceivable ADO.NET database access scenario, however, it’s a lot better to generate a great deal of the code and then modify the code as needed, than to type everything by hand.

Some limitations of this tool include:

Generates only C# code: The best way to make this tool to be language neutral would be to use CodeDom to generate the code, however, this approach would make this tool harder to maintain and extend – It would be an overkill for the scope of this project.
Fortunately, there are lots of C#-to-VB.NET code conversion tools available for VB.NET developers who would like to use this tool.

Lacks support for Output Parameters: This tool only supports input parameters and optionally returns the stored procedure return parameter. The generated code can be manually modified to accommodate other types of parameters.

Lacks support for OLEDB and Oracle Data Providers: This tool only generates code for ODBC and SQL Server Data providers.

Reads only the first Result Set: If your stored procedure returns multiple result sets, one way to handle this is to generate a method for the first result set (choose ExecuteReader), then generate another method as if the second result set were actually the first result set, copy the code that reads the result data and paste it into the first method after calling rdr.NextResults(), change the name of the results variable in the pasted code and pass it back as an out parameter.
Do this for every result set returned.

Lacks support for DbCommand object properties: If you are looking for Transaction, CommandTimeOut, CommandBehavior objects etc. -- It’s easy to modify the generated code for these properties.

Unsuitable for Large Result sets: This tool generates code which returns result sets as an ArrayList or an List of strongly-typed objects. This tool will perform poorly if your stored procedure returned hundreds of thousands of rows because it would have to store all these rows. You should write your own data access method for such scenarios.
Moreover, if your stored procedure returns hundreds of thousands of rows, I recommend you look into implementing some kind of paging mechanism to reduce the number of rows returned.