Sunny Ahuwanya's Blog

Mostly notes on .NET and C#

Multi-Slab Buffers

A noteworthy feature of the bufferpool project is the ability for an individual buffer to span multiple slabs.

Previously, a buffer was confined to one slab, meaning that the maximum buffer size was the slab size.
It was impossible to send or receive data that was larger than the slab.
This commit changed that. Individual buffers can now span multiple slabs, which allows for what is known as Scatter/Gather or Vectored I/O operations.

Buffer spanning multiple slabs


This means buffers of any arbitrary length can be requested from the pool and all needed slabs will be created and added to the pool, to form the buffer.
Conversely, the slabs are freed and removed from the pool, when the buffer is disposed.

This feature is particularly useful for transmitting or receiving large amount of data. It eliminates the need to allocate a large contiguous memory block to hold the data.
Instead, smaller slabs, which are allocated at different convenient locations in memory, are chained to form the buffer.
This significantly lessens the chance of running out of memory, due to heap fragmentation, during socket operations involving a large amount of data.

Buffer Pooling for .NET Socket Operations

The terrific Garbage Collector in the .NET runtime is a compacting one. This means it moves memory blocks in use closer to each other during garbage collection, thus increasing the overall size of contiguous free memory and lessening the chance of running out of memory due to heap fragmentation.
This is pretty important for memory intensive applications as it is a nice guarantee that free contiguous memory will always be available whenever a large object needs to be allocated.

Whenever you perform send or receive socket operations in .NET, the buffer that holds the transmitted or received data is pinned by the .NET runtime. Pinning basically means that the region in memory that holds the data is locked down and is not eligible to be moved around by the Garbage Collector. This is necessary so that the Windows Socket API which handles the actual socket operation (and lives outside managed code) can access the buffer memory locations reliably. This pinning takes place regardless of if the operation is synchronous or asynchronous.

If you are building a memory intensive server application that is expected to serve thousands of clients concurrently, then you have a simmering problem waiting to blow up as soon as the server starts serving a non-trivial number of clients.
Thousands of sockets performing receive and/or send operations will cause thousands of memory locations to be pinned. If your server stores a lot of data in memory, you can have a fragmented heap that even though collectively has enough free memory to store a large object, is unable to do so, because there isn’t enough memory in a contiguous free block to store it. This is illustrated below: 

unpinned memory 
The diagram above depicts the common scenario where a large object is about to be allocated in memory. There is insufficient contiguous memory to store the object, which triggers garbage collection. During garbage collection, the memory blocks in use are compacted, freeing enough room to store the large object. 

pinned memory 
The diagram above depicts the same scenario with some objects pinned. The garbage collector compacts objects that are not pinned which makes some room, but not enough room due to the unmovable pinned objects. The end result is that a nasty Out Of Memory exception is thrown by the runtime because there is no contiguous free memory space large enough for the large object to be allocated.

This problem is discussed at length in this blog post and in this blog post. The solution to the problem, as noted in those posts, is to pre-allocate buffers. This basically means setting up a byte array which your sockets will use to buffer data. If all your socket operations use buffers from the designated byte array, then only memory locations within the region where the byte array is stored will be pinned, giving the GC greater freedom to compact memory.
However, for this to work effectively with multiple asynchronous socket operations, there has to be some kind of buffer manager that will dish out different segments of the array to different operations and ensure that buffer segments allotted for one operation are not used by a different operation until they are marked as no longer in use.

I searched for existing solutions for this problem but I couldn’t find any that was sufficiently adequate, so I wrote one that was robust and dependable. The solution is part of my ServerToolkit project, which is a set of useful libraries that can be used to easily build scalable .NET servers. The solution is called BufferPool and is the first ServerToolkit sub-project.

The idea is that you have a buffer pool that is a collection of memory blocks (byte arrays) known as slabs that you can grab buffers from. You decide the size of the slab and the initial number of slabs to create.
As you request buffers from the pool, it internally allocates segments within the slabs for use by your buffer. If you need more buffers than are available within the pool, new slabs are created and are added to the pool. Conversely, as you dispose your buffers, slabs within the pool will be removed when they no longer contain any allocated segments.  
 
buffer pool

USING BUFFER POOLING IN YOUR APPLICATION

Download the code from http://github.com/tenor/ServerToolkit/tree/master/BufferPool

Compile it to a DLL and reference the DLL from your project
OR
Add the project to your solution and reference the ServerToolkit.BufferManagement project from your main project.

In the code file that performs socket operations, add

using ServerToolkit.BufferManagement;

Create a buffer pool when your application starts with

BufferPool pool = new BufferPool(1 * 1024 * 1024, 1, 1);

This creates a pool that will have 1 MB slabs with one slab created initially. Subsequent slabs, if needed, will be created in increments of one.

It is advisable to create the pool as soon as your server application starts or at least before it begins any memory-intensive operations.

To use the pool in synchronous send and receive socket operations

/** SENDING DATA **/

// const int SEND_BUFFER_SIZE is the desired size of the send buffer in bytes 
// byte[] data contains the data to be sent.

using (var buffer = pool.GetBuffer(SEND_BUFFER_SIZE)) 
{ 
    buffer.FillWith(data); 
    socket.Send(buffer.GetSegments()); 
}


/** RECEIVING DATA **/

// const int RECEIVE_BUFFER_SIZE is the desired size of the receive buffer in bytes 
// byte[] data is where the received data will be stored.

using (var buffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE)) 
{ 
    socket.Receive(buffer.GetSegments()); 
    buffer.CopyTo(data); 
}

To use the pool for asynchronous send and receive socket operations

Sending data:

/** SENDING DATA ASYNCHRONOUSLY **/

// const int SEND_BUFFER_SIZE is the desired size of the send buffer in bytes 
// byte[] data contains the data to be sent.

var buffer = pool.GetBuffer(SEND_BUFFER_SIZE); 
buffer.FillWith(data); 
socket.BeginSend(buffer.GetSegments(), SocketFlags.None, SendCallback, buffer);

//...


//In the send callback.

private void SendCallback(IAsyncResult ar) 
{ 
    var sendBuffer = (IBuffer)ar.AsyncState; 
    try 
    { 
        socket.EndSend(ar); 
    } 
    catch (Exception ex) 
    { 
        //Handle Exception here 
    } 
    finally 
    { 
        if (sendBuffer != null) 
        { 
            sendBuffer.Dispose(); 
        } 
    } 
}

Receiving data:

/** RECEIVING DATA ASYNCHRONOUSLY **/

// const int RECEIVE_BUFFER_SIZE is the desired size of the receive buffer in bytes. 
// byte[] data is where the received data will be stored.

var buffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE); 
socket.BeginReceive(buffer.GetSegments(), SocketFlags.None, ReadCallback, buffer);

//...


//In the read callback

private void ReadCallback(IAsyncResult ar) 
{ 
    var recvBuffer = (IBuffer)ar.AsyncState; 
    int bytesRead = 0;

    try 
    { 
        bytesRead = socket.EndReceive(ar); 
        byte[] data = new byte[bytesRead > 0 ? bytesRead : 0];

        if (bytesRead > 0) 
        { 
            recvBuffer.CopyTo(data, 0, bytesRead);

            //Do anything else you wish with read data here. 
        } 
        else 
        { 
            return; 
        }

    } 
    catch (Exception ex) 
    { 
        //Handle Exception here 
    } 
    finally 
    { 
        if (recvBuffer != null) 
        { 
            recvBuffer.Dispose(); 
        } 
    }

    //Read/Expect more data                    
    var buffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE); 
    socket.BeginReceive(buffer.GetSegments(), SocketFlags.None, ReadCallback, buffer);

} 

There is a performance penalty involved when using the code above. Each time you perform a socket operation, you create a new buffer, use it and then dispose it. These actions involve multiple lock statements, and can become a bottleneck on the shared pool.

There is an optimization that can help mitigate this issue. The optimization is based on the way socket operations work. You can send data any time you like on a socket, but you typically receive data in a loop. i.e. receive data, signal that you are ready to receive more data and repeat. This means that each socket works with only one receive buffer at a time.
With this in mind, you can assign each socket a receive buffer of its own.  Each socket will use its receive buffer exclusively for all its receive operations, thereby avoiding the need to create and dispose a new buffer for each receive operation.
You have to dispose the receive buffer when you close the socket.

I recommend the pattern below.

/** RECEIVING DATA ASYNCHRONOUSLY (OPTIMIZED) **/

// const int RECEIVE_BUFFER_SIZE is the desired size of the receive buffer in bytes.

// const int BUFFER_SIZE_DISPOSAL_THRESHOLD specifies a size limit, which if exceeded by a buffer
// would cause the buffer to be disposed immediately after a receive operation ends.
// Its purpose is to reduce the number of large buffers lingering in memory.

// IBuffer recvBuffer is the receive buffer associated with this socket.

// byte[] data is where the received data will be stored.

// object stateObject holds any state object that should be passed to the read callback. 
// It is not necessary for this sample code.

if (recvBuffer == null || recvBuffer.IsDisposed)
{
    //Get new receive buffer if it is not available.
    recvBuffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE);
}
else if (recvBuffer.Size < RECEIVE_BUFFER_SIZE)
{
    //If the receive buffer size is smaller than desired buffer size,
    //dispose receive buffer and acquire a new one that is long enough.
    recvBuffer.Dispose();
    recvBuffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE);
}

socket.BeginReceive(recvBuffer.GetSegments(), SocketFlags.None, ReadCallback, stateObject);

//...


//In the read callback

private void ReadCallback(IAsyncResult ar)
{
    int bytesRead = socket.EndReceive(ar);

    byte[] data = new byte[bytesRead > 0 ? bytesRead : 0];

    if (recvBuffer != null && !recvBuffer.IsDisposed)
    {
        if (bytesRead > 0)
        {
            recvBuffer.CopyTo(data, 0, bytesRead);

            //Do anything else you wish with read data here.
        }

        //Dispose buffer if it's larger than a specified threshold
        if (recvBuffer.Size > BUFFER_SIZE_DISPOSAL_THRESHOLD)
        {
            recvBuffer.Dispose();
        }
    }

    if (bytesRead <= 0) return;

    //Read/Expect more data
    if (recvBuffer == null || recvBuffer.IsDisposed)
    {
        //Get new receive buffer if it is not available.
        recvBuffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE);
    }
    else if (recvBuffer.Size < RECEIVE_BUFFER_SIZE)
    {
        //If the receive buffer size is smaller than desired buffer size,
        //dispose receive buffer and acquire a new one that is long enough.
        recvBuffer.Dispose();
        recvBuffer = pool.GetBuffer(RECEIVE_BUFFER_SIZE);
    }

    socket.BeginReceive(recvBuffer.GetSegments(), SocketFlags.None, ReadCallback, stateObject);

}

If using the pattern presented above, it’s important to remember to dispose the buffer when closing the socket.
You must explicitly dispose the buffer. It does not define a finalizer, so once a buffer is created, it stays allocated until explicitly disposed.

CONCLUSION

The BufferPool project is the first sub-project of the ServerToolkit project. It is an efficient buffer manager for .NET socket operations.
It is written in C# 2.0 and targets .NET 2.0 for maximum backward compatibility.

BufferPool is designed with high performance and dependability in mind. Its goal is to abstract away the complexities of buffer pooling in a concurrent environment, so that the developer can focus on other issues and not worry about heap fragmentation caused by socket operations.

Probing the ASP.NET State Service Part II

In my previous post, I took a first look at the ASP.NET state service communication protocol. In this post I'll describe the techniques I'm using to piece out and understand the protocol.

I needed to monitor the traffic between the web server and the state service so I installed WinDump, a Windows version of tcpdump.
With WinDump, I could capture the low level tcp traffic between the state service and the web server. The only caveat was that I had to monitor a state service on a different computer because WinDump doesn’t work with the loopback device.

After a while I decided WinDump was too low level for the task at hand. What I really needed was a tcp relay server.

A tcp relay server is a more involved version of an echo server that sits between a server and a client and relays transmitted data to and fro. With a tcp relay server, not only can I capture the transmitted data, I can also modify it.



To have the greatest flexibility, I decided to write one. My relay server is a console application. Here's the code:

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.Threading;

namespace RelayServer
{
    //State object
    public class StateObject
    {
        // Client  socket.
        public Socket WorkSocket = null;
        // Size of receive buffer.
        public const int BufferSize = 1024;
        // Receive buffers.
        public byte[] Buffer = new byte[BufferSize];
        // Socket that connects to other party
        public Socket HandoffSocket = null;
    }

    class Program
    {
        static int relayPort = 24242; //Port to listen on for client connection
        static string serviceHost = "localhost"; //State service host
        static int servicePort = 42424; //State service port
        static StateObject pollStateObj = new StateObject(); //State object polled for disconnection

        static void Main(string[] args)
        {
            IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, relayPort);
            Socket clientListener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            clientListener.Bind(localEndPoint);
            clientListener.Listen(100);
            clientListener.BeginAccept(new AsyncCallback(AcceptCallback), clientListener);

            while (!Console.KeyAvailable)
            {
                Thread.Sleep(200); //check every 200 milliseconds

                //is the state object instantiated and connected?
                if (pollStateObj.WorkSocket != null && 
                     (pollStateObj.WorkSocket.Connected || pollStateObj.HandoffSocket.Connected))
                {                    
                    //check if work (client) socket is disconnected
                    try
                    {
                        //Test for disconnection
                        bool isDisconnected = false;
                        lock (pollStateObj.WorkSocket)
                        {
                            isDisconnected = pollStateObj.WorkSocket.Available == 0 && 
                                pollStateObj.WorkSocket.Poll(1, SelectMode.SelectRead);
                        }

                        if (isDisconnected)
                        {
                            Console.ForegroundColor = ConsoleColor.Red;
                            Console.WriteLine("Client disconnected.");
                            DisconnectSockets(pollStateObj);
                            continue;
                        }
                    }
                    catch (ObjectDisposedException ex)
                    {
                        //do nothing
                        continue;
                    }
                    catch (SocketException ex)
                    {
                        //Was Socket remotely disconnected?
                        if (ex.ErrorCode == 10054)
                        {
                            Console.ForegroundColor = ConsoleColor.Red;
                            Console.WriteLine("Client disconnected.");
                        }
                        else if (ex.ErrorCode == 10053)
                        {
                            Console.ForegroundColor = ConsoleColor.Red;
                            Console.WriteLine("Client aborted connection.");
                        }
                        else
                        {
                            Console.ForegroundColor = ConsoleColor.White;
                            Console.WriteLine("\nError: " + ex + "\n");
                        }

                        DisconnectSockets(pollStateObj);
                        continue;
                    }
                    catch (Exception ex)
                    {
                        Console.ForegroundColor = ConsoleColor.White;
                        Console.WriteLine("\nError: " + ex + "\n");
                        continue;
                    }

                    try
                    {
                        //Test for disconnection
                        lock (pollStateObj.HandoffSocket)
                        {
                            pollStateObj.HandoffSocket.Send(new byte[1], 0, SocketFlags.None);
                        }
                    }
                    catch (ObjectDisposedException ex)
                    {
                        //do nothing
                        continue;
                    }
                    catch (SocketException ex)
                    {
                        //Was Socket remotely disconnected?
                        if (ex.ErrorCode == 10054)
                        {
                            Console.ForegroundColor = ConsoleColor.Red;
                            Console.WriteLine("Server disconnected.");
                        }
                        else if (ex.ErrorCode == 10053)
                        {
                            Console.ForegroundColor = ConsoleColor.Red;
                            Console.WriteLine("Server aborted connection.");
                        }
                        else
                        {
                            Console.ForegroundColor = ConsoleColor.White;
                            Console.WriteLine("\nError: " + ex + "\n");
                        }

                        DisconnectSockets(pollStateObj);
                        continue;
                    }
                    catch (Exception ex)
                    {
                        Console.ForegroundColor = ConsoleColor.White;
                        Console.WriteLine("\nError: " + ex + "\n");
                        continue;
                    }
                }
            }
        }

        private static void AcceptCallback(IAsyncResult ar)
        {
            //Accept incoming connection
            Socket listener = (Socket)ar.AsyncState;
            Socket handler = listener.EndAccept(ar);
            
            //Display Connection Status
            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine("Client is connected.");

            //Accept another incoming connection
            listener.BeginAccept(new AsyncCallback(AcceptCallback), listener);

            // Create the state object.
            StateObject state = new StateObject();
            state.WorkSocket = handler;
            state.HandoffSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

            //Connect to other socket
            state.HandoffSocket.Connect(serviceHost, servicePort);

            //Create another state object to receive information from service
            StateObject recvState = new StateObject();
            recvState.WorkSocket = state.HandoffSocket;
            recvState.HandoffSocket = state.WorkSocket;

            //Set up the Service socket to receive information from service
            state.HandoffSocket.BeginReceive(recvState.Buffer, 0, StateObject.BufferSize, SocketFlags.None,
                new AsyncCallback(ReadCallback), recvState);

            //Set up the client socket to receive information from client
            handler.BeginReceive(state.Buffer, 0, StateObject.BufferSize, SocketFlags.None,
                new AsyncCallback(ReadCallback), state);

            //Reference this State object as the object to poll for disconnections
            pollStateObj = state;
        }

        private static void ReadCallback(IAsyncResult ar)
        {
            // Retrieve the state object and the handler socket
            // from the asynchronous state object.
            StateObject state = (StateObject)ar.AsyncState;

            if (state.WorkSocket.Connected == false)
            {
                return;
            }

            try
            {
                // Read data from the client socket. 
                int bytesRead;
                lock (state.WorkSocket)
                {
                    bytesRead = state.WorkSocket.EndReceive(ar);
                }

                if (bytesRead > 0)
                {
                    //Transform received data
                    byte[] transformedData = TransformData(state.Buffer, bytesRead, state);

                    //Transmit transformed data to other Socket
                    if (transformedData.Length > 0)
                    {
                        lock (state.HandoffSocket)
                        {
                            state.HandoffSocket.BeginSend(transformedData, 0, transformedData.Length, 
                                SocketFlags.None, new AsyncCallback(SendCallback), state);
                        }
                    }
                }

                lock (state.WorkSocket)
                {
                    state.WorkSocket.BeginReceive(state.Buffer, 0, StateObject.BufferSize, 0,
                        new AsyncCallback(ReadCallback), state);
                }
            }
            catch (ObjectDisposedException ex)
            {
                //do nothing
                return;
            }
            catch (SocketException ex)
            {
                if (ex.ErrorCode == 10054 || ex.ErrorCode == 10053)
                {
                    //do nothing
                    return;
                }
                Console.ForegroundColor = ConsoleColor.White;
                Console.WriteLine("\nError: " + ex + "\n");

                return;
            }
            catch (Exception ex)
            {
                Console.ForegroundColor = ConsoleColor.White;
                Console.WriteLine("\nError: " + ex + "\n");
                return;
            }

        }

        private static void SendCallback(IAsyncResult ar)
        {
            // Retrieve the socket from the state object.
            StateObject state = (StateObject)ar.AsyncState;

            if (!state.HandoffSocket.Connected)
            {
                return;
            }

            // Complete send
            try
            {
                lock (state.HandoffSocket)
                {
                    state.HandoffSocket.EndSend(ar);
                }
            }
            catch (Exception ex)
            {
                Console.ForegroundColor = ConsoleColor.White;
                Console.WriteLine("\nError: " + ex + "\n");
                return;
            }
        }

        private static byte[] TransformData(byte[] Data, int Length, StateObject State)
        {
            //Just display Transmitted Data
            if (State.WorkSocket == pollStateObj.WorkSocket)
                Console.ForegroundColor = ConsoleColor.Cyan;
            else
                Console.ForegroundColor = ConsoleColor.Green;
            
            Console.Write(Encoding.UTF8.GetString(Data, 0, Length));

            byte[] output = new byte[Length];
            Array.Copy(Data, output, Length);
            return output;
        }

        private static void DisconnectSockets(StateObject state)
        {
            lock (state.WorkSocket)
            {                                        
                if (state.WorkSocket.Connected)
                {
                    state.WorkSocket.Shutdown(SocketShutdown.Both);
                }
                state.WorkSocket.Close();                    
            }

            lock (state.HandoffSocket)
            {
                if (state.HandoffSocket.Connected)
                {
                    state.HandoffSocket.Shutdown(SocketShutdown.Both);
                }
                state.HandoffSocket.Close();
            }
        }
    }
}

It's important to note that the tcp relay server must also relay connections and disconnections from either end, in addition to transmitted data.

To test the relay server:

1. Create a new ASP.NET web application in Visual Studio.

2. Add the following line in the system.web section of the Web.config file:

<sessionState mode="StateServer" stateConnectionString="tcpip=localhost:24242" cookieless="false" timeout="20"/>
<!-- -->

3. Add the following lines of code to the Page_Load event handler.

 Session.Add("Test2", "Test");
 Session.Abandon();

4. Run the relay server.

5. Start the local ASP.NET State Service (Located at Control Panel -> Administrative Tools -> Services) .

6. Run the ASP.NET Application

You'll see the gem below in the relay server console window.



This tool will be extremely useful as I spec out my implementation of the state service. For instance, if I want to see how the state service reacts if there is no Exclusive header, I'll simply modify the TransformData method to detect and remove the header.

I have also found Microsoft’s specification of the ASP.NET state service protocol. It doesn't look coherent but it will help me make my implementation compatible with their specifications.


Probing the ASP.NET State Service Part I

I have been thinking about developing an alternative ASP.NET State Service which can transparently replace the Microsoft-provided state service.

To be successful, the alternative state service has to convince the web server that it is the state service. That means I need to know the high-level communication protocol used between the state service and the web server.
   
How does the web server talk to the state service? Do they communicate via .NET remoting calls? Is there a proprietary protocol? Or is the state service really a fancy web service?
 
Let's investigate.

I'll create an echo server that will listen on a port. I can then set the echo server’s address as the state service address for a web application. If the state service protocol runs over TCP, I should be able to see data transmitted by the web application.

Here's the code snippet for a simple windows socket server that echoes received data:

int port = 24242; //as opposed to the default 42424 :)

IPEndPoint endPoint = new IPEndPoint(IPAddress.Any, port);
Socket server = new Socket(AddressFamily.InterNetwork, SocketType.Stream,ProtocolType.Tcp);
server.Bind(endPoint);
server.Listen(20);
Socket clientHandler = server.Accept();
byte[] transmission = new byte[1024];
clientHandler.Receive(transmission);

//print transmitted data
Console.WriteLine(Encoding.Default.GetString(transmission));
Console.ReadKey();
(To run the code snippet, you'll need to add using statements to reference System.Net and System.Net.Sockets namespaces.)

I then create a test web application and add the following line in the system.web section of the Web.config file:

<sessionState mode="StateServer" stateConnectionString="tcpip=localhost:24242" cookieless="false" timeout="20"/>
<!-- -->

This configures the ASP.NET application to use the state service, except it's actually our echo server.

I also add the following line of code to the web application's Page_Load event handler.

 Session.Add("Test Entry", "Test Data");

Now, I start the echo server (actually a console application) and fire up the web application to see if any transmitted data is captured and displayed in the console window.



Wow. Is that HTTP?
I see a PUT verb followed by some non-standard headers, followed by some binary data.
Well, that certainly makes my task easier considering that if the service was using .NET remoting calls; I'd have needed to delve into the innards of the .NET Remoting API.

I also notice that the transmitted data is not encrypted. I can clearly see "Test Data" in the transmission. Um, Microsoft -- this is not good.

How about the gibberish looking tidbit in the PUT statement? That must be some kind of identifier.

I add the following line in the system.web section of the Web.config file to see if it will cause the data to be encrypted.

<machineKey validationKey='016E9B3DAA748525DCDBAAC999BB390D63E7E1095F56B737887C10291567085B5A3A2142E6C86F06F07558D77260122C1174419212B6A117B6977B285EA8722B' />
<!-- -->

No such luck, however, the encoded data in parenthesis in the PUT verb line changed.
Hmm. Let's try to make some sense out of all these data.

The PUT Verb With Encoded Data:

After URL-decoding the data, I get  /3e50a960(iE+KOE6bwMI7BuHXun98zlcnkb8=)/miztsjiek5gvzu55km3xun55. The backslash is probably a delimiter and the text in parentheses is probably base64-encoded. (the "=" sign gave that away)

After running and observing the web application a few times, I figured the three parts of the PUT verb line are:

Part A: /3e50a960 is constant (I have no clue what it represents. Application ID? Machine ID? Some kind of magic number? I'll investigate)
Part B: (iE+KOE6bwMI7BuHXun98zlcnkb8=) is derived from MachineKey and is used by the state service to differentiate session data from different machines
Part C: /miztsjiek5gvzu55km3xun55 is the session ID

The Headers:

Host: The Hostname of the web application.
Timeout: The number of minutes a session can be idle before it is discarded.
Content-Length: The length of the binary data.
ExtraFlags: No clue. I'll investigate.
LockCookie: Looks eerily familiar. I'll investigate.

Binary Data:

This could be either information about an item to be updated, added or removed from the state service, in which case the web application retrieves and updates items as needed OR it could simply be a serialized list of all items to be stored in the session, in which case the web application reads all items at the beginning of a web request cycle and updates them at the end of the request cycle. To test my assumptions; I'll add a couple more lines of code to the Page_Load event handler.

Session.Add("test entry 2", "some test data");
Session.Add("test entry 3", "even MORE test data");

I then run the web application to see the transmitted data.



I can see the new items I added in the binary data area. I can now conclude that at the start of a web request, the web application retrieves this list, possibly modifies it and sends it back to the service at the end of the request.

Let me pause right here and think about this for a minute.

Is this model efficient? Is it better to read all items in the session at once and update them all at once, or is it better to read and update items as needed?

Well, since the items are stored by session ID and only one session ID is assigned to one user at any point in time. It means the chance that a session item is requested by multiple users/machines is negligible. Therefore there is no need to retrieve and update items as needed because there is no need for concurrency.

Another advantage with this model is that the state service does not need to know about items being stored. It simply stores whatever data is sent to it. This makes development easier.

Also, retrieving and updating items as needed increases the number of connections to the state service. This can greatly affect the scalability of a web application.

The downside to reading and updating all items at once is bandwidth-related: If an application stores a lot of information in the session but uses only a few at a time, the web application will move a lot of unnecessary data to and fro, which may clog the network. This may not be an issue if your web server and state service are physically close or run on the same computer.

A new question comes up; why is the web application NOT querying the state service for the session data when it starts? Does this mean any information already stored by the state service will be overwritten whenever the web application starts?

I ended up with more questions than I began with and so I still need to investigate further but at least I have gleaned some basic facts needed to start working on an alternative state service.

1. The ASP.NET web server and the state service communicate via HTTP.
2. The state service stores the binary data sent by the web server (probably in a large dictionary) using the Session ID + a derivative of the machine key as the key.
3. The state service does not need to itemize the data to be stored because all items are read and updated at once.
4. The transmitted data is not encrypted (at least not by default). The state service is not concerned about this since it simply stores data.