Home

Modular Pipelines

Modular

consisting of separate parts that, when combined, form a complete whole

Pipeline

any channel or means whereby something is passed on

Fundamentals

The building blocks of your pipelines are called Modules. Modules should be granular and do the smallest thing necessary.

Module

a self-contained unit or item, such as an assembly of electronic components and associated wiring or a segment of computer software, which itself performs a defined task and can be linked with other such units to form a larger system

Modules can retrieve other modules, and access information from them.

Strong Typing

Modules are strongly typed, so we can return clear, concrete objects, and other modules have direct access to those strong objects, without any need for casting or guessing the type, or guessing keys from a dictionary.

var myModule = await GetModule<MyFirstModule>();
var string1 = myModule.Value!.MyFirstString;
var string2 = myModule.Value!.MySecondString;

Custom Types

A module isn't restricted to a pre-determined type either. You can pass the Type of object that you want to return when you inherit from the base Module class

public class MyModule : Module<MyCustomClass>

public class PingApiModule : Module<HttpResponseMessage>

You'll then be instructed by the compiler to make sure the return type of your main ExecuteAsync method matches the Type you've set up.

protected override async Task<ModuleResult<MyCustomClass>?> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)

Optional Data

You can choose not to set a Type and the default will be an IDictionary<string, object>.

Now returning an object isn't mandatory either. You can return null or use the method NothingAsync();.

Automatic Parallelisation and Explicit Dependencies

Modules will all try to run in parallel if possible. But if a Module depends on another Module, it is smart enough to automatically wait for the dependent module to finish before executing.

Dependencies are configured by adding an attribute on your Module. This also makes it clear to navigate through your pipeline, as with your IDE/Intellisense, you can click through to other Modules with ease.

[DependsOn<MyOtherModule>]
public class MyModule : Module

Checking a Module's status

When you get another Module, you'll be passed an object that has the data you returned, as well as some information about its execution. So you can have logic in your pipeline for if another module was skipped for example.

var myModule = await GetModule<MyOptionalModule>();

if (myModule.ModuleResultType == ModuleResultType.Skipped)
{
    return null;
}

return await DoSomethingAsync();

or if a Module failed, but it was configured to not stop the pipeline, you could check its Exception.

var myModule = await GetModule<MyOptionalModule>();

if (gitModule.Exception is ItemAlreadyExistsException)
{
    return null;
}

return await DoSomethingAsync();

Example

So for example, we want to provision some Azure services like this:

• A user assigned identity
• A blob storage account that can only be accessed via the user assigned identity we created
• A blob storage container under that account
• An azure function, with our user assigned identity being used for its Identity

That would look like this:

public class ProvisionUserAssignedIdentityModule : Module<UserAssignedIdentityResource>
{
    protected override async Task<ModuleResult<UserAssignedIdentityResource>?> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
    {
        var userAssignedIdentityProvisionResponse = await context.Azure().Provisioner.Security.UserAssignedIdentity(
            new AzureResourceIdentifier("MySubscription", "MyResourceGroup", "MyUserIdentity"),
            new UserAssignedIdentityData(AzureLocation.UKSouth)
        );
        
        return userAssignedIdentityProvisionResponse.Value;
    }
}

public class ProvisionBlobStorageAccountModule : Module<StorageAccountResource>
{
    protected override async Task<ModuleResult<StorageAccountResource>?> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
    {
        var blobStorageAccountProvisionResponse = await context.Azure().Provisioner.Storage.StorageAccount(
            new AzureResourceIdentifier("MySubscription", "MyResourceGroup", "MyStorage"),
            new StorageAccountCreateOrUpdateContent(new StorageSku(StorageSkuName.StandardGrs), StorageKind.BlobStorage, AzureLocation.UKSouth)
        );
        
        return blobStorageAccountProvisionResponse.Value;
    }
}

[DependsOn<ProvisionBlobStorageAccountModule>]
public class ProvisionBlobStorageContainerModule : Module<BlobContainerResource>
{
    protected override async Task<ModuleResult<BlobContainerResource>?> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
    {
        var blobStorageAccount = await GetModule<ProvisionBlobStorageAccountModule>();

        var blobContainerProvisionResponse = await context.Azure().Provisioner.Storage.BlobContainer(
            blobStorageAccount.Value!.Id,
            "MyContainer",
            new BlobContainerData()
        );
        
        return blobContainerProvisionResponse.Value;
    }
}

[DependsOn<ProvisionBlobStorageAccountModule>]
[DependsOn<ProvisionUserAssignedIdentityModule>]
public class AssignAccessToBlobStorageModule : Module<RoleAssignmentResource>
{
    protected override async Task<ModuleResult<RoleAssignmentResource>?> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
    {
        var userAssignedIdentity = await GetModule<ProvisionUserAssignedIdentityModule>();
        
        var storageAccount = await GetModule<ProvisionBlobStorageAccountModule>();
        
        var roleAssignmentResource = await context.Azure().Provisioner.Security.RoleAssignment(
            storageAccount.Value!.Id,
            new RoleAssignmentCreateOrUpdateContent(WellKnownRoleDefinitions.BlobStorageOwnerDefinitionId, userAssignedIdentity.Value!.Data.PrincipalId!.Value)
        );
        
        return roleAssignmentResource.Value;
    }
}

[DependsOn<ProvisionUserAssignedIdentityModule>]
[DependsOn<ProvisionBlobStorageAccountModule>]
[DependsOn<ProvisionBlobStorageContainerModule>]
public class ProvisionAzureFunction : Module<WebSiteResource>
{
    protected override async Task<ModuleResult<WebSiteResource>?> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
    {
        var userAssignedIdentity = await GetModule<ProvisionUserAssignedIdentityModule>();

        var storageAccount = await GetModule<ProvisionBlobStorageAccountModule>();
        var blobContainer = await GetModule<ProvisionBlobStorageContainerModule>();
        
        var functionProvisionResponse = await context.Azure().Provisioner.Compute.WebSite(
            new AzureResourceIdentifier("MySubscription", "MyResourceGroup", "MyFunction"),
            new WebSiteData(AzureLocation.UKSouth)
            {
                Identity = new ManagedServiceIdentity(ManagedServiceIdentityType.UserAssigned)
                {
                    UserAssignedIdentities = { { userAssignedIdentity.Value!.Id, new UserAssignedIdentity() } }
                },
                SiteConfig = new SiteConfigProperties
                {
                    AppSettings = new List<AppServiceNameValuePair>
                    {
                        new()
                        {
                            Name = "BlobStorageConnectionString",
                            Value = storageAccount.Value!.Data.PrimaryEndpoints.BlobUri.AbsoluteUri
                        },
                        new()
                        {
                            Name = "BlobContainerName",
                            Value = blobContainer.Value!.Data.Name
                        }
                    }
                }
                // ... Other properties
            }
        );
        
        return functionProvisionResponse.Value;
    }