Programmatic access of AWS CDK CLI

[hassankhan]

Description

Hi there, thanks for creating this package! I was just wondering if there's any way of triggering CDK CLI commands programatically? This would really help with respect to integrating with existing tools and workflows.

RFC 300: Programmatic access of AWS CDK CLI

Roles

Role	User
Proposed by	@hassankhan, @mrgrain
Author(s)	@mrgrain
API Bar Raiser	@rix0rrr
Stakeholders

See RFC Process for details

Workflow

• Tracking issue created (label: status/proposed)
• API bar raiser assigned (ping us at #aws-cdk-rfcs if needed)
• Kick off meeting
• RFC pull request submitted (label: status/review)
• Community reach out (via Slack and/or Twitter)
• API signed-off (label api-approved applied to pull request)
• Final comments period (label: status/final-comments-period)
• Approved and merged (label: status/approved)
• Execution plan submitted (label: status/planning)
• Plan approved and merged (label: status/implementing)
• Implementation complete (label: status/done)

---

Author is responsible to progress the RFC according to this checklist, and
apply the relevant labels to this issue so that the RFC table in README gets
updated.

[rix0rrr]

Hi @hassankhan,

The aws-cdk NodeJS package does expose some library functions that can be used from JavaScript/TypeScript projects directly. However, it's not a polished experience at the moment, because our attention has been focused more towards the construct library so far. You're very welcome to try it and tell us if you run into any issues, though. As a last resort, you can always shell out to the command-line tool.

I'm also curious: could you tell us something about the kinds of integration that you're thinking of?

[hassankhan]

Hi @rix0rrr,

Thanks for the info! In an ideal world, I'd love to see something like:

import CDK from '@aws-cdk/cdk';

import MyApp from './MyApp';

const cdk = new CDK({ app: '', versionReporting: false });

cdk.deploy(new MyApp())
  .then(() => {
    return remove(new App());
  })

With regards to integration, essentially, our aim was to define resources such as Lambdas and DynamoDB tables once in our application code, and generate valid snippets from them and insert them into a template for deployment. For example, a theoretical DynamoDB model would look something like this (using dynamodb-data-mapper):

@table('pets')
class Pet {
    @hashKey()
    id: string;

    @rangeKey({defaultProvider: () => new Date()})
    createdAt: Date;

    @attribute()
    isCat?: boolean;
}

The table schema would then be read from this class as part of the deployment process, converted into its constituent Construct and added to the App/Stack. We could use the same approach for other resource types such as SNS Topics and Lambda functions.

[rix0rrr]

Okay, thanks. I definitely see the value of the first one, especially in defining one-off tasks like EMR jobs or integ tests.

For the second one: you can already do this today, even without tool integration.

If you define you data model in a library, you can use this library both in your runtime code as well as introspect it in your infrastructure code. Your build step would produce both a CloudFormation template as well as a JAR, and during deployment you deploy both together.

[rui-ktei]

+1
I mentioned this in a duplicate ticket as well.

[BDQ]

I'd like this too, in my case I'd like to wrap a deploy of an ECS cluster so I can scale it manually after the deploy is done (I create it with desiredCount: 0 so the CloudFormation creating is quicker).

[dsilvasc]

Additional use cases from aws/aws-cdk#1133 - synthesizing stacks from an IDE debugger and embedding into JVM build systems (like Gradle) without having to shell out to a CLI that depends on node.js.

[mmoulton]

My team is currently programmatically invoking CDK to support dynamic composition of constructs to create stacks based on the answers to question we ask in a custom CLI tool we wrote. We are currently primarily using it to spin up either SAM apps, or static sites using codepipeline, codebuild, rds, vpc, ssm params, s3 buckets, cloudfront, certs, zones, etc.

I'd very much like an officially supported way of accomplishing this. Currently we are doing some very dirty things to mimic the nature of the CDK CLI and the fact it executes the cdk app multiple times to meet dependencies.

If the CDK team does not have plans to officially support this, I'd very much appreciate not making it harder to achieve this as the notes in aws/aws-cdk#2044 and aws/aws-cdk#2016 suggest.

[KarthickEmis]

Is there anyway of triggering AWS CDK ClI (CDK synth , CDK deploy) commands pro-grammatically in typescript ?

[auser]

I found a way to handle this. I'll create an example repo (hopefully) to demonstrate how I accomplished this process, but the short version looks like this:

Create an AppStack instance and set the synthesizer to the output of app.run():

import { AppStacks } from 'aws-cdk/lib/api/cxapp/stacks'
import { Configuration } from 'aws-cdk/lib/settings';
// ...
const out = this._app.run();        
const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

const appStacks = new AppStacks({
    aws: this.sdk,
    configuration: configuration,
    synthesizer: async () => out
})

Using the AppStack instance, create an instance of the CdkToolkit:

const appStacks = await this.getAppStacks();

const provisioner = new CloudFormationDeploymentTarget({
    aws: this.sdk,
})
const cli = new CdkToolkit({ appStacks, provisioner });

In order to get the credentials, we'll need to muck around at a low-level with the AWS-SDK:

import { debug } from '@aws-rocket/core';
import AWS = require('aws-sdk')
import { SDK } from 'aws-cdk/lib/api/util/sdk';
import { Mode } from 'aws-cdk/lib/api/aws-auth/credentials'

async function getCredentialsConfig(sdk: SDK): Promise<any> {
    const region = await sdk.defaultRegion()
    const defaultAccount = await sdk.defaultAccount()
    const credentials = await (sdk as any).credentialsCache.get(defaultAccount, Mode.ForReading)
    
    return {
      region,
      credentials
    }
  }

export const awsCredentialsMiddleware = async (argv: any) => {
    debug(`Setting profile to: ${argv.profile}`)
    if (argv.profile) {
        const sdk = new SDK({
            profile: argv.profile
        })
        const credentials = await getCredentialsConfig(sdk)
        AWS.config.update(credentials)
        AWS.config.setPromisesDependency(Promise);
        argv.AWS = AWS
        argv.SDK = sdk
    }
    return argv
}

Finally, this all comes together using the actual deploy like so:

const appStacks = await this.getAppStacks();
const allStacks = await appStacks.listStacks()
const allStackNames = allStacks
    .map((s: cxapi.CloudFormationStackArtifact) => s.name)

const cli = await this.getCdkToolkit()
try {
    const res = await cli.deploy({
        stackNames: allStackNames,
        // `requireApproval` is so that the user never needs to 
        // accept anything for the deploy to actually occur
        requireApproval: RequireApproval.Never,
    })
    console.log('deploy result ->', res);
} catch (e) {
    console.error(`Error deploying`, e);
    process.exit(-1)
}

Hope this helps and hopfully this becomes a part of the actual library. I'll try to submit a PR if I can, but no promises. If anyone else has any better way of handling this, I'd love suggestions.

[jd-carroll]

This is all very interesting and hopeful that it will get implemented 🤞

@auser - Were you ever able to put together a repo / gist with all of this information? I follow the logic but I'm not sure how I follow how to assemble all of the pieces. Is each code snippet supposed to be in the same file and if so does each code section follow one an other? I can guess, but I also don't know where the this._app and this._argv come from.

@fulghum / @shivlaks - It looks like there is some movement towards an implementation. Do feel programmatic access might be limited in some areas or is the goal to have a full API access the same you would have on command line?

More or less, do you think something like the following would be supported:

import AWS from 'aws-sdk';
import { deploy } from 'aws-cdk';
import MyStack from './my-stack';

const cfnOutput = await deploy(new MyStack());
console.log(`Successfully deployed stack:\n${cfnOutput}`);

const s3FileAndDetails = { /* ... */ };
const s3 = new AWS.S3();
await s3.putObject(s3FileAndDetails).promise();

Also, instead of needing to run the configuration like @auser has above:

const argv = {
    '$0': this._argv['$0'],
    '_': this._argv['_']
}
const configuration = new Configuration(argv)
await configuration.load()

It would be great if we could skip the command line altogether and just pass in a json object with relevant configuration. (Thereby eliminating the need for a cdk.json, but one could exist with overrides)

// Use factory function to allow reading config overrides in async
const configuration = await Configuration.create({ /* json from cdk.json */ })

[ryan-mars]

If you want to synth programmatically to the directory of your choice...

// Let's say you want it synthed to ./build
const outdir = path.join(process.cwd(), 'build') 
// App's constructor let's you specify the outdir 
const app = new cdk.App({outdir})
new ApiLambdaCrudDynamoDBStack(app, 'ApiLambdaCrudDynamoDBExample')
app.synth()

If you want to inspect what was produced app.synth() returns a CloudAssembly which can be interrogated with getStack('your stack name')

I haven't tried deploying programmatically yet.

[jd-carroll]

When implementing this feature, would it make sense to do it in the context of aws-sdk-js-v3?
https://github.com/aws/aws-sdk-js-v3

[ralovely]

(One of) the premise of CDK being "programmatic CloudFormation", it opens the way for fully programmable infrastructure, potentially even starting with the account creation
(and that's awesome, we (building an "Ops as a Service" product) have been hoping for this for quite some time, even looking into developing our own solution).
Creating a full, real-life infrastructure requires more features than CloudFormation can offers, even considering CustomResources.
Some of those will require the SDK ; some others need to talk to 3rd party services ; fetch data before ; output/notify after.

Being able to use CDK not so much as a toolkit/cli, not as a complete solution, but as a lib, as a way to integrate the infrastructure-control layer into a programm would be great. As great as CDK promise to be, it's not a complete solution, probably can't, and shouldn't even try to be one.
Of course, one can always shell out, wrap around CDK CLI, but that's not ideal.

It ties in with a few other GitHub issues:

• Support for app.postDeploy() for post-deploy functions
• CLI: way to get outputs from deployed stack
• Stacks and/or Contructs should emit events
• Document how to use stack parameters (to some degree)