adapt-migrations

Todos

#1

Commands API

https://github.com/cgkineo/adapt-migrations/blob/master/api/commands.js

• load({ cwd, cachePath, scripts }) - loads all migration tasks
• capture({ cwd, content, fromPlugins }) - captures current plugins and content
• migrate({ cwd, toPlugins }) - migrates content from capture to new plugins
• test({ cwd }) - tests the migrations with dummy content

Migration script API

Functions:

• describe(description, describeFunction) Describe a migration
• whereContent(description, contentFilterFunction) Limit when the migration runs, return true/false/throw Error
• whereFromPlugin(description, fromPluginFilterFunction) Limit when the migration runs, return true/false/throw Error
• whereToPlugin(description, toPluginFilterFunction) Limit when the migration runs, return true/false/throw Error
• mutateContent(contentFunction) Change content, return true/false/throw Error
• checkContent(contentFunction) Check content, return true/false/throw Error
• addPlugin(description, pluginConfig) Add a plugin
• updatePlugin(description, pluginConfig) Update a plugin
• removePlugin(description, pluginConfig) Remove a plugin
• throwError(description) Throw an error
• testSuccessWhere(description, { fromPlugins, toPlugins, content }) Supply some tests content which should end in success
• testStopWhere(description, { fromPlugins, toPlugins, content }) Supply some tests content which should end prematurely
• testErrorWhere(description, { fromPlugins, toPlugins, content }) Supply some tests content which will trigger an error

Arguments:

• describeFunction = () => {} Function body has a collection of migration script functions
• contentFilterFunction = content => {} Function body should return true/false/throw Error
• fromPluginFilterFunction = fromPlugins => {} Function body should return true/false/throw Error
• toPluginFilterFunction = toPlugins => {} Function body should return true/false/throw Error
• contentFunction = content => { } Function body should mutate or check the content, returning true/false/throw Error
• fromPlugins = [{ name: 'quickNav , version: '1.0.0' }] Test data describing the original plugins
• toPlugins = [{ name: 'pageNav , version: '1.0.0' }] Test data describing the destination plugins
• pluginConfig = { name: 'pageNav , version: '1.0.0' } Describes a plugin
• content = [{ _id: 'c-05, ... }] Test content for the course content

Grunt Commands

grunt migration:capture # captures current plugins and content
# do plugin/fw updates
grunt migration:migrate # migrates content from capture to new plugins
grunt migration:test # tests the migrations with dummy content
grunt migration:test --file=adapt-contrib-text/migrations/text.js # tests the migrations with dummy content

Description of how

The whole describe function block is executed as a normal function, from top to bottom, always. It does not return early.

When the describe function block is executed, we're effectively using javascript function calls (the step functions) to define a single migration script (task) and its steps and then that migration script (task, and its steps) is run in part, to ascertain if it is applicable (using the where section), or in full when it is applicable by running through every step.

The step functions (whereFromPlugins, mutateContent, etc) have two phases:

Step function phases

1. Task definition phase: Adding themselves as steps inside a task for later execution

describe(description, async () => { // Make a task
  // where/selection/applicability section
  whereFromPlugin(description, version) // Define as step 1 in the task
  whereContent(description, () => {}) // Define as step 2 in the task
  // mutation section to make changes
  mutateContent(async content => {}) // Define as step 3 in the task
  // checking section to ensure changes, content is immutable 
  checkContent(async content => {}) // Define as step 4 in the task
  // plugin progression section
  addPlugin(description, { name, version }) // Define as step 5 in the task
  updatePlugin(description, { name, version }) // Define as step 6 in the task
  removePlugin(description, { name, version }) // Define as step 7 in the task
  // testing data for grunt migration:test
  testSuccessWhere({ fromPlugins, toPlugins, content })
  testStopWhere({ fromPlugins, toPlugins, content })
  testErrorWhere({ fromPlugins, toPlugins, content })
})

2. Execution phase: When used inside any other executing utility function block

describe(name, async () => {
  mutateContent(name, async content => { // Execute the task step 1
    if (whereFromPlugin(name, version)) { // Execute the function immediately
      // the plugin version is matched
    }
  })
}) 

It's these two phases which decouple the definition of and execution of migration scripts, whilst using the same block of javascript. This was the simplest form I could think of, without having too many rules or nesting but whilst also providing flexibility, imply an order and convey concise meaning in as few words as possible.

We define variables and steps in the definition phase of the migration script (task) for later execution. The describe function doesn't return early at whereFromPlugins, both because functions can't implicitly return early, and because the function is executed in a definition phase, where it's just adding a description of itself to a task for later use.

whereFromPlugins is a single step in a task, it will be executed multiple times as the migrations progress, this is to find out whether the task is applicable and whether the task should proceed through all of the steps until conclusion.

Tasks and steps have three results: success (true), stop (false) or error (throw Error). Using those return values and having some of the step functions marked up as "where" functions, we can selectively define and execute a variety of migrations scripts, for a variety of courses, with a fun array of predictable outcomes.

References: Migration files are loaded:

adapt-migrations/lib/Task.js

Lines 293 to 296 in 1b156d8

	for (const filePath of modules) {
	Task.currentFile = filePath
	try {
	await import('file://' + filePath.replace(/\\/g, '/'))

Capturing the description and callback of each describe function call to make a new task:

adapt-migrations/api/describe.js

Line 11 in 1b156d8

new Task({ description, load })

Each of the describe blocks in the file are executed one by one:

adapt-migrations/lib/Task.js

Lines 297 to 302 in 1b156d8

	for (const task of Task.described) {
	Task.items.push(task)
	Task.stackUp(task)
	await task.load()
	Task.stackDown()
	}

The step functions are executed and deferred:

adapt-migrations/api/where.js

Lines 4 to 10 in 1b156d8

	export function whereContent (description, callback) {
	return deferOrRunWrap(({ content }) => {
	return successStopOrErrorWrap('whereContent', description, async () => {
	return callback(content)
	})
	}, { description, type: 'where' })
	};

The step functions add themselves as tests or steps to the currently loading task:

adapt-migrations/lib/lifecycle.js

Line 23 in 1b156d8

Task.current[options.type === 'test' ? 'tests' : 'steps'].push(callback)

After the loading phase, on each run of the migrations scripts, applicable tasks are selected for execution until no more tasks can be executed:

adapt-migrations/lib/Task.js

Lines 320 to 323 in 1b156d8

	for (const task of clonedTasks.filter(item => !item.isComplete)) {
	const isApplicable = await task.isApplicable({ cwd, journal, logger })
	if (isApplicable) toRun.push(task)
	}

Tasks are applicable if all of their where steps come back success:

adapt-migrations/lib/Task.js

Lines 184 to 197 in 1b156d8

	const wheres = this.getImmediateNextStepsOfType(0, 'where')
	for (const where of wheres) {
	this.isRunning = true
	const result = await where(this.context)
	this.isRunning = false
	if (result) continue
	shouldRun = false
	break
	}
	this.context = null
	Task.isRunning = false
	Task.stackDown()
	return shouldRun

The applicable tasks are run:

adapt-migrations/lib/Task.js

Lines 330 to 336 in 1b156d8

	for (const task of toRun) {
	const { hasErrored } = await task.run({ cwd, journal, logger })
	if (hasErrored) {
	journal.undoToIndex(lastJournalEntryIndex)
	break
	}
	}

The last success, stop or error step determines if the task failed or completed successfully or stopped because it wasn't applicable:

adapt-migrations/lib/lifecycle.js

Lines 35 to 41 in 1b156d8

	successOrStop = await callback()
	} catch (err) {
	error = err
	logger.error(`${taskName} -- ${description}`, `${error}`)
	error.stack = Object.entries(Task.mapCacheToSource).reduce((stack, [cache, source]) => {
	return stack.replaceAll(cache, source)
	}, error.stack)

Here is checkContent, freezing the data and dealing with its result:

adapt-migrations/api/data.js

Lines 13 to 24 in 1b156d8

	return successStopOrErrorWrap('checkContent', description, async () => {
	context.journal.freeze()
	let result
	try {
	result = await callback(context.content)
	} catch (err) {
	context.journal.unfreeze()
	throw err
	}
	context.journal.unfreeze()
	return result
	})