AI Powered transition from blocks to text

[vritant24]

An AI powered walkthrough showing which blocks map to which lines of code in a step-by-step basis accompanied with explanations.

Example flow:

When transitioning from blocks to text, provide a dialog asking if the user would like a walkthrough. If accepted, a walkthrough is dynamically generated through AI and a new view opens to progressively build the text file.

The UX:

• Two panes: left being blocks and right being the text file, with an area to show explanations.
  
• The blocks are all seen in the left pane and the text file starts out empty
• One-by-one, block(s) are highlighted, and the corresponding text code is added to the file, accompanied by an explanation on what the code is doing in the "explanation" view below. This continues until the entire text file is built.
• The walkthrough ends and the user is taken to the regular text file view that exists today.

Where AI comes in

The mapping from blocks to code exists today from what I understand, so progressively building the text code is relatively straightforward and doesn't require AI. Where AI would be useful is creating natural language-based explanations to accompany each step within the context of the game.

For example, from going to this block to text in the falling duck game:

controller.anyButton.onEvent(ControllerButtonEvent.Pressed, function () {
    mySprite.vy = -100
    animation.setAction(mySprite, ActionKind.Walking)
    mySprite.startEffect(effects.rings, 300)
})

The explanation may be

When any button is pressed, the duck jumps upward with a velocity of 100 with an effect of rings added to the duck.
<some explanation on how callbacks work>

A summary of the game can be generated through the text code and possible visual assets as well (for multimodal APIs). The summary can then be used as part of the context for each step for better explanations

Reasoning

• The step-by-step revealing of code leads to less cognitive overload as it allows the user to slowly build understanding.
• The natural-language explanations are useful guides for those new to writing text-based code.
• The block-to-text transition becomes slightly easier when you see a side-to-side mapping of block and text.
• The explanations would be useful for cases where there is no block mapping, for example enum and namespaces. As well as explaining in-editor the arcade api (controller, anim, game etc)

Possible future features:

• Ability to ask further questions for a specific step in the walkthrough. eg: "what is controller and why is it used here? I don't see it defined"