Skip to content

Commit c5016e4

Browse files
essweineessweine
committed
document diff utils
1 parent 95197b2 commit c5016e4

File tree

3 files changed

+227
-0
lines changed

3 files changed

+227
-0
lines changed

‎doc/bpmn/diffs.rst

+217
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
Diff Utilities
2+
==============
3+
4+
.. note::
5+
6+
This is a brand new feature so it may change.
7+
8+
It is possible to generate comparisions between two BPMN specs and also to compare an existing workflow instance
9+
against a spec diff to provide information about whether the spec can be updated for the instance.
10+
11+
Individual diffs provide information about a single spec or workflow and spec. There are also two helper functiond
12+
for calculating diffs of dependencies for a top level spec or workflow and its subprocesses, and a workflow migration
13+
function.
14+
15+
Creating a diff requires a serializer :code:`registry` (see :ref:`serializing_custom_objects` for more information
16+
about this). The serializer already needs to know about all the attributes of each task spec; it also knows how to
17+
create dictionary representations of the objects. Therefore, we can serialize an object and just compare the output to
18+
figure which attributes have changed.
19+
20+
Let's add some of the specs we used earlier in this tutorial:
21+
22+
.. code-block:: console
23+
24+
./runner.py -e spiff_example.spiff.diffs add -p order_product \
25+
-b bpmn/tutorial/task_types.bpmn \
26+
-d bpmn/tutorial/product_prices.dmn
27+
28+
./runner.py -e spiff_example.spiff.diffs add -p order_product \
29+
-b bpmn/tutorial/gateway_types.bpmn \
30+
-d bpmn/tutorial/{product_prices,shipping_costs}.dmn
31+
32+
./runner.py -e spiff_example.spiff.diffs add -p order_product \
33+
-b bpmn/tutorial/{top_level,call_activity}.bpmn \
34+
-d bpmn/tutorial/{shipping_costs,product_prices}.dmn
35+
36+
./runner.py -e spiff_example.spiff.diffs add -p order_product \
37+
-b bpmn/tutorial/{top_level_script,call_activity_script}.bpmn \
38+
-d bpmn/tutorial/shipping_costs.dmn
39+
40+
The IDs of the specs we've added can be obtained with:
41+
42+
.. code-block:: console
43+
44+
./runner.py -e spiff_example.spiff.diffs list_specs
45+
46+
09400c6b-5e42-499d-964a-1e9fe9673e51 order_product bpmn/tutorial/top_level.bpmn
47+
9da66c67-863f-4b88-96f0-76e76febccd0 order_product bpmn/tutorial/gateway_types.bpmn
48+
e0d11baa-c5c8-43bd-bf07-fe4dece39a07 order_product bpmn/tutorial/task_types.bpmn
49+
f679a7ca-298a-4bff-8b2f-6101948715a9 order_product bpmn/tutorial/top_level_script.bpmn
50+
51+
52+
Model Diffs
53+
-----------
54+
55+
First we'll compare :bpmn:`task_types.bpmn` and :bpmn:`gateway_types.bpmn`. The first diagram is very basic,
56+
containing only one of each task type; the second diagram introduces gateways. Therefore the inputs and outputs of
57+
several tasks have changed and a number of new tasks were added.
58+
59+
.. code-block: console
60+
61+
./runner.py -e spiff_example.spiff.diffs diff_spec -o e0d11baa-c5c8-43bd-bf07-fe4dece39a07 -n 9da66c67-863f-4b88-96f0-76e76febccd0
62+
63+
Those diagrams don't have dependencies, but :bpmn:`top_level.bpmn` and :bpmn:`top_level_script.bpmn` do have
64+
dependencies (:bpmn:`call_activity.bpmn` and :bpmn:`call_activity_script.bpmn`). See
65+
:ref:`custom_classes_and_functions` for a description of the changes. Adding the :code:`-d` will include
66+
any dependencies in the diff output.
67+
68+
.. code-block:: console
69+
70+
./runner.py -e spiff_example.spiff.diffs diff_spec -d
71+
-o 09400c6b-5e42-499d-964a-1e9fe9673e51 \
72+
-n f679a7ca-298a-4bff-8b2f-6101948715a9
73+
74+
We pass the spec ids into our engine, which deserializes the specs and creates a :code:`SpecDiff` to return (see
75+
:app:`engine/engine.py`.
76+
77+
.. code-block:: python
78+
79+
def diff_spec(self, original_id, new_id):
80+
original, _ = self.serializer.get_workflow_spec(original_id, include_dependencies=False)
81+
new, _ = self.serializer.get_workflow_spec(new_id, include_dependencies=False)
82+
return SpecDiff(self.serializer.registry, original, new)
83+
84+
def diff_dependencies(self, original_id, new_id):
85+
_, original = self.serializer.get_workflow_spec(original_id, include_dependencies=True)
86+
_, new = self.serializer.get_workflow_spec(new_id, include_dependencies=True)
87+
return diff_dependencies(self.serializer.registry, original, new)
88+
89+
The :code:`SpecDiff` object provides
90+
91+
- a list of task specs that have been added in the new version
92+
- a mapping of original task spec to a summary of changes in the new version
93+
- an alignment of task spec from the original workflow to the task spec in the new version
94+
95+
The code for displaying the output of a single spec diff is in :app:`cli/diff_result.py`. I will not go into
96+
detail about how it works here since the bulk of it is just formatting.
97+
98+
The libary also has a helper function `diff_dependencies`, which takes two dictionaries of subworkflow specs
99+
(the output of :code:`get_subprocess_specs` method of the parser can also be used directly here). This method
100+
returns a mapping of name -> :code:`SpecDiff` for each dependent workflow that could be matched by name and a list
101+
of the names of specs in the new version that did not exist in the old.
102+
103+
Instance Diffs
104+
--------------
105+
106+
Suppose we save one instance of our simplest model without completing any tasks and another instance where we
107+
proceed until our order is displayed before saving. We can list our instances with this command:
108+
109+
.. code-block:: console
110+
111+
./runner.py -e spiff_example.spiff.diffs list_instances
112+
113+
4af0e043-6fd6-448d-85eb-d4e86067433e order_product 2024-07-02 17:46:57 2024-07-02 17:47:00
114+
af180ef6-0437-41fe-b745-8ec4084f3c57 order_product 2024-07-02 17:47:05 2024-07-02 17:47:30
115+
116+
If we diff each of these instances against the version in which we've added gateways, we'll see a list of
117+
tasks whose specs have changed and their states.
118+
119+
.. code-block:: console
120+
121+
./runner.py -e spiff_example.spiff.diffs diff_workflow \
122+
-s 9da66c67-863f-4b88-96f0-76e76febccd0 \
123+
-w 4af0e043-6fd6-448d-85eb-d4e86067433e
124+
125+
We'll pass these IDs to our engine, which will return a :code:`WorkflowDiff` of the top level workflow and
126+
a dictionary of subprocess id -> :code:`WorkflowDiff` for any existing subprocesses.
127+
128+
.. code-block:: python
129+
130+
def diff_workflow(self, wf_id, spec_id):
131+
wf = self.serializer.get_workflow(wf_id)
132+
spec, deps = self.serializer.get_workflow_spec(spec_id)
133+
return diff_workflow(self.serializer.registry, wf, spec, deps)
134+
135+
We can retrieve the current spec and its dependencies from the instantiated workflow, so we only need to pass in
136+
the newer version of the spec and its dependencies.
137+
138+
The :code:`WorkflowDiff` object provides
139+
140+
- a list of tasks whose specs have been removed from the new spec
141+
- a list of tasks whose specs have been updated in the new spec
142+
- a mapping of task -> new task spec for each task where an alignment exists in the spec diff
143+
144+
Code for displaying the results is in :app:`cli/diff_result.py`.
145+
146+
If you start an instance of the first version with a subprocess and stop after customizing a product, and
147+
compare it with the second, you'll see completed tasks from the subprocess in the workflow diff output.
148+
149+
Migration Example
150+
-----------------
151+
152+
In some cases, it may be possible to migrate an existing workflow to a new spec. This is actually quite
153+
simple to accomplish:
154+
155+
.. code-block:: python
156+
157+
def migrate_workflow(self, wf_id, spec_id, validate=True):
158+
159+
wf = self.serializer.get_workflow(wf_id)
160+
spec, deps = self.serializer.get_workflow_spec(spec_id)
161+
wf_diff, sp_diffs = diff_workflow(self.serializer.registry, wf, spec, deps)
162+
163+
if validate and not self.can_migrate(wf_diff, sp_diffs):
164+
raise Exception('Workflow is not safe to migrate!')
165+
166+
migrate_workflow(wf_diff, wf, spec)
167+
for sp_id, sp in wf.subprocesses.items():
168+
migrate_workflow(sp_diffs[sp_id], sp, deps.get(sp.spec.name))
169+
wf.subprocess_specs = deps
170+
171+
self.serializer.delete_workflow(wf_id)
172+
return self.serializer.create_workflow(wf, spec_id)
173+
174+
The :code:`migrate_workflow` function updates the task specs of the workflow based on the alignment in the
175+
diff and sets the spec. We have to do this for the top level workflow as well as any subwokflows that have
176+
been created. We also update the dependencies on the top level workflow (subworkflows do not have dependencies).
177+
178+
This function has an optional :code:`reset_mask` argument that can be used to override the default mask of
179+
:code:`TaskState.READY|TaskState.WAITING`. The children of matching tasks will be dropped and recreated based
180+
on the new spec so that structural changes will be reflected in future tasks.
181+
182+
In this application we delete the old workflow and reserialize with the new, but that's an application
183+
based decision and it would be possible to save both.
184+
185+
We can migrate the version that we did not advance with the following command:
186+
187+
.. code-block:: console
188+
189+
./runner.py -e spiff_example.spiff.diffs migrate \
190+
-s 9da66c67-863f-4b88-96f0-76e76febccd0 \
191+
-w 4af0e043-6fd6-448d-85eb-d4e86067433e
192+
193+
Deciding whether to migrate is the hard part. We use a simple algorithm in this application: if any tasks with
194+
specs that have been changed or removed have completed or started running, or any subprocesses have changed, we
195+
assume the workflow cannot be migrated.
196+
197+
.. code-block:: python
198+
199+
def can_migrate(self, wf_diff, sp_diffs):
200+
201+
def safe(result):
202+
mask = TaskState.COMPLETED|TaskState.STARTED
203+
tasks = result.changed + result.removed
204+
return len(filter_tasks(tasks, state=mask)) == 0
205+
206+
for diff in sp_diffs.values():
207+
if diff is None or not safe(diff):
208+
return False
209+
return safe(wf_diff)
210+
211+
This is fairly restrictive and some workflows might be migrateable even when these conditions apply (for example,
212+
perhaps correcting a typo in completed task shouldn't block future structural changes from being applied). However,
213+
there isn't really a one-size-fits-all decision to be made. And it could end up being a massiveeffort to develop a
214+
UI that allows decisions like this to be made, so I haven't done any of that in this application.
215+
216+
The hope is that the `SpecDiff` and `WorkflowDiff` objects can provide the necessary information to make these
217+
decisions.

‎doc/bpmn/index.rst

+8
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,14 @@ Custom Tasks
6565

6666
custom_task_spec
6767

68+
Diffs
69+
-----
70+
71+
.. toctree::
72+
:maxdepth: 2
73+
74+
diffs
75+
6876
Logging
6977
-------
7078

‎doc/bpmn/script_engine.rst

+2
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ You'll get an error, because imports have been restricted.
5050
two script engines (that is the only difference between the two configurations). If you've made any
5151
serializer or parser customizations, this is not likely to be possible.
5252

53+
.. _custom_classes_and_functions:
54+
5355
Making Custom Classes and Functions Available
5456
=============================================
5557

0 commit comments

Comments
 (0)