I am aware of only two workflow managers that support cyclic workflows. Cylc and StackStorm. I won’t enter into details about these two tools, but I must note that I worked on Cylc during my employment with NIWA, in New Zealand.
In this post I will only show a very simple workflow called
five first using
Cylc, and then the same workflow with StackStorm.
First let’s take a look at the source code of this workflow with Cylc 8 and plot it.
[scheduling] cycling mode = integer initial cycle point = 1 [[queues]] [[[default]]] limit = 1 [[graph]] R1 = "prep => foo" P1 = "foo[-P1] => foo => bar" [runtime] [[root]] script="sleep 5" [[prep]] [[foo]] [[bar]]
"foo[-P1] => foo => bar" is where the recursion occurs, creating
a cycle in the workflow.
Installing Cylc requires just
pip install cylc-flow. After that, with the workflow
installed, we are ready to run it.
cylc install -c ~/cylc-src/five --flow-name five cylc play --no-detach five/run1
The workflow will run forever, incrementing the cycle points, and triggering the tasks
five workflow source. So you will have
foo.1 (foo in the first cycle point),
that triggers both
bar.1 and also
foo.2 (foo in the second cycle point) and so it
StackStorm requires more work to get everything up and running. Luckily they provide a Docker Compose installation. So after the servers have been started with Docker we are ready to create a “pack” (a neat way to organize separate installation files).
mkdir -p /opt/stackstorm/packs/kinow/ touch /opt/stackstorm/packs/kinow/pack.yaml
--- name : kinow description: kinow version: 1.0.0 author: kinow email: kinow@localhost
And install the new pack.
st2 pack install file:///opt/stackstorm/packs/kinow
Now we create a new workflow and an action to run the workflow — I think this step is optional, and you could have just an action but I was following one section of the docs that had it this way.
mkdir -p /opt/stackstorm/packs/kinow/actions/workflows touch /opt/stackstorm/packs/kinow/actions/five.yaml touch /opt/stackstorm/packs/kinow/actions/workflows/five.yaml
--- name: five pack: kinow description: five runner_type: orquesta entry_point: workflows/five.yaml enabled: true
And now create the action in StackStorm, so we can run it via command line or with the UI.
st2 action create /opt/stackstorm/packs/kinow/actions/five.yaml
And here’s the
five workflow source for StackStorm, producing something very
similar (if no identical) to the graph produced by Cylc 8.
version: 1.0 description: five tasks: prep: action: core.local cmd="sleep 5" next: - when: <% succeeded() %> do: - foo foo: action: core.local cmd="sleep 5" next: - when: <% succeeded() %> do: - foo - bar bar: action: core.local cmd="sleep 5"
foo is calling itself, creating a cycle in the workflow. And to run the
st2 run kinow.five
The StackStorm UI does not appear to support showing the graph of the workflow static or dynamically. But there is a community contributed UI called rehearsal that plots an Orquesta workflow given its source.
Both Cylc and StackStorm support Directed Cyclic Graphs in workflows, which is really rare amongst workflow managers (or workflow standards, as I think WDL/CWL also do not support cyclic workflows yet.)
There are many pros and cons for each tool but that will have to be for a future
post. To finish this post here’s a screenshot of the StackStorm UI, followed by
one of the Cylc 8 UI. Both showing the workflow