For Cylc 8 we are adding an initial implementation of GraphQL, to replace the previous REST API. Besides the technologies in the API’s, another difference is that for the REST API, its main consumer was a PyGTK GUI.
The new GraphQL API, on the other hand, will be used mainly by a Vue.js Web application. So a few things need to be done in a different way due to the jump from Desktop GUI to Web GUI.
The current implementation is under review in a pull request at the moment. It includes Python libraries for GraphQL, as it is expected, but also a Protobuf data model that can be visualized in the figure below.
In Cylc, the
Scheduler object is responsible for managing Tasks in a Workflow. It does so in its
main loop, which is similar to the main loops in game engines. It runs periodically checking the
state of several objects, and updates them.
A new class
JobPool is being added in Cylc 8 to maintain a pool of jobs and their states. This
is used by the
Scheduler to update the jobs, similarly to how it does for tasks.
Every time the main loop runs, it calls a method (
Scheduler.update_data_structure) to update the data
structures used by Cylc, including the data structures used by the GraphQL engine in Cylc. Other methods
that are responsible for updating the state of Tasks are now updating the state of Jobs too.
WsDataMgr, another new object being added.
is where the Workflow Tasks, their parent Tasks, Task Families, and other objects required for
the Workflow such as Prerequisites and Conditions are calculated, and organized into dictionaries.
These dictionaries are kept in memory, updated periodically by
Scheduler, and accessed by the
After the initial run of the main loop, we are ready to serve GraphQL data from our workflow
in Cylc. That is done by the
SuiteRuntimeServer, a PyZMQ server. It got a new method
pb_entire_workflow that populates the
PbEntireWorkflow object, and serializes it back to the client.
Cylc uses a SQLite database for part of the workflow state (e.g. recovery) but for the GraphQL we are not really accessing any database directly.
So the implementation in Cylc is probably different than what most blogs and tutorials use as example.
It is still early work, and many things may change, but Cylc 8 is getting near its first alpha release with GraphQL. Check out https://github.com/cylc/cylc-flow for more.
The following references are useful to track the rationale behind this work in Cylc: