Blog Taxonomy

Home

How the Skosmos Widget Wiki plugin works

kinow @ Dec 24, 2018 15:43:43 ()

Skosmos can be extended through plugins, or widgets (a widget is a plugin for Skosmos). You can read more about how plugins work in Skosmos here.

This post is a note-to-self, explaining how the Skosmos Widget Wiki plugin works. This is a plugin - or widget - that displays Wikipedia information when the concept supports it.

Here’s an image of the plugin in action. Or you can go to a live instance of Skosmos that has the plugin enabled.

Skosmos Widget Wiki plugin in action

Under the hood

The plugin exposes a JavaScript callback via its plugin.json file. In the callback, it receives a data object, with the following properties.

  • uri, the concept identifier, e.g. http://www.yso.fi/onto/yso/p107650”
  • prefLabels, the concepts preferred labels, e.g. “Central Asia”@en, “Keski-Aasia”@fi
  • pageType, defaults to page
  • json-ld, which contains the JSON-LD @context, as well as a graph property

If the pageType is not page, or if there are no preferred labels, or if there is no JSON-LD data, the plugin is not activated.

( Read more ... )

Running Cylc tasks on PBS Torque with Docker

kinow @ Dec 22, 2018 11:32:28 ()

A few days ago I saw a post at the Cylc Google Group, about file permissions for files generated by Cylc. The post was related to content created by Cylc, but in an environment with PBS.

For context, Cylc is an Open Source meta-scheduler, written in Python, that allows you to define cycle points with dependencies. These cycle points can be simple incremental integer numbers, or ISO8601 periods or points (e.g. run every 5 minutes, from 10 days ago until the next year). Cylc takes care to create an execution schedule for you, and delegate that to a system that runs your workflow. I work full time on this amazing Open Source tool!

Such system could be the local computer in background, batch systems such as at, or PBS. PBS was created for NASA, to manage executing jobs taking into consideration cluster resources, and also using queues, priorities, and other features useful for HPC programming. Later PBS was acquired by Altair, an Open Source version OpenPBS was created, and later abandoned. And there is another fork called PBS Torque. I first encountered PBS at the São Paulo University, in Brazil, where they had a PBS Torque cluster.

Running PBS Torque with Docker

Even though I have access to an environment with Cylc and with PBS, I decided to give it a try and see how hard it would be to reproduce it with Docker. One thing that I like about this approach is the possibility to share the work with others online. I believe it improves communication, agility, and can be useful for posterity.

( Read more ... )

What the weather forecast looks like in Sao Paulo, Brazil

kinow @ Oct 25, 2018 19:34:11 ()

I grew up watching the weather forecast in Sao Paulo, Brazil. Where it is called “previsão do tempo”, which is the literal translation in Portuguese.

This post contains some recent screen shots of the weather forecast as it is presented in one of the main TV channels, as well as some screenshots from an online video of other local institutions that either provide the forecast, or help running the NWP models.

( Read more ... )

A couple of class diagrams of JupyterHub

kinow @ Oct 06, 2018 21:43:44 ()

Started on a new project last Monday. One of the tasks in this project involves a new design for the Web layer. And as the application is quite similar to JupyterHub, we are all learning more about its internal API and general system design.

This post contains only two class diagrams created with PyCharm. One is actually a SQLAlchemy ORM diagram, below.

And the class diagram (which I removed object and a tried to make it simpler to interpret).

I enjoyed the parts of the code and the part of the system design that I could read about so far. But that’s all for now, until I have more time to learn more about the project and the code.

p.s. there are spawners and other implementation classes in other GitHub repositories… so a more complete diagram may come later on

Cylc Scheduler Internals - Part 3

kinow @ Aug 18, 2018 18:27:37 ()

This is part 2, in a series of posts about Cylc internals. The part 1 had the beginning of the workflow. part 2 documented from the moment the method configure() is called. This post will continue right after the continue() method returns, going on with the next method: run().

NB: this is a post to remember things, not really expecting to give someone enough information to be able to hack the Cylc Scheduler (though you can and would have fun!).

At this point, the Suite Server program must have been initialized, with the objects that it requires, and with everything configured. So this method is the one that starts the whole work on the tasks & proxies.

The runahead points, i.e. what are the next available cycle points, are calculated and scheduled. In the scheduling, tasks are queued for execution.

Most of the interesting action takes place when process_task_pool() is called, and in the submit_task_jobs(). The latter is a method from TaskJobManager, and it is here where - in this case - my shell command is executed through a temporary shell script file.

The graph was created from the initial execution of a suite that was starting from scratch (they can also be reinitialized). If there are multiple tasks waiting, or if a suite was restarted, the diagram would look considerably different.

You can download the source file for the diagram used in this post, and edit it with draw.io.