Blog

Posts about technology and arts.

Running Cylc tasks on PBS Torque with Docker

A few days ago I saw a post at the Cylc Google Group, about file permissions for files generated by Cylc. The post was related to content created by Cylc, but in an environment with PBS.

For context, Cylc is an Open Source meta-scheduler, written in Python, that allows you to define cycle points with dependencies. These cycle points can be simple incremental integer numbers, or ISO8601 periods or points (e.g. run every 5 minutes, from 10 days ago until the next year). Cylc takes care to create an execution schedule for you, and delegate that to a system that runs your workflow. I work full time on this amazing Open Source tool!

Such system could be the local computer in background, batch systems such as at, or PBS. PBS was created for NASA, to manage executing jobs taking into consideration cluster resources, and also using queues, priorities, and other features useful for HPC programming. Later PBS was acquired by Altair, an Open Source version OpenPBS was created, and later abandoned. And there is another fork called PBS Torque. I first encountered PBS at the São Paulo University, in Brazil, where they had a PBS Torque cluster.

Running PBS Torque with Docker

Even though I have access to an environment with Cylc and with PBS, I decided to give it a try and see how hard it would be to reproduce it with Docker. One thing that I like about this approach is the possibility to share the work with others online. I believe it improves communication, agility, and can be useful for posterity.

What the weather forecast looks like in Sao Paulo, Brazil

I grew up watching the weather forecast in Sao Paulo, Brazil. Where it is called “previsão do tempo”, which is the literal translation in Portuguese.

This post contains some recent screen shots of the weather forecast as it is presented in one of the main TV channels, as well as some screenshots from an online video of other local institutions that either provide the forecast, or help running the NWP models.

A couple of class diagrams of JupyterHub

Started on a new project last Monday. One of the tasks in this project involves a new design for the Web layer. And as the application is quite similar to JupyterHub, we are all learning more about its internal API and general system design.

This post contains only two class diagrams created with PyCharm. One is actually a SQLAlchemy ORM diagram, below.

Cylc Scheduler Internals - Part 3

This is the part 3, in a series of posts about Cylc internals. The part 1 had the beginning of the workflow. part 2 documented from the moment the method configure() is called. This post will continue right after the continue() method returns, going on with the next method: run().

Use of Logging in Java Image Processing libraries

For IMAGING-154 I was trying to think in a solution for the existing Debug class. This class was the issue of discussion during a previous 1.0 release vote thread.

Subscribe