Blog Taxonomy

Posts tagged with 'pbs'

Running Cylc tasks on PBS Torque with Docker

kinow @ Dec 22, 2018 11:32:28

A few days ago I saw a post at the Cylc Google Group, about file permissions for files generated by Cylc. The post was related to content created by Cylc, but in an environment with PBS.

For context, Cylc is an Open Source meta-scheduler, written in Python, that allows you to define cycle points with dependencies. These cycle points can be simple incremental integer numbers, or ISO8601 periods or points (e.g. run every 5 minutes, from 10 days ago until the next year). Cylc takes care to create an execution schedule for you, and delegate that to a system that runs your workflow. I work full time on this amazing Open Source tool!

Such system could be the local computer in background, batch systems such as at, or PBS. PBS was created for NASA, to manage executing jobs taking into consideration cluster resources, and also using queues, priorities, and other features useful for HPC programming. Later PBS was acquired by Altair, an Open Source version OpenPBS was created, and later abandoned. And there is another fork called PBS Torque. I first encountered PBS at the São Paulo University, in Brazil, where they had a PBS Torque cluster.

Running PBS Torque with Docker

Even though I have access to an environment with Cylc and with PBS, I decided to give it a try and see how hard it would be to reproduce it with Docker. One thing that I like about this approach is the possibility to share the work with others online. I believe it improves communication, agility, and can be useful for posterity.

( Read more ... )