Tag

Posts tagged with ‘python’

Processing Vaisala Radiosonde data with Python, and creating GRUAN-like NetCDF files

kinow @ Jul 12, 2016 21:18:03

One of my last projects involved parsing a few GB’s of data that was in a certain binary format, and convert it into NetCDF files. In this post I will describe what was done, what I learned about radiosonde, GRUAN and other geek stuff. Ready?

Vaisala Radiosonde data

When I was told the data was in some binary format, I thought it would be similar to parsing a flat file. That this would contain a fixed length entry, with the same kind of item repeated multiple times.

Well, not exactly.

The files had been generated by an instrument made by Vaisala, a Finnish company. This instrument is called a radiosonde. It is an instrument about the size of an old mobile phone, that is launched with a balloon into the atmosphere.

I was lucky to be given the chance to release one of these balloons carrying a newer version of this equipment.

Radiosonde balloon launch

The balloon can carry equipments for measuring different things, like air pressure, altitude, temperature, latitude, longitude, relative humidity, among others. Equipments like the radiosonde send the data back to a ground-level station via radio, normally in a short and constant interval.

( Read more … )

Too many SQL variables exception in Graphite with SQLite3

kinow @ Jul 04, 2013 13:54:53

Having run Graphite for a while, today I found a rather annoying issue. We were using events, and everything was working perfectly fine so far. But for the 24 last hours, the graph was blank.

Actually, in the dashboard, the graph was missing, being displayed as a gray box. Enabling the web inspector in Google Chrome I got the graph URL. Opening the link in a new tab gave me the exception message: too many SQL variables (1).

After some research, I found out this was a bug in SQLite. After trying to hack the code, and being concerned about using a patched version of Graphite and having to update it later, I decided to switch database.

But to avoid losing the graphs, users and other settings, including the events, I migrated the SQLite database to a MySQL server. This MySQL server was already installed in the server machine, since this machine hosted a Zabbix server too.

Here are the steps required to migrate your database from SQLite to MySQL (2).

  • Download sqlite3_mysql python script from http://www.redmine.org/boards/2/topics/12793
  • Stop Apache/Nginx
  • mysql -u user -p -e “create database redmine graphite set utf8;”
  • sqlite3 graphite.db .dump | sqlite3-to-mysql.py | mysql -uroot -pyourpass graphite
  • Start Apache again

After tail’ing the Graphite webapp log file,

tail f- storage/log/webapp/error.log

I noticed the Python MySQLdb wasn’t installed.

ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb

My server was an Ubuntu 13.04, so I installed the module simply with the following command.

apt-get install python-mysqldb

Hope that helps!

1 You may have to enable *DEBUG* in your Django settings for seeing the exception in your browser

2 Zabbix needed some minor tweaks in order to use [MariaDB](https://mariadb.org/), but probably you can give it a try too

Graphite: Broken images

kinow @ Apr 17, 2013 11:36:38

This morning I was setting up a Graphite server to collect metrics with statsd, LogStash and jmxtrans. After following the instructions from @jgeurst, I’ve successfully installed Graphite.

I had previously installed another test box, so I decided to take a deeper look at the settings, write a puppet manifest and prepare this new box to become a production server. However, after browsing the webapp, all graphs were broken.

After googling a while, reading forums and bugs, I decided to open the $GRAPHITE_HOME/webapp/graphite/render/views.py, adding log.rendering(…) statements (not the most elegant solution, I know).

By following the program workflow I found out it was entering a block after checking if it should remotely render the image. This feature is turned on/off by REMOTE_RENDERING = True/False, in local_settings.py.

After setting this to False the problem was solved.