A few weeks ago Jasper Knulst, a colleague from Incentro, created a custom mapreduce program for Hadoop called “Hadoop QVX Converter” which produces files with a .qvx format. Qvx is an open format readable by Qlikview. This specific format loads its data a lot faster into Qlikview compared to CSV, Databases or other sources. The only format faster compared to QVX is the QVD format. Also a document specific for Qlikview but this format is not accessible for public use.

This mapreduce program gave me the idea to create a similar plugin for Pentaho Data Integration. So today i present to you: The Pentaho Data Integration QVX Writer plugin. The version available for download is a demo version and is limited to write up to 100.000 rows into a .qvx file.

The plugin uses a library called QVXConverter created by Ralf Becher from TIQ Solutions.

Installation:
Download the plugin
Extract the content into:<pentaho data integration install folder>/plugins/steps/. When extracted there should be a folder inside the ‘steps’ folder called “QVXWriterDemo”.

Start Pentaho Data Integration. Create a new transformation (or use an existing one). There should be a new entry in the ‘output’ category called “QVXWriter Demo Plugin”.

Usage:
Define a location and file name (.qvx) in the “Output File Location” field. Give the file a table name in the second field. Make sure the columns got the correct data types (Integer, Number, Date, String). When you are not sure, use the “Select Values” step in Pentaho to manually add or change the meta values of your columns.

Feel free to contact me if you want more information regarding this subject.

2 Comments

  1. Bram,

    Very interesting. As a regular user of both PDI and QlikView, this is something I will definitely find a use for. Excellent work!

    Kind regards,
    Barry

  2. Great work Bram!

    I know there are many QV users in search of good ETL tools to manage pumping data into QV. If you are interested, I would be happy to chat with you if you’re interested in contributing your work to the Kettle project, or feel free to post links to your plugin on our community page – http://wiki.pentaho.com/display/EAI/List+of+Available+Pentaho+Data+Integration+Plug-Ins

    Best regards,
    Jake

You must be logged in to leave a reply.