Transparent Proxy for Broadway ( Gtk3 / HTML5 backend )

I’m a big fan of Gtk+. One of the interesting additions since version 3 is the ‘broadway’ backend. Broadway is a rendering backend that lets you distribute your Gtk+ app via a browser. It does this by serving up a simple page with an HTML5 canvas and some Javascript, and then pushes GUI updates over web sockets. It’s an intriguing approach, and very useful if you need to distribute to a bunch of people without installing dependencies ( which is still painful on Windows ).

There are some shortfalls of the broadway component. It only supports a single client per broadway process. If a 2nd client connects to the port broadway is listening on, the 1st client gets disconnected. There is only very simple authentication offered, which is set on a per-user ( ie Linux user ) basis. These 2 issues make it difficult to use broadway in a production environment.

While I hope Gtk+ people will flesh out broadway functionality further, I’ve managed to bring up a proof-of-concept transparent proxy for broadway that goes a long way to addressing the above issues. Here’s how I currently have things working:

  • A simple authentication page at /auth does application logins. I’ve implemented this in PHP, just to remind myself how much I hate it 🙂 Successful logins result in:
    • A new broadwayd process being launched on an available port
    • A new instance of my Gtk+ app being launched, and directed to use the new broadway process we just launched
    • A new random string ( $random_string ) being generated ( eg 20 characters )
    • A cookie being set, auth_key=$random_string
    • A database entry being created that maps the auth_key values to broadway ports – this is timestamped and gets culled regularly
    • Finally, the web client is redirected to /app
  • An nginx reverse proxy that adds https termination, and forwards requests to /app to the port that the transparent proxy is listening on ( I use 10666 )

Next … the proxy code:

This code is originally by Peteris Krumins ( details in the code ). I’ve modified it slightly, adding the concept of a ‘ConnectionFuture’, which doesn’t need to know ‘in advance’ where a connection is being proxied to.

All the magic happens in the syswrite() method. I pass the incoming buffer to HTTP::Request, so I can parse out the cookie ( if one exists ). You’ll see in the code I then look for a cookie named ‘port’:

if ( $cookies =~ /port=(\w*)/ ) {
    $port = $1;
} elsif ( $cookies = 'cookie1=test' ) {
    $port = 10000;

This is obviously a simplification that is looking for the port number directly in the cookie. There is also a check for a cookie named ‘cookie1’, which is hard-coded to port 10000. I’ve done this in the example code so you can quickly verify that the general approach will work for you. You should remove this whole block, and replace with some logic that looks for the auth_key cookie, checks in the database that it exists, fetches the correct port, and sets $port accordingly. As this will depend a lot on your particular environment, I’ve left it as an exercise for the viewer 🙂 You should also add some cleanup code in disconnect() to kill the app process, broadway process, and delete the database entry for the auth_key and port.

There are most likely bugs ( eg we should explicitly disconnect clients that don’t have our cookie, or redirect them to /auth ).

I wondered what performance would be like, with the proxying code having to inspect each incoming request to determine where to redirect it. It’s actually extremely efficient. As luck would have it, there are only a couple of ‘requests’ for the entire duration of a Gtk+ / broadway application. The 1st couple of requests fetch the html and javascript. After that, as the connection is ‘upgraded’ to websockets, all subsequent traffic exists as a part of the same ‘request’, so interacting with the application once it’s started performs as well as a direct broadway connection ( ie no proxy ). There is very low CPU usage for me.

Hopefully this is useful to someone who likes Gtk+ and is looking for a secure way of deploying apps via broadway. Hopefully it also inspires someone more handy in C that I am to implement something like the above in the broadway component itself.

Enlightenment animated background from animated gifs

I’ve been using different versions of Enlightenment since I started using Linux, about 17 years ago. It’s by far my favourite desktop environment, always pushing the envelope in UI design and visual bling. Some of the more interesting features for an end user is the complex interactive UI components that can be easily assembled. These are defined in ‘edc’ files, which have named components, mouse events, and ‘programs’ for animation and other tasks, and probably lots more – I’m certainly no expert, but I dabble.

I’ve designed a few animated backgrounds over the years ( which I really should add to the downloads section here ). Recently I bumped into some amazing animated gifs over at Extreme Instability.

( I’ve had to remove a direct link to an image on the above site – apparently that’s not cool with them? )

I immediately wanted them as animated backgrounds. So I downloaded a couple of gifs, and loaded them up as backgrounds, but Enlightenment only loaded 1 frame – I guess it doesn’t natively support animated gifs. So I set off figuring out how to split the gif into png frames, and then went back to my older animated backgrounds to figure out how I’ve built animated backgrounds before. Faced with so many of these beautiful storm animations to choose from, I then thought what was really needed was a script to convert animated gifs into Enlightenment edj files. So here it is:

And here’s the result:

Here’s what the script does:

  • uses imagemagick’s “convert” utility to split the animated gif into a series of png frames
  • assembles an edc file with the relevent png frames inserted into it
  • calls edje_cc to compile the edj object
  • copies the edj object into your ~/.e/e/backgrounds folder

My bash scripting is kinda basic. I use Perl for everything mildly more complex, but it seemed overkill for this.

Enjoy 🙂

New Apple virus / botnet

I host a couple of websites on my home server, this one included obviously, and others for community organisations. They don’t need a lot of maintenance. I’ve written some perl scripts to monitor apache logs for obvious signs of comment-spam, and this keeps manual work to a minimum. So I was surprised to see this morning a large spike in traffic that happened on the 9th. Check it out:


Now this site hasn’t posted any new content in quite a while. Nor is there any particular page the traffic was going to. It turns out it’s all comment-spam related ( as usual ), and targets all pages equally. Something did stand out though:


That’s a lot of iOS & Mac OS users! In fact, if you subtract these 2 groups from the total usage stats for the day, it goes back down to normal. It sure smells like there’s a new Apple virus / botnet. If you get a virus on your desktop / laptop, it’s not so bad. But a virus on your mobile device is pretty harsh, because it can chew up a lot of your limited bandwith. I’d hate to be an Apple user at the moment. Acually, I’d hate to be an Apple user any time, but especially at the moment.

Hey I wonder if this is the new $10k Apple watch?

Posted in IT.

JewelKit-1.0 released

I’m happy to announce another initial release: JewelKit-1.0 – a Windows distribution of Perl, Gtk3, and bindings ( which are the “glue” between the Perl language and the Gtk3 toolkit ). Gtk3 developers rejoice! Details of the project are in the ‘permanent’ WordPress page ( as opposed to these ‘posts’ ).

Powercom-Utils-1.1 released

I’ve just released version 1.1 of my powercom-utils. This release mainly brings some greatly improved graphing code ( I don’t use any graphing libraries – I use Cairo directly ).


Full changelog:

  • New configs for the graph’s min and max hour.  The previous method of calculating the min & max per day  meant that the start & end of the graph represented a different  time for each day’s readings, making it hard to compare across days
  • Changed calculation of X position in graph to use a ‘minutes past earliest’ concept – the earliest being a config option
  • Render graphs with a Cairo gradient – looks much nicer
  • Added vertical and horizontal lines from each X & Y axis label
  • Fixed position of X & Y axis labels
  • Summary and details datasheets are now next to each other – they don’t take up much horizontal space. Added a vertical pane to hold these 2 datasheets
  • Added a horizontal pane to hold the graphs and datasheets
  • Started decimal place formatting in datasheets

You can download the latest version from the links down the right-hand side of the page.

Gtk3::Ex::DBI-3.1 released

Hot on the tails of the tech-preview of the stand-alone Datasheet library released earlier, I’m proud to now release the next major incarnation of my cross-platform RAD libraries for database development, Gtk3::Ex::DBI.

This release is a culmination of many years of hacking and contains over 700 individual commits. It drives the GUIs of a number of production systems currently in use. Major features in this release:

  • Ported from Gtk2 to Gtk3
  • Added support for Netezza
  • Added preliminary support for Teradata
  • Added proper support for non-autoincrementing primary keys, and non-numeric primary keys
  • Added more hooks for events ( eg on_delete etc )
  • Added support for autoconstruction of recordset tool buttons, eg Add, Delete, etc, and record navigator spin buttons
  • Merged Form and Datasheet functionality into a single package
  • Many, many bug fixes and optimisations

This release has been extensively tested using Gtk3’s new Broadway backend, which renders the GUI in HTML5. This means deployment to Windows desktops can be simplified considerably; no installation of software is necessary, and users can just point their browsers to a port that broadway ( and the app ) is running on.

I’m working on a Broadway session manager, which I’ll be releasing as soon as it’s ready. This utility will be able to launch new instances of broadway and Gtk3 apps when a web user successfully authenticates.

I’ll be posting HOWTOs on using Gtk3::Ex::DBI in the coming weeks / months – time permitting.

You can download the latest version of Gtk3::Ex::DBI using the links on the right, or from CPAN.

Tutorial: Netezza Emulator on Linux in KVM

What is the Netezza Software Emulator?

Netezza is a data warehouse appliance from IBM. It’s a multi-million dollar investment, so for developers and proof-of-concept projects, an emulator is provided, which consists of 2 VMWare appliances. One is the ‘Host’, which is the front-end that applications talk to via JDBC / ODBC. The other is a ‘SPU’, which an emulated ‘snippet processing unit’. In a real Netezza appliance, these SPUs consist of a combination of storage and custom hardware ( FGPAs ). This hardware augments the processing that a CPU would perform in a traditional database server.

Why Convert this emulator to KVM / QEMU?

The VMWare offering is not a good option for my people:

  • It’s built using a very old version of VMWare which doesn’t run on modern OSs
  • It’s packaged as a Windows-only offering, making it of little use to serious developers
  • It uses VMWare’s VIX, and API for controlling VMWare hosts, which seems non-funtional on 64bit Linux hosts

Besides this, VMWare is not an option for many because of licensing restrictions and lagging OS support. VMWare ( and VirtualBox for that matter ) drivers are typically only available for select versions of Linux kernels, and usually versions I stopped using years ago. So I set out to get the Netezza Emulator working in KVM / QEMU, a completely open-source virtualisation solution. This is important to people who take software licensing seriously. While home users can click the “sure, I’m just using this for evaluation purposes” button and forget about it, this approach is a little more precarious for developers who are clearly using the technology as an integral part of their daily work.

It took many days and much swearing, but it’s now working, and even better, rock solid. Here’s how I did it, and what I learned along the way.

How to convert the Netezza Software Emulator

Download Netezza Emulator and Preparation

Download the Netezza emulator from IBM. I can’t get a ‘clean’ link ( it’s a mile long ), but search for ‘netezza emulator download’ and it will be the 1st link ( titled ‘Requirements for the IBM Netezza Software Emulator’ ). You will have to be logged in with an IBM account to download it, from memory.

Extract the components of the download. If you can figure out how to do this without using Windows, go ahead. Otherwise you may need to install the emulator into a Windows box first. Either way, you’re after .vmdk files only. Get all of these. For the host, there will be a LOT. For the SPU, there are 5.

On your Linux box, create a folder called HOST and a folder called SPU. Put the .vmdk files in the relevent folder.

For the next step, you’ll have to have some packages installed. You can probably install the packages ‘qemu’ and ‘virt-manager’ and you’ll get all the rest of the dependencies.

Netezza Disk Image Conversion

Now we’ll convert the VMWare .vmdk disk images to the QCOW2 format. ‘cd’ into the HOST folder and run:

qemu-img convert -O qcow2 *.vmdk HOST.img

This will create a single image from the 91 or so .vmdk files. Next do the SPU images like this:

for I in *.vmdk
echo “converting $I …”
    qemu-img convert -O qcow2 $I $I.qcow2


This will create 1 qcow2 image per .vmdk image. Now move all the .img files into 1 folder somewhere. We don’t need the HOST / SPU folders any more.

KVM Pre-Configuration

Now start up ‘virt-manager’. This is a GUI for configuring and managing various VMs. It seems to require ‘polkit’. If you’re running Gnome ( or probably KDE ), you’ll have a polkit process or whatever running already. If you’re running something else ( I’m running Enlightenment-0.19 ), you’ll have to start it manually like this:


Once you’re in virt-manager, click ‘Edit ==> Connection Details’. The 1st thing we’ll do here is make the disk images we created visible. Click the ‘storage’ tab. Click the ‘Add’ button in the bottom-left corner of the window. Navigate to where you put the .img files. This should add the folder you’ve selected to the ‘storage pool’. Alternatively, you could have located the default storage pool, and moved your .img files into it. Either way is fine.

Next, click the ‘virtual networks’ tab. Here we’re going to define the network that the Netezza Host and SPU virtual machines use to communicate over. Interestingly, this is also the network that your KVM host ( ie your linux box ) is going to have to use to communicate with the Netezza Host – more on this later. Click the ‘Add’ button, in the bottom-left corner. Name the network ‘nznet’. Complete the wizard, using the network and disabling DHCP. The Netezza Host will run it’s own DHCP server, and things will not work well at all if you have 2 DHCP servers on this network.

Virtual Machine Setup

Now we’re finally ready to create the 2 Netezza VMs in KVM. Close the current ‘connections’ window, which should leave the original virt-manager window. Click the ‘Create New Virtual Machine’ button, in the top-left corner. Type HOST for the name, and select ‘Import existing disk image’.

Netezza HOST creation

Click ‘Browse’ to locate the Netezza Host disk image we converted ( and think yourself lucky we concatinated all the 91 images into 1 ). Select OS Type: Linux and Version: Red Hat Enterprise Linux 6. Select your chosen memory & CPU count. I’m not sure how many CPUs these Netezza VMs require, but I gave mine 2 each. I recommend at least 2GB for both Netezza Host and SPU VMs. Finally, click the ‘customize configuration before install’ checkbox, and then click ‘Finish’.

You’ll be taken to a management screen for the VM. Click on SCSI Disk 1. Under ‘Advanced options’, select Disk Bus: Virtio. This will give you maximum disk performance. Luckily the Redhat base OS supports this device.

Netezza HOST virtual disk setup

Next we’ll configure our 2 network devices. Click the 1st ‘NIC’. Select:

Source Device: Host Device eth0 : macvtap
Device model: Virtio
Source mode: VEPA

Netezza HOST networking setup

Note: if you’re using a wireless network device instead of wired, your device name might be something like wlan0 instead. However, also note that this ‘macvtap’ is a kind of bridged networking, and some wireless devices don’t supported bridged networking ( and virt-manager mightn’t even allow you to select your wireless ). If that’s the case, you’re out of luck. This will mean that other devices on your LAN won’t be able to connect to your Netezza box. Just select your eth0.

Now we’re going to add another network device. Click the ‘Add Hardware’ button in the bottom-left, select ‘Network’, and then select:

Host Device: Virtual Network ‘nznet’ : Isolated network, internal and host routing only
Device model: Virtio

That’s the end of the Netezza Host configuration 🙂 You can now hit the ‘Begin Installation’ button in the top-left, and then power on the VM. If all is well, your Host should boot. Note that there seems to be a strange graphics corruption bug when switching between modes on bootup. Just close the Host window completely, then double-click on the Host VM again in virt-manager to re-open it. Now log into the Host ( root / netezza ). Your eth0 device will not have come up. Type:

rm /etc/udev/rules.d/70-persistent-net.rules

This file provides a memory of eth devices to MAC addresses, and your new VM’s ethernet’s MAC addresses will not be the same as in the original VMWare images ( unless you made a point of copying them in ). After removing this file, reboot and log in again. You should now have an eth0 and eth1. eth0 should have an IP from your DHCP server ( assuming you have a DHCP server on your LAN ). eth1 should be If you want to have a fixed IP for eth0, copy the MAC address and paste it into your DHCP’s configuration for fixed IP addresses.

At this point, you should be able to ssh into eth1 ( ) from your KVM host ( ie your linux box ), and other computers on the network should be able to ssh into eth0 ( dhcp-provided address ), but your KVM host will not be able to reach the NZ Host’s eth0.

Now for the Netezza SPU. Repeat the same process as before, calling this VM ‘SPU’. There are a few differences in the SPU configuration:

  • The SPU only has 1 ethernet device, and it’s on the nznet VPN. Choose device e1000 – the SPU doesn’t have Virtio drivers
  • When configuring the disks, create them all as SATA disks
  • Under ‘Boot Options’, you should have only ‘Network (PXE)’ selected.

Now, with your Host VM fully booted, power up the SPU VM. It should get an IP address from the Host’s DHCP server, and then should boot from the network. If at this point, the network boot is slow, or appears to get stuck, you have probably left on the DHCP server for the ‘nznet’ VPN.

Hopefully, your SPU will boot. However there are still issues to overcome. Now the SPU will try to register against the Host, but fail. This is because the topology of the disks has changed from when everything was hosted in VMWare. You can flush this issue out by doing a ‘reinitialisation’ on the Host. Log into the Host as root. Then type:

su – nz
nzinitsystem -reinit
nzstart -newSystem

Your SPU should reboot. If you’re lucky, when it comes back up, it will register against the Host, and you’ll be ready to go! If it doesn’t work, repeat these last 2 steps until it does. It seems to be quite flakey, unfortunately. I’m not sure what the problem is. Once it’s working, it appears to stay working, but it doesn’t like that first initialisation.

When you it working, shut down both. On the Host, log in as root, and do:

su – nz

I then always log into the SPU (root / fruitloop ) and ‘halt’ it. It won’t power down the VM; just do it manually.

Now clone both VMs, in case something goes wrong. In virt-manager, right-click each VM, and select ‘clone’.

Finally, you’re truly ready to get some work done. Enjoy 🙂

Networking ( aka I hate bridges )

For this tutorial, I used KVM’s macvtap networking to provide a network device on the LAN. Before going down this path, I tried to use bridged networking ( which I’ve successfully used before ), however there were just too many issues with it this time around. Firstly, my distribution ( Gentoo ) doesn’t have the same level of bridged networking support using systemd as using openrc. Secondly, bridged networking causes NetworkManager to freak out, and unfortunately some services depend on NetworkManager … which means they won’t come up properly once you start using bridged networking. Using KVM’s macvtap doesn’t have this problem, and doesn’t require hacking your network configuration or init scripts. It does have the bizzare issue that the KVM host can’t see the guest’s network device on the other side of the ‘bridge’ … however other devices on the network can. It’s a reasonable compromise.

Coming soon …

I’d like to get all this running in Xen next. I’ll try to document the process for other VM technologies as I get time.

Need Netezza Experts?

I’m currently working for Smart Associates. We do Netezza consulting, design, migrations, virtual DBA, and a lot more. Contact us via our corporate site for a talk about what we can do for you.

Please let me know if you have any suggestions. I did most of this post from memory – it’s been a while since I set it all up.