MQTT & Mosquitto Server Setup Notes

Overview
I've been playing around with the MQTT protocol. As a learning exercise I wrote a client for MQTT: climqtt [1]. A lot has already been written about MQTT so below are just some of my notes and links to stuff that helped me.


MQTT
MQTT is a simple publish/subscribe protocol. A client connects to a MQTT server and then sends a 'connect' message and the server responds with a 'connection acknowledgement'. At that point the client may subscribe to one or more topics or publish to topics. When subscribing to a topic a client may choose to specify wildcards. The topic matching mechanism is handled on the server-side.

Another interesting aspect of MQTT is the last will & testament (LWT) where if a client disconnects it may have preconfigured a LWT to emit a message on a predetermined topic. This is useful to indicate that client has gone offline.

Default MQTT Unsecure Port: 1883
Default MQTT Secure Port: 8883


Mosquitto
Mosquitto[2] is a library/server/client combo that handles all things MQTT. It seems to be pretty mature.

Install Mosquitto

• Installing Mosquitto on a Raspberry Pi [4]
• Useful information on generating certificates & setup can be found here [3]
• Mosquitto provides some sparse docs on certificates here [5]

Mosquitto Server .conf file

listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/grant.crt
keyfile /etc/mosquitto/certs/grant.key
require_certificate true

Mosquitto Pub
Use mosquitto to publish to the "/test" topic and also set a LWT:

mosquitto_pub -t "/test" -l -r --will-topic "/test" --will-retain --will-payload "goodbye..." -h 127.0.0.1 -q 2 -p 8883 --cafile ca.crt --cert grantscomp.crt --key grantscomp.key

Mosquitto Sub
Use mosquitto to subscribe to all topics (i.e. the "#" matches all topics):

mosquitto_sub -t "#" -v -v -h 127.0.0.1 -q 2 -p 8883 --cafile ca.crt --cert grantscomp.crt --key grantscomp.key

Links:

1. https://bitbucket.org/grantj/climqtt/overview
2. http://mosquitto.org
3. http://rockingdlabs.dunmire.org/exercises-experiments/ssl-client-certs-to-secure-mqtt
4. http://jpmens.net/2013/09/01/installing-mosquitto-on-a-raspberry-pi/
5. http://mosquitto.org/man/mosquitto-tls-7.html

Foot-switch MIDI Controller

Recently I've been playing around more with MIDI control of audio programs like Ableton Live and POD Farm effects program on OS X. Creating my own foot-switch controller seemed like an easy weekend project.

Ingredients:

1. A footswitch ( https://www.sparkfun.com/products/11192 )
2. Someway to get input into my computer. Teensy 3.0 ( http://pjrc.com/store/teensy3.html )
3. Unnecessary number of LEDs ( https://www.sparkfun.com/products/11492 and all the bits that go with that: breakout board, LED rotary encoder )
   
   
   

Instructions:

• Foot-switch goes into the teensy: simple pull-up resistor
• Wire all up the rotary encoder breakout board pins to the teensy
• Write some arduino code to read from/write to the serial port
• Write an OS X program to read from/write to the teensy board via the serial port

Source Excerpts:

On the teensy I attach an interrupt on the pin the footswitch is attached to:

void setup() {
  attachInterrupt(foot1Pin, footSwInterrupt,CHANGE );
}

static int foot1PinDown = 0;
void footSwInterrupt() {
  foot1PinDown = (digitalRead(foot1Pin) == LOW);
}

void loop() {
  static int last_state = 0;
  
  if( last_state != foot1PinDown ) {
      uint8_t buf[5];
      buf[0] = 0x01;  // COMMAND
      buf[1] = (foot1PinDown) ? 0x1 : 0x0;
      buf[2] = 0;
      buf[3] = 0;
      buf[4] = 0;
      Serial.write(buf, 5);
      
      last_state = foot1PinDown;
    }
}

I don't use the rotary encoder for anything but the "Encoder" library provided by pjrc works well with the sparkfun breakout board/rotary encoder.

The ring-o-LEDs is just two 8-bit shifts out:

void updateRing( uint16_t v ) {
  digitalWrite(ringLatch, LOW);  

  shiftOut(ringDat, ringClk, MSBFIRST, v>>8);
  shiftOut(ringDat, ringClk, MSBFIRST, v&0xFF);

  digitalWrite(ringLatch, HIGH);
}

The most interesting part for me was figuring out how OS X's midi library works. This turns out to be pretty simple after looking at maxbok's CoreMIDITemplate.

On app startup register as a MIDI source. I.e. other MIDI programs that "read" MIDI messages will show "Footswitch MIDI Source" as one of the control surfaces:

MIDIClientCreate((CFStringRef)@"Footswitch MIDI Client", NULL, NULL, &_client);

MIDISourceCreate(_client,(CFStringRef)@"Footswitch MIDI Source", &_endpointRef );

MIDIOutputPortCreate(_client, (CFStringRef)@"Footswitch Output Port", &_outputPort);

When you want to send a note message:

- (void)sendNote:(uint)note on:(BOOL)on {

    // http://www.onicos.com/staff/iz/formats/midi-event.html

    const UInt8 data[]  = { on ? 0x90 : 0x80, note, 127 };

    ByteCount size = sizeof(data);

    

    Byte packetBuffer[sizeof(MIDIPacketList)];

    MIDIPacketList *packetList = (MIDIPacketList *)packetBuffer;

    

    MIDIPacketListAdd(packetList,

                      sizeof(packetBuffer),

                      MIDIPacketListInit(packetList),

                      AudioGetCurrentHostTime(),

                      size,

                      data);

MIDIReceived( _endpointRef, packetList );

}

The rest of the code is pretty teensy specific for opening it's USB serial port and reading from it in an asynchronous mode:

- (void)startSerialReader {

struct termios settings;

__block int fsw1_state = 0;

int port = open("/dev/cu.usbmodem12341", O_RDWR);

if (port < 0) {

fprintf(stderr, "Unable to open %s\n", "/dev/cu.usbmodem12341");

return;

}

__weak typeof(self) weak_self = self;

// Configure the port

tcgetattr(port, &settings);
cfmakeraw(&settings);
tcsetattr(port, TCSANOW, &settings);

_serialHandle = [[NSFileHandle alloc] initWithFileDescriptor:port closeOnDealloc:TRUE];

__block NSMutableData *comm_buf = [[NSMutableData alloc] init];

_serialHandle.readabilityHandler = ^(NSFileHandle *fh) {

// DO INTERESTING THINGS WITH SERIAL DATA FROM TEENSY HERE!

};

}

Results:

Here is a short video of the rotary encoder's RGB LED's cycling through colors while I push the footswitch down.

The hardware overall works great. You kinda get what-you-expect from the $5 foot-switch from sparkfun. It works but it has no tactile feel and you need to push it down quite hard to ensure it's actually switched. I'd like to experiment with some sort of optical switch arrangement instead of this cheap mechanical one.

InfiniteRay: A View-based platform

Introducing a side-project I wrote over the holidays. InfiniteRay is a cross-platform graphics platform which uses Lua scripting to manipulate a hierarchy of views. The goal of this project is to make it easy to write complex graphical applications. It turned out to be a pretty cool way to do some rapid prototypes.

You can check out the source here (BSD Licensed).
There are no binaries yet available. However it does build under win32 using mingw and OS X using xcode assuming you have all the prerequisite libraries (see below). Note: The Lua bindings are automatically created by gen.py in the /gen directory.

It's also still alpha-quality so if you're looking for a more mature platform check out the Love 2D Game Engine. InfiniteRay differs from the Love engine in a number of different ways but primarily InfiniteRay is C-based and Love is C++ based.


(Short Video Demo)


(The loader screen)

Features:
- Lua scripting language
- Structured view system to easily create complex compositing heirarchies
- Asynchronous resource loading from files or zip archives
- Drawing support utilizing the cairo graphics library
(http://www.cairographics.org/)
- Font rendering using the freetype font engine
- Asynchronous TCP/IP network support

Library Dependencies:
- FreeType 2.3.11
- Cairo Graphics Library (libcairo,libpixman,libpng)
- zlib


(The asteroid example)

cocoaevo

After seeing Mona and Lena and the inspiration behind that, Roger Alsing's blog post on genetic programming, I found myself writing my own slightly different version. Written in Cocoa it only uses two command verbs: brush and triangle. The generation algorithm simply pairs brush and triangle commands together whenever it places a new triangle. I also cheat by letting it sample from the source image. It still plays around with the color of the triangles after the initial color sampling.

Source:
CocoaEvo 106 KB
Disclaimer: hacked together in one night. The actual mutator code runs two worker threads, I ripped that code from another one of my projects, so the CodeMutator module is kind of a mess -- and that's really where the interesting stuff happens.

Some results:


Elapsed time for Dodo (illustration by Roland Savery) was ~19 minutes.



Elapsed time for Circle (illustration by Me;) was ~9 minutes.


Fun. Fun.

Sandboxing Applications in OS X

Last month I spent most of my time studying the OS X Mach-O file format and the dyld. After playing around with Sandboxie on windows I thought it'd be fun to see how it could potentially be implemented on Mac OS X. The following is basically a dump of what I learned as I ventured into the world of the dyld on Mac OS X.

First, I should define what sandboxing means in this context. Sandboxing an application involves wrapping applications within a lightweight container which restricts what an application is able to do and in some cases changes the behavior of system functions completely. The goal of this type of sandboxing is to prevent an application from arbitrarily writing into the filesystem. Reads come from the source filesystem but writes are redirected into the sandbox filesystem. In this way sandboxed applications cannot change system files.

Is there a need for sandboxing on OS X?
I'm not a security expert so I cannot comment on whether or not sandboxing in this form is effective, beneficial or necessary on OS X. From my average developer's standpoint there are benefits such as:

• Security: Yes, there may not be (m)any malicious apps targeted towards OS X but this could change. Assuming the sandboxing application is lightweight enough to run transparently to the user, there would be little reason to not run inside a sandbox.
  


• Testing: Run app that is to be tested inside a sandbox. After running the app the sandbox can be deleted when done. No extra cleanup required, system never really was effected by the app.
  


• Auditing: Track what application adds/removes/modifies files. To be fair there are auditing abilities built into OS X already, see audit(2). Accessed files can also be seen within the Activity Monitor application under the Inspect, 'Open Files and Ports.'
  

Ways to Sandbox Applications on OS X

• Kernel Extension: From just a glance at sandboxie it seems to use a kernel-mode driver to get most of it's work done. Not having ever done kernel development before, I took a different approach. It could be that a kernel extension would be the cleanest way to achieve app sandboxing however the kext may then become a target itself if it were poorly implemented.
  


• Some Sort of Emulation: This approach would involve running the process under an emulation layer like Valgrind. Patches do exist to get valgrind running under OS X and would be worth investigating feasibility. One downside would be the extreme performance penalty.
  


• Function Interposing via Dynamic Library Insertion: Probably the most viable user-level way to achieve sandboxing. Below I have described how it would work.
  


• Function interposing via Executable Modification: The details would be pretty nasty but you could modify the symbol table within the mach-o executable to point to other functions in other libraries. The problem is that all the other loaded dynamic libraries would need to be modified as well and that's where it'd be really messy.
  


• Other ways? Certainly I'm probably missing other (easier) methods for doing this. If so let me know!
  

Function Interposing via Dynamic Library Insertion
(Before continuing, note that OS X occasionally uses two-level namespaces, which means that linux method of overriding functions may not necessarily work. Interposing does provide a nice way to address this issue.)

This approach involves directing the dyld (the dynamic link editor) to load a specified library at the process launch time. This is simply done by setting the 'DYLD_INSERT_LIBRARIES' environment variable to the target library before execution of the an application. The inserted library to be loaded has a special '__interpose' segment in the __DATA section which is interpreted by the dyld. Amit Singh's fantastic book has a chapter on interposing. Essentially it is a table of new and old function pointer pairs. Whenever the dyld needs to locate a symbol it will first look at the interposing table. This means that any dynamically linked symbol may be overridden by the library you inserted. So now you can redirect open/read/write system calls from the dynamic library.

The caveat of this method being that you can only call the real functions from within the library. This means that everything you need to do has to happen within the library.

So now we can insert our own versions of functions to be used by arbitrary applications. The question now is what functions should be interposed?
Nearly all the system calls are located within libSystem on OS X but there are also multiple versions of most system calls. From the average developer's standpoint this is handled transparently. In this case, though, it causes issues if the library is to robustly handle any OS X executable. For example, there are actually three different 'open' function calls:

open
open$UNIX2003
open$NOCANCEL$UNIX2003

Since we're not necessarily sure which version the executable is going to use each of the three versions of the functions need to interposed. See the multiple symbol variants link below for further explanation.

Of the system calls we are generally interested in the ones that deal with file and directory manipulation directly:

open
close
read
write
fstat
link
unlink
mkdir
mknod
mmap
munmap
opendir
...etc...

The list ends up being very long and there are a huge number of special cases. Such as how mmap function is handled, or how the /dev/* special nodes are handled.

Current Development State
So that's where I'm at now. Early on, I was able to run Safari inside of another shadow directory simply by hijacking the open call and opening a shadow file (first copying the original to the shadow directory) and returning the file descriptor to the shadow file. This worked surprisingly well but I wanted to get more ambitious and run any application from a single shadow volume file. To do this I've chosen to use sqlfs with sqlite. The result should be an injectable library which runs in user-mode without need for special privileges which writes into one file which can easily be discarded. Well, that's the idea anyway, it's quite a bit of work to be done still and I'm not sure if there is any demand for such a product.



Miscellaneous and References:

dyld was re-written in C++ from a C/asm version for rosetta 10.4/10.5? However, interposing is not included in the open source release?

Guard Malloc(libgmalloc) uses only DYLD_INSERT_LIBRARIES envar to override malloc and it does so by using an interpose table.

Two Level Namespaces

Executing Mach-O Files

dyld Library Reference

Mach-O Reference

Multiple Symbol Variants

OS X Kernel Programming

From the cold snowy hill

A post before the end of 2008

Preface:
My mote-m experiment has effectively ended. Although it's still available in the store it has been superseded by team efforts from real artists looking to (and succeeding) at making it big on the app store. I wrote a postmortem about my experience about two months ago. I never published it because the iPhone App Store scene continues to change at such a rapid pace that anything written is immediately out-dated and irrelevant. In spite of that fact, I will share a few parts.

Introduction:
Let me say that these are my personal experiences and observations as an independent/first-time iTunes App Store game developer. I built Mote-M from the ground-up. I created the web site, wrote the code, created the art and answered everyone's kind emails. I consider Mote-M to be a success in it's original goals and a failure to achieve it's full potential. I might be wrong on any conclusions I make and apologise in advance for my poor grammar and inconsistent tense. You have been warned.

Development:
Mote Massacre, Mote-M for short, is a Tower Defense game in the style of the Desktop Tower Defense games. It started it's life in December of 2007 as a flash multiplayer game. When the iPhone SDK was announced the decision to port the game engine was made. After being approved by Apple in late-july (along with the rest of the non-special developers), I ported over the basic game and decided to launch the app as a fun toy-app for $0.99 in mid-August. The original goal was to build a fun game which would also pay for the developer registration. I really only expected maybe 20 people to buy the game. Since August I've watched the app store grow and evolve.

A Learning Experience:
Revelations I had as a complete n00b in the iTunes app store game business.

Quality of apps: The quality of apps and the relationship with an app's price and sales volume has been discussed elsewhere. It's still important to mention that with very low app prices a developer has to make money somewhere else or succeed in selling a very large volume.

Top sales lists DO matter: there are a set of passionate gamers who read the blogs and the forums and participate in the app store community but I'm willing to bet that the majority buy off the front page and the top paid apps lists. This is not surprising -- I probably wouldn't visit the forums and blogs if I weren't developing for the platform.

Rampant Piracy of Apps: The day you release your app for purchase is the same day anyone with a jailbroken phone can download your app for free. It's a fact of software -- you can't stop it, so don't try. The amount of time I spent trying to figure out the convoluted app DRM scheme could've been spent on developing a better app. To be fair the code signing stuff is pretty much a one time setup but from a goals standpoint it *just* fails.

The app store has huge potential: A developer can make a lot of money if they launch the app perfectly. Where perfectly in that sentence means: releasing an eye-catching, functional app which encourages impulse buys. Impulse buys fuel the app store right now. This may also be contributing to lower prices.

The Community:
This is one area where I was pleasantly surprised. The community of people who gathered around Mote-M are passionate and highly engaged. The e-mails I've received have been almost universally helpful and encouraging. Any developer not embracing their community is definitely losing valuable feedback.

Future Plans:
As I write this I have xcode open to a new super-secret non-game project for the mac. Once that project matures I'll return to the app store to see what sort of climate there is for indie game developers. Right now there seems to be a new flood of ports from existing games: SimCity (tm), TETRIS(r), Brothers In Arms(r), etc. Without users being able to demo apps they may stick to more well-known and bigger names instead of supporting software devs who create more original apps.



About gxjones.com:
Independent software developer Grant Jones started gxjones.com in late 2006. Grant grinds away creating another independent game with the experience he has gained from Mote Massacre and past applications he has written.

Mote-M Base Source Code

Version v1.2 of Mote-M is complete so I took some time to bundle up the base 'framework' I used to create Mote-M. It's pretty unexciting but it takes care of the details which are necessary to setup an OpenGL ES rendering context on the iPhone.

Read further details here or just download it here (released under the BSD license).