Not So Distant Future

Blog Revival through GitHub Actions

2020-05-22T00:00:00-06:00

This blog was first created via GitHub Pages, and worked fine for quite some time. Then Microsoft bought Github, and made private repositories free. I jumped on that opportunity to save a few bucks a month, and switched my plan. I didn’t realize at the time that publishing to GitHub Pages from a private repository was not included. It took me quite a few months to realize I’d broken it.

I didn’t like that state of broken, and was frustruated that a config change did it. I decided to move the blog hosting over to S3, and use the newly released Github Actions to auto publish the site when changes were made. After struggling my way through GitHub Actions in multiple attempts, it’s finally working again!

So here we are. I’m embarrased to have let the blog die as I did, but happy with how it is currently setup.

Nerd License Renewal - Arduino and Alexa Hot Water Monitor

2018-03-04T00:00:00-07:00

A few months ago, I read about a contest that represented a perfect opportunity for a Nerd License Renewal.

Nerd License Renewal: A demonstration of nerd aptitude and interests that make it clear the applicant qualifies as a nerd and may be recognized as such. Renewals should be published on a periodic basis to renew your license.

The contest called for a project that involved any Arduino product, and used an Alexa voice interface. I had a project in the back of my brain for a little while that seemed like a perfect fit. By installing a water flow meter and two temperature sensors on my hot water heater, I could measure hot water use and be able to ask Alexa (through a custom skill) how much hot water was left.

The details and required parts for the project are over at the project page at Hackster.io, but I’ll include some additional background here for those interested.

Flow Meter

I chose a flow meter that could be easily installed in a hot water heater. The fittings match the flexible pipes used on hot water heaters, which makes for an easy install. The sensor has a small fan inside with a small magnet on it. A hall-effect sensor on the outside of the pipe detects the spinning of the internal fan, and sends a pulse down a wire. The flow range matches regular household hot water use. The sensor does have an operating temperature limit of 60 degrees celcius. That is the upper range of hot water heaters, and I mitigated the temperature risk by isntalling the flow sensor on the cold water input rather than the hot water output.

The device is listed as compatible from 4.5v to 18v, but it operated just fine at the 3.3 volts I was using.

Under use, I would occasionally read just a few pulses on the signal line without any real water flowing. I don’t know why the internal fan was spinning, but may have had something to do with minor fluctuations in volume due to temperature changes in the hot water heater or pipes.

Temperature Sensors

Measuring the temperature difference between the input and output temperatures is critical for calculating the required energy (and therefore time) to heat water. This is used to approximate the rate at which water is heated to replace the hot water used.

I used two one wire temperature sensors so that I could daisy chain them on the same wires and only use one data pin. This worked really well, except for attaching them to the pipes. I designed and 3D printed clips to hold them to the pipes. This is my first time designing clips like this, and they didn’t clip on strong enough to hold the sensors in place. I’ll need to iterate on the design to make it clip tight. Iteration is difficult, as I don’t have a 3D printer. I’ve been using the printer at the local library (which is awesome to have access to) but limites my opportunities for interation and design changes.

Wemos D1 Mini

I used a Wemos D1 mini as the microcontroller for the project. This little microcontroller has been a favorite of mine. With a built in wifi chip and enough pins for most small projects, It is a very convenient board to use. It’s cheap price makes it even easier to use on many projects. You can usually find it for $7-$8 on Amazon but if you can plan ahead further, you can usually find it for $2.50 - $4 on AliExpress. The shipping times are longer, but the lower price makes it easy to over-order and keep a few spares on hand for the next project!

There are a number of expansion boards for the Wemos D1 Mini as well, and I like to keep protoboards on hand for easy projects. I often prototype on one board with headers to fit a breadboard, then transfer the components onto the protoboard and combine that with another Wemos D1 Mini without the long headers.

The Contest

The instructions in the contest were slightly vague. It was a requirment to use an Arduino ‘product’, but it was not clear if that meant the Arduino IDE, or if it required an official Arduino board. Though there are official Arduino boards with wifi, their cost is high and the availability is low. There were other projects without official Arduino boards, so hopefull my entry counts.

The contest also required an Alexa Skill, but this too was vague. I first understood it to mean that my project instructions were to include how a user would set up their own skill, and I spent significant time including the instructions on how that would be done. Most of my instructions were related to that process. As it turns out, I think they wanted a prepared Alexa skill that could be used by any buider of the device. I could have done that, but by the time I realized what the instructions might have meant, the project was already submitted. I hope my entry still qualifies.

The Code

On GitHub. Please use and provide feedback.

Future Work

There are a whole host of things that can be improved on my design. Here are a few, for anybody working on a similar project, or future me:

Imperial/Metric option. Right now, it’s measured in Metric, but reported in Imperial.
Better temperature detection algorithms.
Sensor for direct heating mode detection.
Easier calibration for new users.
Easier setup for wifi access point.
Calculation of heating cost, based on heating time and wattage of hot water heater.

How I helped run an Instagram Contest with artoo.js

2017-04-03T00:00:00-06:00

Over the last few weeks, my wife ran a gear giveaway on our CurrentlyWandering Instagram account. I saw a chance to help her out with a little code hacking, and dove in. My code examples here are a bit hacky, but this was only used for a short time and wasn’t worth the time investment.

Backstory

Last year Jess was involved in an Instagram contest, and I wrote some code to tally qualifying users and randomly pick a winner. Between then and now, Instagram made significant changes to their API and the former approach would not have worked on the short time schedule I had to get app approval. The only way to get this done in the timeframe I had was some client-side scraping magic.

Enter Artoo.js

I’ve used Artoo.js for scraping projects before, and I was pleased. It was an easy choice. Using Artoo.js clientside is as easy as using a bookmarklet. After adding that magic to your bookmarks bar, one click will load Artoo.js onto the page and make it available for use. Loading the Chrome Developer Tools (F12) will show you that it is loaded and give you a place to enter your custom commands.

Loading the Comments

By default, Instagram doesn’t show you all the comments when you load an Instagram post. They show you a selection of comments, and then load an additional group of comments every time you click the ‘load more comments’ link. What I needed to do was keep clicking that link until it disappeared, and then all the comments would be loaded into the page.

Luckily, Artoo.js has an autoExpand function built just for this purpose. First, the code I used to load comments:

//expand comments
artoo.autoExpand({
  expand: function($) {
    $('ul._mo9iw>li:eq(1)>button').simulate('click');
  },
  canExpand: 'ul._mo9iw>li:eq(1)>button',
  isExpanding: function($) {
    return $('ul._mo9iw>li:eq(1)>div._jf5s3').is(':visible');
  },
  throttle: 5000,
  done: function() {
    console.log('Done expanding every comment!');
  }
});

I should mention that the odd css class names you see in my code (_mo9iw, _jf5s3) are the result of Instagram using a css compiler. Those classes are very likely to change, but didn’t change during the two weeks I need it to work. Finding the right CSS classes is an easy task. Right click the thing you want classes for, and choose ‘inspect’ from the context menu. The element path complete with CSS classes will be displayed in the developer tools. I copied and pasted the above code into the console of the developer tools. When the done message was printed, I knew I was ready for scraping.

For more details about the configuration options I used, consult the Artoo.js documentation

Comment Scraping

My next task was to turn a pile of comments displayed in HTML into a data structure that I could use.

//scrape comments
all_comments = artoo.scrape({
    iterator: "ul._mo9iw>li:gt(0)",
    data: {
        user: {sel: 'a', attr: 'title'},
        comment: function($) {
                return $('span', this).clone().children().remove().end().text();
        },
        mentions: {sel: 'span>a', method:'text'}
    },
    params: {
        done: function(result){}
    }
});

The CSS selector used for the iterator looks similar to the selectors used for expanding, but I needed to skip the original post description that was the first element returned for the given selector. Using :gt(0) did the trick. After this code completes, the all_comments array is full of objects that follow the form specified in the data key of the configuration. This is mostly vanilla and explained well in the Artoo.js documentation. The one wild piece of code was in the comment attribute. In this case, the <span> element selected had some internal span comments wrapped around any @mentions. I wanted to strip the mentions out, and resoted to jqery specific antics to make that happen. Without those specific needs, you can stick with the more simple selector syntax.

Filtering

I wanted to quickly remove the comments made by us as we responded to people in the comments. Those comments were not relevant for tallying up contest entries. Note that you can filter by anything here, like the presence of a word or hashtag.

//filter out currentlywandering comments
filtered_comments = all_comments.filter(function(c){
    return c.user != 'currentlywandering';
});

Contest Entry Processing

Next, I grouped all the comments by user, and looked to see if their comment was long enough, and if they mentioned somebody else to gain an extra entry. For convenience, I also remembered the user comments.

//construct dict of username:entrycount
users = {};
filtered_comments.forEach(function(c){
    // newly found user
    if(users.hasOwnProperty(c.user) == false){
        users[c.user] = {
            user: c.user,
            hascomment: false,
            hasmention: false,
            entries: 0,
            comments: []
        };
    }
    var user = users[c.user];
    // calculate entries
    if(c.comment.length > 10){
        user.hascomment = true;
    }
    if(c.mentions.length > 3){
        user.hasmention = true;
    }
    if(user.hascomment && user.hasmention){
        user.entries = 2;
    } else if(user.hascomment || user.hasmention) {
        user.entries = 1;
    } else {
        user.entries = 0;
    }
    //store comments
    user.comments.push(c);
});

Now, I get the entries ready for picking a winner. I construct a list, and put the user’s username in once per entry. Think of the entry_list array like a hat. Each entry is one slip of paper, and we’ll randomly pull one out as a winner. Those with two entries have a better chance of being pulled out.

//construct array of entries by count
entry_list = [];
for(var username in users){
    var u = users[username];
    for(i=0;i<u.entries;i++){
        entry_list.push(u.user);
    }
}
entry_count = entry_list.length;

console.log("comment count", filtered_comments.length);
console.log("unique users", Object.keys(users).length);
console.log("entry count", entry_count);

I print out some stats that we wanted to know. The biggest surprise is how many poeple only had one entry.

Picking a Winner

Finally, the step we’ve been waiting for.

//chose random entry, display username and comments.
var winning_user = entry_list[Math.floor(Math.random()*entry_list.length)];
console.log("winner", winning_user, users[winning_user]);

There were a few things to verify for winners that I couldn’t easily do in the code here. We had to verify that they followed the sponsors, for example. I would pick a winner, and then we would verify that by hand. If the user didn’t qualify, we would pick another winner. Another time, we had a Canadian win a contest with only US shipping.

And That’s All He Wrote

The contest worked, and I saved Jess a heap of trouble trying to do this by hand. I’d prefer good API access to screen scraping for this sort of thing, but it’s nice to have good tools when scraping is the last option. Next time you need to scrape, consider giving Artoo.js a try.

Resolving timing conflicts with Neopixels and IR signal processing

2016-12-04T00:00:00-07:00

I’ve been teaching my kids about microcontroller programming this year as part of our homeschool curriculum. For a fun holiday project we’ve added a strip of 60 Neopixels to our project. Learning to animate them has been a fun project, and casts fun lights all over our Airstream home.

Our microcontroller for our learning is a Plumduino, which has an IR sensor and IR remote attached to it. Our remote came with our Plumduino package, but is similar to this one on Amazon. We’ve written some code to change animations from the remote. A simplified version of our main loop program looks like this:

void loop(){
  //handle IR
  if (irrecv.decode(&results)) {
    switch (results.value) {
      case IR_Up:
        offset_delta = 10;
        break;
      case IR_Down:
        offset_delta = 1;
        break;
    }
    irrecv.resume(); // Receive the next value
  }

  // play animation
  offset += offset_delta;
  fill_solid(leds, NUM_LEDS, CHSV(offset, 255, 100));

  // send instructions to Neopixels
  FastLED.show();

  // standard delay for 30 FPS
  delay(30);
}

The problem that we experience is very unreliable IR signal processing. The signal is received, but is nothing like the expected message after decoding. The remote has kind-of worked, but it has been frustruating. I have finally figured how the actual cause of the problem and a solution. First, we need a little background.

IR Sensing with Arduino-IRRemote

My library of choice for sensing and processing Infrared Remote signals is Arduino-IRremote. The library is easy to work with. It uses interrupts to measure the timing of of the rising and falling edges of the incoming IR signal. After a complete message has been received, it is decoded and made available for use. IR remotes take about 65 milliseconds to send a complete signal.

Neopixel Control with FastLED

Neopixels are the common name of the versatile WS2812 LEDs. It’s use of only one signal wire requires a very specific signal timing. Neopixel libraries accomplish this specific timing by disabling interrupts on the processor during the time the signal is sent down the control wire. Luckily, this is quite fast. It takes about 3 milliseconds to update 100 neopixels.

Timing is Everything

The problem occurs when FastLED.show() is called in the middle of an IR transmission. FastLED shuts off interrupts during the 3ms it uses to send the signal to the LEDs. When the interrupts are re-enabled afterwards, the missing few milliseconds of interrupts causes the IR library to misinterpret the IR signal, resulting in a corrupted message. As you can see from our loop() function above, we repeat the loop approximately every 33ms. This causes the IR message to be interrupted twice during the 65ms IR transmission. Under these timings, the IR message is never received properly. If we lengthen our delay(30) to delay(100) or greater, we can increase the chance that a 65ms IR signal can be received without corruption. This delay does slow down our animation and a significant number of IR messages are still corrupted. Repeatedly pressing a remote button and not seeing the desired change can be quite frustruating. This is especially true for a kid who is not sure their code is going to work anyway.

The Solution

The way to make these two timing specific processes get along is to avoid having them run at the same time. Newer versions of the library have an isIdle() method that is perfect for the job. I’m still using the slowly dying codebender, and they have an older version. I wrote the following function that detects the absense of an ongoing IR message.

bool IR_idle() {
  return irrecv.decode(&results) || results.rawlen == 0;
}

This method returns true if no IR message is in process. irrecv.decode(&results) returns true if a received message is completed, and also populates a few members in the results struct. If a message has not completed, we check the length of the IR timing buffer results.rawlen. If an IR message has started that buffer will be non-zero. A true result of this function indicates that a message has been fully received, or no message has been started.

Rather than mindlessly calling FastLED.show() and potentially disrupting an IR message, we can replace that call with a guarded call like this:

  if (IR_idle()) {
    FastLED.show();
  } else {
    Serial.println("Skipped FastLED.show()");
  }

This avoids interrupting an IR message with an interrupt disabling library call. The else clause and Serial.println() is optional of course, but is sure useful in testing to detect when the guarded call prevents a timing accident.

Note: if you are using the newer library version with the built in isIdle(), it would look like this:

  if (irrecv.isIdle()) {
    FastLED.show();
  } else {
    Serial.println("Skipped FastLED.show()");
  }

Using this method, my IR remote has become extremely reliable without unnecessary slowing of my animation.

Other Uses

This same technique can be used with other interrupt-using libraries. The method of detecting an in-process message will vary, but the results should be the same. If you happen to be building a library that uses interrupts, please help everyone by adding a similar idle method.

Full Code Sample

For completeness, here is a full example program, with my fix in place.

#include <FastLED.h>
#include <IRremote.h>

#define IR_Recv 3
#define DATA_PIN 4    //pixel pin, used for FastLED Library
#define NUM_LEDS    68
#define SERIAL_SPEED 115200

#define IR_Up 0xFF02FD
#define IR_Down 0xFF9867

CRGB leds[NUM_LEDS];        //array to hold FastLED pixel data

IRrecv irrecv(IR_Recv);
decode_results results;

//variables used
byte offset = 0;
byte offset_delta = 1;

void setup()
{
  FastLED.addLeds<NEOPIXEL, DATA_PIN>(leds, NUM_LEDS);  //start FastLED

  irrecv.enableIRIn(); // Start the receiver

  Serial.begin(SERIAL_SPEED);
}

bool IR_idle() {
  return irrecv.decode(&results) || results.rawlen == 0;
}

void loop(){
  //handle IR
  if (irrecv.decode(&results)) {
    switch (results.value) {
      case IR_Up:
        offset_delta = 10;
        break;
      case IR_Down:
        offset_delta = 1;
        break;
    }
    irrecv.resume(); // Receive the next value
  }

  // play animation
  offset += offset_delta;
  fill_solid(leds, NUM_LEDS, CHSV(offset, 255, 100));

  //FastLED.show();

  if (IR_idle()) {
    FastLED.show();
  } else {
    Serial.println("skipped FastLED show()");
  }

  //standard delay for 30 FPS
  delay(30);
}

Homeschool Project: Microcontroller Powered Pumpkins

2016-11-12T00:00:00-07:00

Home school this year involves microcontroller programming using arduino compatible development boards. A month before haloween, we decided to build a microcontroller based pumpkin light with some fun behavior. We worked on the code several times before we actually had a pumpkin to test with, and only made minimal changes. I turned out awesome. Before I get into the details of how, here is a preview of what the final result looked like.

The Hardware

Our main hardware for teaching is the Plumduino by Plumgeek with the optional expansion board. We didn’t use the expansion board itself inside the pumpkin, but it was used in testing colors, and we use the Passive Infrared (PIR) sensor that comes with the expansion board.

We also used some jumper wires, a long USB cable, and an Anker USB Battery.

Making a Flame

We wanted the pumpkin to look like a candle during normal mode, and so some testing was required. I wrote a quick sketch that helped us choose the right color configuration. The sketch varied the hue of the NeoPixel lights with one variable resistor slider, and varied the brightness with the other. A button press changed the number of LEDs that were lit. Once every few seconds, I printed out the current settings via the serial port, to be displayed on the connected chromebook in a serial monitor.

We took our hardware and the chromebook into the bathroom with a lit candle for testing, and decided that 2 LEDs with a hue of 37 and brightness of 100 was the best match.

During in-pumpkin testing, we later adjusted the hue to decrease the red tint for a better color match. We also added some flicker behavior to mimic the look of a real candle. We also adjusted the flicker later to match, and you’ll likely need to do the same to match any other candles present in neighboring pumpkins.

Angry Eyes

We hooked up our Passive Infrared sensor, and our sketch triggers an ‘angry eyes’ mode upon sensing motion. For our angry eyes, we lit all 8 NeoPixel LEDs with a red color at full brightness, and then waited for the motion to stop before returning to normal candle flikering mode.

The Pumpkin

Rachel carved the pumpkin with a great pumpkin face, and we cut a small hole in the back for the USB power cable and the jumper cables for the PIR sensor. We hooked 3 sets of jumper wires together to make it long enough for a good sensor position outside the pumpkin. We could have placed the sensor insde the pumpkin, but I didn’t want it’s shadow to obsecure the pumpkin face with the light behind it. The position outside also allowed for sensor position adjustment on the steps for proper triggering.

We carved a little bit of space from inside the pumpkin for the board to sit. After bending over the jumper wires to minimize thickness, we wrapped the board in plastic wrap to protect it from pumpkin slime. A small section of wooden skewer held it nicely in place inside the pumpkin.

Things To Change

The Anker battery we used has an auto-shutoff feature. When the charging device draws less than 50 milliamps of power, it guesses that the phone has reached full power and shuts off automatically. This gave us some trouble, as the power draw in flicker only mode was less then that. If nobody tripped the sensor and the Angry Eyes mode, the batter would shut off automatically. The Angry Eyes mode did draw enough power to prevent shutoff. We could have kept a time running, and triggered the angry eyes mode without any motion detected just to prevent the battery from shutting off. Or we could have used a less sophisticated battery without an auto-shutoff feature.

Color Testing Code

This was written by me, and used for finding the ideal hue and brightness of the lights for a candle effect. I recommend testing with the other pumpkin lights you plant to use, either real candles or LED. This will help avoid last-minute adjustments to color.

In both code examples, the PlumHardware.h file contains pin definitions and other state management code for the NeoPixel lights. This file is an adjustment of the one provided by PlumGeek for my own purposes.

#include <FastLED.h>
#include "PlumHardware.h"

byte hue;
byte brightness;
byte ledcount;

long last_print;

void setup()
{
	hardwareBegin(); //sets up hardware on the board.
	ledcount = 0;
	last_print = millis();
}

void loop()
{
	//read hue from slider 1
	byte hue = map(analogRead(Slide1), 0,1024, 0,255);
	
	//read brightness from slider 2
	byte brightness = map(analogRead(Slide2), 0,1024, 0,255);
	
	//button press changes number of leds
	if(digitalRead(Button) == HIGH){
		ledcount = (ledcount + 1) % 8;
		delay(100);
	}
	
	FastLED.clear();
	
	fill_solid (leds, ledcount+1, CHSV(hue, 255, brightness));
        
    FastLED.show();
    
    if(millis() - last_print > 5000){
    	last_print = millis();
    	Serial.print("Hue\t");
    	Serial.println(hue);
    	Serial.print("Brightness\t");
    	Serial.println(brightness);
    	Serial.println("---------------");
    }
    delay(50);
    
} 

For completeness, below are the relevant sections of the PlumHardware.h file.

// ***************************************************
// Pin defines
// ***************************************************

#define Button 2
#define DATA_PIN 4    //pixel pin, used for FastLED Library

//Pins for Plumduino Expansion Board
#define Slide1 A4
#define Slide2 A5

// ***************************************************
// end Pin defines
// ***************************************************

// ***************************************************
// General hardware
// ***************************************************
#define SERIAL_SPEED 9600    //serial speed used for Serial Monitor window
#define NUM_LEDS    8         //used for the FastLED library 

CRGB leds[NUM_LEDS];        //array to hold FastLED pixel data

void hardwareBegin()
{
	Serial.begin(SERIAL_SPEED);

	FastLED.addLeds<NEOPIXEL, DATA_PIN>(leds, NUM_LEDS);  //start FastLED
	FastLED.clear();
	FastLED.show();
}

Pumpkin Code

This code was written by Rachel, with a bit of guidance and help from me along the way. We were focused on appropriate use of functions for code organization, which is why they are in heavy use in this code.

#include <FastLED.h>
#include "PlumHardware.h"
int Hueofflame = 37;
int Brightnessflame = 100;

void setup()
{
	hardwareBegin(); //sets up hardware on the board.
}
void Growflame()
{
	for(byte b = 0; b <= Brightnessflame; b++)
	{
		fill_solid( leds, 2 , CHSV (Hueofflame , 255, b));
		FastLED.show();
		delay(15);
	}
	//Grow to full flame from black.
}
void Fadeflame()
{
	//Animate decrease brightness all the way to black
	for(byte b = Brightnessflame; b > 0; b--)
	{
		fill_solid( leds, 2 , CHSV (Hueofflame , 255, b));
		FastLED.show();
		delay(15);
	}
}
void steadyflame()
{
	//chill
	fill_solid( leds, 2 , CHSV (Hueofflame , 255, Brightnessflame));
	FastLED.show();
	delay(250);
}
void FlickerFlame()
{
	//Animate decreasing brightness
	byte flickerlow = 50;
	for(byte b = Brightnessflame; b > flickerlow; b--)
	{
		fill_solid( leds, 2 , CHSV (Hueofflame , 255, b));

		FastLED.show();
		delay(5);
	}
	// wait at low brightness
	delay(50);
	//Animate increase brightness
	for(byte b = flickerlow; b <= Brightnessflame; b++)
	{
		fill_solid( leds, 2 , CHSV (Hueofflame , 255, b));
		FastLED.show();
		delay(5);
	}
}
void Angry_eyes()
{
	fill_solid( leds, NUM_LEDS , CRGB::DarkRed);
	FastLED.show();
}

void loop()
{
	FastLED.clear();
	Growflame();
	while(digitalRead(Motion) == LOW)
	{
		
		int r = random(100);
		if( r < 10)
		{
			FlickerFlame();
		}
		else
		{
			steadyflame();
		}
	}

	Angry_eyes();
	while(digitalRead(Motion) == HIGH)
	{
		delay(50);
	}
	delay(1000);
}

Extensible APIs

2016-11-07T00:00:00-07:00

APIs are awesome. We need more of them.

Every API is about something specific

Flickr is all about pictures. OpenTable is all about reservations, Twitter is all about short posts. Generally, these are singleton APIs: only one of each type exists. Some APIs have many hosts with a compatible API. Any host running the WordPress API can be used by any tools that speak that API. The MetaWeblog API is a good long-running example of a cross-platform common API in a common space.

What if we want an API that is about pictures, reservations, AND short posts? We need a new class of API.

Extensible APIs are Composable Collections of APIs

A single API host can host many different types of APIs, while sharing common infrastructure.

Each API host can host the types of APIs needed for the entity it represents. The particular extensions chosen will depend on the type of entity and should be commonly shared with other APIs representing entities of the same type.

A BusinessHours API could be used by all types of businesses and could respond with information about… yep, business hours. A store might use that in addition to a ProductInventory API, while a restaurant might use it alongside a Menu API and a Reservation API.

The mix and match of API types allow easily interfacing with many different types of businesses with common code, and also allows multiple API definitions to be used simultaneously as new ones emerge and old ones are deprecated.

Technical Thoughts

Extensible APIs allow the combination of many different APIs on the same host. Each API extension lives within a different namespace to avoid path conflicts.

A well-known discovery URL can return a list of extensions supported by the host.

Each namespaced API needs to conform to a spec, likely something like swagger.

Conventions can allow multiple APIs to share API keys on the same host.

Extensible APIs will enable many different systems to build to common API standards, without having to agree on absolutely everything.

Critical Missing Infrastructure

I believe that Extensible APIs are required for distributed and independent systems. It is critical for healthy IOT ecosystems, and the growth of independent and distributed entities. Personal APIs are one type of Extensible API.

Open Questions

Is Extensible APIs the right name for this?
How should namespaces be organized? Having a predictable namespace for a given API spec will make it easier to hit the specific extension you want.
Linking namespace to an interface specification?
What are the minimum behaviors that extensible APIs must support?

What Personal APIs can learn from the Internet of Things

2016-09-27T00:00:00-06:00

Many IoT devices exist in an occasionally connected state. For both battery and bandwidth conservation, most IoT devices are only intermittently connected. They sleep between data collection points, only connecting to cloud resources to transmit data for status. This presents a challenge in communication, placing the burden of interrupted communication on the consuming service.

A pattern has emerged in IoT that uses a cloud resource for an IoT device. The cloud resource acts as a highly available proxy for the device itself, usually containing a record of last device state, data history, and allowing messages or data to be queued to the device. An occasionally connected temperature sensor can now make its temperature information highly available without maintaining a constant connection.

This concept is present in Picos, and also shows up in more limited form on the AWS IoT platform in the form of a Device Shadow. The pattern of pairing a cloud representation to a physical IoT device applies directly to Personal APIs.

My Agent

Establishing a cloud representation of individual people brings all the benefits demonstrated in the IoT world. My personal API is a highly available representation of me, allowing communication with me in spite of my partially connected state.

Our connectivity as humans is limited by gaps in cellular coverage, but more significantly by our need to not be glued to our digital devices.

Personal APIs can serve to answer questions about us without directly interfacing with us. If I’ve granted permission, you should be able to ask about my food preferences and allergies, request a phone call, or send me a message. You can do all these things no matter my physical location, cellular connection status, or my preferences for interruption.

Since the beginning of the Internet, we humans, the ‘users’ of the Internet, have been granted online presence only through the graces of established ‘always on’ businesses. With the exception of those who arranged their own domain, DNS, and hosting, most humans are represented online only by companies who sell access to us for a profit. Our presence on the Internet is shaped by their financial desires.

We Can Do This

If we want to be first class citizens in an API world, we need to find a way to establish an online presence on our own terms, and under our own control. The challenges of creating an API to represent us individually (including all the requirements for high availability) are difficult but not insurmountable. The reward for such effort is an Internet of peers, connected by choice.

Happy Birthday, Arduino Style

2016-09-19T00:00:00-06:00

We homeschool our kids, which gives us flexibility in the curriculum department. This year, I decided that microcontroller programming would make a nice addition to school for my 11 and 9 year olds.

I found a serendipitously timed Kickstarter for the Plumduino, an Arduino based board board with 8 NeoPixel LEDs, two proximity sensors, a button, and a wonderful expansion board with two sliders, a microphone, piezo buzzer, IR transmit and IR receive with an included remote.

I backed the project, and received it just in time for school this fall. My 11 year old Rachel is been the first student to attack this wonderful piece of hardware, and I’ve been guiding her through learning all sorts of things about Arduino based microcontroller programming. (My 9 year old is spending some additional time with Hour of Code based learning to prepare for it, and will be beginning later this year.)

Rachel just turned 11, and I wanted to prepare a small birthday surprise.

I found a code snippit that plays the happy birthday melody out of the piezo buzzer, and combined it with some LED code and use of the IR receiver and the IRRemote library to recognize a press of the play button on the remote.

The code is organized like this:

Play simple animation on the LEDs.
When the IR receiver receives the play button, play the birthday song.
During each note, play a random LED set to a random hue.
After the song completes, return to the simple animation.

And the code, via CodeBender:

I wrote this the evening before Rachel’s birthday, and left it out on the table. When she woke up, I told her how what to do to activate it, and she loved it! The first ‘happy birthday’ song she was sung was provided by an Arduino!

We Need Personal APIs

2016-09-05T00:00:00-06:00

Imagine a world where only businesses had phone numbers. You can call any business you like, but connecting a phone call to another person requires that you ask a business to help you. Businesses can’t call us either, so the only way to have phone calls is to stay on the line with a business in case somebody wants to talk. If both you and I are connected on the phone to the same business, we can ask to be connected, and then we can talk. The business may place restrictions on our phone call, including which words we can speak and the length of the call. They may even regularly interrupt the call with advertisements to ‘enhance our experience.’

This scenario might sound completely ludicrous, but we are all living in a reality that functions just this way. We all have apps on our phones that stay connected to Facebook, Snapchat, and Instagram. We get our updates through the app. Our contacts are all maintained in the app, and our communication is required to fit the requirements of the apps that we use.

Imagine if your phone calls were ‘algorithmically rearranged’ for your convenience.

None of us would put up with our voice calls being interfered with in this manner, but we don’t think twice about allowing our other forms of communication to exist in such a perverse way.

We Need Phone Numbers

To solve this problem, we need a way to communicate directly in all forms of our communication. Software talks to other software via APIs, allowing systems to coordinate, automate, and share data. APIs already exist all over the internet, but only businesses have them. Your Facebook app on your phone calls a Facebook API on Facebook’s servers to load information. Businesses communicate with other businesses with APIs. I use a Business called IFTTT to repost my Instagram Posts to Facebook.

The existence and use of all these Business APIs is great. The Internet is a much better place with these APIs in use. What we need is a way for businesses (and other people) to call us, to call our API.

In order to be first class citizens in the api economy, we must have APIs ourselves.

Remaining in our current state will only further cement our position as ‘assets’ to the business who sell access to our eyeballs and tightly control the data behind their APIs.

A Loose Beginning to Personal APIs

The first blog post that I read mentioning Personal APIs was written in 2013 by Foursquare co-founder Naveen Selvadurai. He wrote about tracking personal metrics in one place, and posted the start of a personal data aggregation api that appears to no longer work. Naveen’s work demonstrates the value of an aggregated Personal API. A user of his API does not need to know where the data originated, leaving Naveen in control of the interaction. Indeed the backend host of the data could change without changing the API interface in any way.

This was followed up a few months later with a post on ReadWrite titled Want To Reach Me? Call My API. Naveen’s API was read only, but ReadWrite author Brian Proffitt expanded the concept from data sharing to a request for interaction. Having a read/write API allows for conversations and transactions to occur that extend far beyond the concept of data sharing.

There has been other related work which I will address in future posts, but for now, I want to focus on these two main points: an aggregated api, and the ability to have conversations. While data sharing is valuable, I believe the ability to have conversations is the greatest of these two.

Personal APIs are NOT about data, they are about conversations. Even if those conversations are about data.

Much work has gone into data sharing and authorization and delegation protocols. This is valuable work. Personal APIs have the potential to move beyond data sharing and synchronization and into a world where regular API conversations occur from businesses to people, and between people directly.

Brian highlighted a perfect example of such an exchange in his post when he dreams about coordinated personal communication. I can envision a future where an API exchange happens prior to placing a phone call. It might look something like this, in a conversation between my phone and my friend Kurtis’s Personal API:

MyPhone: Hey KurtisAPI, Sam wants to talk on the phone.
//KurtisAPI does some status checking for Kurtis's personal location, phone status, and preferences.
KurtisAPI: Now's not great, but Kurtis should be available in the next half hour.
//MyPhone asks me if sometime in the next half hour would work. I say yes.
MyPhone: Please register a low-priority request for a call back within the next hour.
KurtisAPI: Request received.
//MyPhone displays a message to me that a callback request has been made.

I took some liberties for the sake of brevity, but I believe the main idea is clear. Consider the advantages of this approach:

Kurtis was not directly interrupted during this exchange.
None of Kurtis’s private information such as location or phone call state was shared.
This exchange was faster than actually talking with Kurtis and coordinating the same thing.
Kurtis now can be automatically notified of the call request when his own systems determine the appropriate time to interrupt him.
This exchange didn’t require the cooperation of any 3rd party apps or businesses.

There are many scenarios that describe the value of Personal APIs, and I hope to explore them in future posts. I will also explore the complexities and challenges of Personal API implementation. I hope you will join in this conversation with me as we explore and develop the possibilities.

Thanks to Phil Windley and Kurtis Welch, whose conversations have contributed significantly to clarity in thinking.

Long Term Blogging Requirements

2016-09-02T00:00:00-06:00

I’ve maintained the code for a blog before, which led me use a hosted solution. That hosted solution was shuttered following an acquisition. I’m now attempting to navigate the gap between inventing my own and leaving myself vulnerable to service shutdown. Here are some of my thoughts on how to navigate this gap.

Fail Open

The frustrating shutdown of a service would be made nicer if the existing content remained in place, rendered statically and available for the future. This sort of happened with the Posterous service. A site called PostHaven popped up immediately that would import Posterous sites. It is still live, and they have adopted the ‘forever’ principle into their current operations. If the software that generates the site fails or is no longer maintained, the site should exist in it’s current state. Forever if possible.

Cheap and Easy

If I want my content to last forever, it has to be cheap, and it has to be easy. Our family adventure blog runs on hosted WordPress, but it’s an active site that is worth having on a more capable platform. My personal blog will receive much less attention, and needs to not be a financial burden. It must also not consume undue amounts of time to maintain. I don’t mind the occasional fix, but I don’t want to spend lots of time to keep it up. Installing security patches or having the blog compromised is a painful time-suck.

Future Friendly

The single most important requirement here is to have long-lived URLs. Changing URLs breaks things, and that must not happen if the host or tools change.

My first blog stored content in a relational database, which is both unnecessarily complicated and unnecessary. Simple data storage means it can be easily used by multiple tools. Low host requirements make it easy to move the blog to a new host if necessary. Finally, I want low client-side requirements for any software needed to update the blog. Ideally, only a browser would be the only required client-side tool.

Jekyll: Almost good enough

Anybody familiar with the Jekyll static site generator will realize that it fills many of my requirements. Indeed, Jekyll is closer to my ideals than anything else I can find. It does fail spectacularly in the client-requirements requirement however, with poor Windows support and locally installed Ruby with a collection of Ruby libraries.

Where Jekyll really shines is the format used for storing blog posts. Meaningful filenames in various markup languages annotated with frontmatter. The site can be generated from these files, a few config options, and a few template files. I consider this format to be extremely future friendly, and plan for it to be part of my long term solution.

Now

I’m using Github pages, which has built in Jekyll support without a local Jekyll install. I’m using a custom domain to allow moving the content to another host in the future. I’m using prose.io as an editing tool for the content on GitHub. Prose.io has a great file editor, draft automation, and even media upload. For now, this is a good-enough solution that reserves plenty of flexibility for the future.

Later

I have a vision in my mind (and even a bit of code) of a Chrome extension that maintains a Jekyll blog hosted on S3. This would keep the client requirements low, use the same future friendly file format, and be cheap and trouble free to maintain. This would be a javascript port, and so could not maintain perfect Jekyll compatibility. It would be much easier to use than jekyll but have more full featured options than allowed by GitHub Pages. This vision would allow migration from my current situation when the project reaches sufficient maturity. Timeline? No idea, but it heavily depends on the comfort of my current setup.