ConnectorDB

For the past 2 years, I have been working with Joseph Lewis on a platform for quantified self data. Today, it is finally ready for use.

There are many apps that gather data for you. On your laptop, there is selfspy. For phones, there are multiple, each integrated with varying amounts of fitness trackers. Unfortunately, this means that if you wanted to track the details of your life, you’d need to find several services, and try downloading their data periodically. This isn’t even mentioning the security implications of letting companies have a detailed view of your every move.

I personally wanted something with the following properties:

• Self-Hosted - I want to keep very personal data. I don’t want anyone to have access except me, and I am willing to set up a server to do it!
• Open Source - Closed source solutions are ultimately tied to their creators. Once the maintainers are done, you’re out of luck, unless it is OSS.
• Multi-Device - while single-focus apps exist for almost every device, combining data is not a fun task. The “ultimate quantified self solution” must be able to integrate data from multiple sources - from all of my computers, my phone, and any sensors I might get.
• Automated - 2 years ago, I built a web-app using which I tracked details of my life manually. It was tedious work, and I quickly forgot about adding data. A real solution to data gathering must be automated - most data must be gathered without any of my input, so that I can forget about it for weeks at a time.
• Easy analysis - Data is useless if it isn’t actually being used. A real solution must make it easy to perform custom analysis using my programming language of choice. The goal here is not to do analysis myself, but to figure out a way to do Machine Learning on my data!

With these goals in mind, we’ve created ConnectorDB. It isn’t much yet - it doesn’t do any analysis or visualization for you. But, it gathers data. A lot of data.

And it is easy to add devices, so that it gathers even more data.

Let me show you ConnectorDB!

I have an ubuntu server set up on DigitalOcean, where I have ConnectorDB running.

After logging in, ConnectorDB shows me my input screen. On this page, I can rate how well my life is going - these ratings are all saved, and can be accessed when performing analysis, so that my data can be correlated with my actual mood!

You can set up any ratings you like by clicking the star icon - these are the ones I found to be useful. Of course, star ratings are just one form of input - you can manually track however many things you’d like by creating custom streams (the + icon).

As I mentioned before, though, manual data entry quickly becomes tedious, so it is important to set up ConnectorDB so that it gathers data for you!

ConnectorDB comes with a python laptop-logger app and an android app. The laptop logger works on both linux and windows. It gathers data in the background of my laptop, saving the number of keys I pressed, the titlebars of the currently active window, and other such metrics.

The laptoplogger syncs with ConnectorDB every hour, and the data shows up in ConnectorDB immediately after sync.

Notice that this page is fairly empty. This is because ConnectorDB does not yet have any visualization! In the future, this page would offer simple ways to see and analyze your data.

ConnectorDB also has an associated Android app, which gathers all sorts of goodies from my phone:

Admittedly, this isn’t much. There already exist many apps that do visualization much better, and allow you to see much more detail. Hopefully this can be fixed quickly - the GUI was only recently created, and still has much work in store.

Python API

Of course, this would be useless if I couldn’t access my data. I am particularly proud of the Python API, since I think it allows really easy access and analysis of data.

For example, this is all the code you need to create a custom device that syncs to ConnectorDB once an hour:

def getTemperature():
    #Your code here
    pass

from connectordb.logger import Logger

def initlogger(l):
    l.apikey = raw_input("apikey:")
    l.addStream("temperature",{"type":"number"})
    l.syncperiod = 60*60

l = Logger("cache.db", on_create=initlogger)
l.start()

while True:
    time.sleep(60)
    l.insert("temperature",getTemperature())

All of your IoT devices can be included into one large dataset with only a couple lines of code!

What do you think?

ConnectorDB is definitely a work in progress. But hopefully, it can quickly gain missing features to become a one-stop location for quantified self data! You can set up your own copy of ConnectorDB by following the installation instructions on the website.