App for Markov Chaining Facebook statuses

As a small but cool programming project, I thought it'd be interesting to Markov chain friend FB statuses. The memory-less nature of Markov chains make it pretty simple to implement with a dictionary and a random function in Python: http://en.wikipedia.org/wiki/Markov_chain.

This project was pretty straightforward but still interesting. It was mainly to experiment with Markov chains. I had originally wanted to do FFT or fast fourier transforms to identify frequencies in music and then Markov chain the different notes together. It would be extra cool if we could simply enter different YouTube links in and have them parsed.


The source is here: https://github.com/deloschang/markov-chain-fb-statuses

Sample result:

In general, a larger corpus yields "higher-quality" results. So I set off to scrape as much as I could from my Facebook friends. Here's how I did it

First, I implemented Facebook Auth with sufficient permissions to check friend statuses. With the access token in hand, my objective was to iterate through each of the statuses and scrape the message. Also, any comments from the user within those status too.

Facebook limits the number of statuses per API call so the offset parameter will need to be looped like so:

    full_data =  graph.get(FB_DESIGNATED+'/statuses?limit=100&offset='+str(offset))
   ....open corpus.... 

    while not not full_data['data']:
       ......  scrape the data here .......
        offset += 100
     
        full_data = graph.get(FB_DESIGNATED+'/statuses?limit=100&offset='+str(offset))
 
    corpus.close()

 The 'not not' is quite pythonic and checks if we've reached the end of all the status update loops. If so, Facebook API will return with empty data.

Once I scraped the API calls, I just needed to save them to an external file. Then Markov chaining them involved iterating through each line, splitting the words and placing them in correct key value pairs.

In the near future, beyond uploading it onto a server, I probably won't be updating this project further. But here are some interesting ideas for anybody that wants to fork this repo.

• Integrating with twitter. You could go into friend's about mes and look for Twitter IDs. Then scrape the twitter posts for a larger corpus. 
• Virality. Give every person their own page for Markov chaining their statuses. Then when friends Markov chain each other, they can simply link their friend to that page via automated FB post. And when those friends come check their page out, encourage them to Markov chain their own friends, hopefully leading to a viral coefficient > one.
• Integrate with photos. As a developer who worked on an Internet memes startup, it'd be sweet to add these Markov chained texts as captions to random friend photos. I don't think this has been done yet and it'd be really interesting to see the results.

Actually.... now that I've listed these ideas out, I'm a little tempted....

Concerning Privacy Concerns

I'm seeing many "data-learning" apps shy from speaking up about the data-collection portion of their application and how that affects user privacy. How much data are you collecting? Where is it all going?   It's intuitive: people like privacy and if you collect more data about them, they'll protest and leave. Right?

This isn't a post about privacy concerns.
Rather, it's a post about how we should be asking for more data.

What?

Like Timely, these are the kinds of apps that gather data about you and process it to offer you some service, hence some value. To clarify, these are not apps that collect data they don't necessarily need (doesn't that remind you of spyware?). That's a big distinction that I think people conflate so often that in most people's minds, data collection = bad.

I believe that apps like Timely that can leverage more data to provide more value shouldn't fear.  Of course, it  needs to be a sufficient amount of value or people wouldn't risk their personal data to even use it.

The privacy status-quo is changing. Look at how people gave up their privacy for Facebook. Sure, there are privacy concerns about where our data is going and how people are using it, but overall, it seems that becoming more socially interconnected was worth the price. The common response now to privacy evangelists is "if you want to keep your data, don't use Facebook."

Leveraging more high-quality data spurs innovation. It unlocks doors. Big Data, though hyped, is real and it's powerful. From an application point of view, data is data is data. Higher quality data intelligently processed means more value, although this isn't always the case.

The problem for developers is that talking about how you want more user data isn't just eccentric, it's creepy by current social standards. So many app developers tiptoe around this by choosing not to talk about it or obfuscating the topic.

It's obvious  they  aren't following through with their convictions because they fear negative backlash. That's fair; people tend to be conservative with their personal data. But I'm curious.

What if they had followed through on this insight?

What's possible?

Playing with Bashrc

Recently discovered how time-saving aliases are.  For git, the following are what I use the most:

alias gp='git push'
alias glog='git log'
alias gs='git status'
# for easy git committing
function gc() {
  git commit -m "$*"
}

Because alias doesn't accept parameters, function gc() allows you to type in something like:

gc this is my commit

Or for a bash script that commits and pushes to server, you could write:

echo "Please type commit message"

read commit_message

git commit -am "$commit_message"

git push 

"Pushing onto repo"

Also,  essential for aliasing is:

alias bashrc='mvim ~/.bashrc && source ~/.bashrc'

This will let you edit bashrc and then source it automatically afterwards.

As I commit pretty often, these aliases have saved me quite a bit of time.

 

GDB Set disassembly

Recently saw that many linux books are referencing "set dis intel" or "set disassembly intel" when looking at breakpoints and registers. In OSX, this spits out the ambiguous response "Ambiguous set command."

This can be fixed using "set disassembly-flavor intel." And to preserve it on startup, simply set it in ~/.gdbinit via sudo echo "set disassembly-flavor intel" > ~/.gdbinit

Seed funding for Timely!

It's been a busy but great week so far.  We spent about 1.5 weeks building a beta product and received $2000 in seed funding.  

What is Timely?

At Timely, we believe that time management is broken.

1. When we think about time management, it means inputting events in the future and then trying to budget our time based on how long we think our tasks will take. But this is crazy!  Why are we budgeting our most valuable asset based on guessed estimations?  It'd be unwise to budget our money that way. 
2. There's no easy way to look back and evaluate how you've spent your time. You could open your calendar, but you'd really only see fixed events. There are some applications like RescueTime (which is a great solution for tracking time on your computer) to other automatic time-tracking applications.  But many applications we've used are pretty disconnected from our lives.  We're changing that. 

We're incredibly excited about Timely, not only because it's an idea that solves our own problems, but because it's also an interesting programming challenge.  It seems to me that smartphones are still in their nascency -- location-tracking and memory capabilities are limited and have a long way to go. In fact, Paul Graham notes that getting to the edge of smartphone programming could take a year.

OAuth Complications

Recently, tackling an interesting problem regarding OAuth 2. We want to allow the user to be able to register via Google OAuth on a mobile app and simultaneously store their records on the server (email address etc.) In addition to that, we also need offline access via a token (refresh token). If the user registers via the mobile app, we need to grab this offline access token, send it to the server, which should return a cookie. Then from there, the mobile app can process as it normally does and push the requisite data to the server.

Update:
Seems like Google Play Services only allows you to get an access token (not a refresh token), which is a shame because the Account Chooser is a pretty slick feature (no need for the user to enter in a username or password). I added in a quick async task to check what the token is, spitting it out onto LogCat.

  // Fetches the token successfully and shows in LogCat as async thread

  private class fetchToken extends AsyncTask<Void, Void, Void> {

    /* (non-Javadoc)

     * @see android.os.AsyncTask#doInBackground(Params[])

     */

    @Override

    protected Void doInBackground(Void... params) {

      // TODO Auto-generated method stub

        try {

          String token = credential.getToken();

          Log.d("calendar", token);

        } catch (IOException exception) {

          // TODO Auto-generated catch block

          exception.printStackTrace();

        } catch (GoogleAuthException exception) {

          // TODO Auto-generated catch block

          exception.printStackTrace();

        }

        return null;

    }

  }

Returning:  ya29.AHE.....  (an access token). 

The challenge is to figure out how to get a refresh token for the server, as we cannot assume that the user will onboard on the web app first.

Something New

Shipping something new, soon.