Bomber Hack Project: CommRater

The CommRater site was originally Aviana’s idea which she came up with one of the many times we were procrastinating in the updown this past year. We talked about the project sometime before (or at the beginning of second semester) but development didn’t start until this summer. I really wanted to start during the year but got caught up with work and became too lazy to start a side project. In the end I think this was the best decision since it resulted in a much higher quality website being built by Max and I.

At my summer internship the project I’m working on involves using Django so we thought it would be a good idea to make this a Django project. Neither of us had ever used Django before but we decided that since I had to learn the framework for work anyway. Django has many built-in features that something more lightweight like Flask doesn’t offer out of the box which also made it a good choice. We began the project towards the end of June (or sometime thereabouts) and have been working on the site almost every night ever since. We committed most of our nights after work and a fair amount of our weekend time the past few weeks. The site is now live but is still very much in development right now.

The basis of the site is fairly simple. People have to write comm.prods (funny snippets at the end of emails to the floor) and there was no quantitative way to rate the quality of these comm.prods The site initially started as a simple voting system of up and down votes for each prod allowing the floor to collectively decide which prods were actually good or bad.

As time went on, however, Max and I decided that the site should be much more than a rating system and should really be a home online for Bombers. I’ll briefly outline the systems that we have in place currently as well as our vision for what the site is to become in the future. If you are (or were) a Bomber and you want to register email us at: comm.prod@mit.edu and we’ll hook it up.

Scraping:

The basis of the site is being able to take prods from the floor’s email and post them to our site. This functionality took a lot of trial and error due to dealing with the nightmare of email parsing. I must of dumped the database 100 times before we got the regexes and email thread splitting functionality working at the level that we wanted it.

The basic flow is as follows:

First we added a gmail to the floor mailing list so that we could have a reliable place to get the emails to the floor from. We then setup a MIT XVM machine running a cronjob to ping Gmail every minute and check for new mail. If mail is received it is parsed and posted to the website so we have (almost) live data.

The overall process is fairly simple however the details were hard to get right. The parser accounts for the hundred different ways that email clients decided to make threads (we need to chop off the old content to avoid duplicate prods) and all of the ways that people write ‘ a btb “content here” comm.prod’. It was a fun process and definitely helped me get more savvy with regexs and parsing. Another complication was that people send emails to the floor from multiple emails (MIT mail, Gmail, etc) so we setup a ‘half’ user profile for this case and give users the options to claim these account which merges the two accounts into one.

UserProfiles:

Each user has a profile page in which we display some basic stats of how your rank against the floor and against your class. We also offer the ability to filter the user’s prods with filters such as Recent, Best, or Worst. Each user also has a score which is, based on the number of upvotes for commprods and corrections (discussed below) that they submit.

Correction System:

The parsing problem is one that will never be perfectly solved. Users can write their email in such a way that it is incorrectly parsed leaving wrong content or missing content on the site. To help fix this I built a correction system that allows users to try and fix the parsing mishaps. The basic idea is that the problem can be fixed by crowd sourcing. Users can submit a correction which can then be upvoted and submitted (a correction needs 5 upvotes) or downvoted and deleted (5 downvotes). This is probably the least used feature on the site right now except by admin use but with V2.0 it is more accesible for users.

Display:

The display of commprods is largely influenced by Max’s experience working at Twitter (he copied some of the CSS directly 🙂 ) giving each commprod a “tweet” look and feel. Each prod has a voting option, current score, author and content. In addition you can hover for a snippet of the original email content or click to get the permalink page to see the full content or submit a correction. Commprods containing media content (a url, image, or youtube video) are displayed inline so that the prod is fully shown during the render. The latest release will also allow users the favorite commprods so you can go back to ones that you like.

Trending Score and Data:

Another way to sort prods is by trending score. Each vote adds a fixed amount to the trending score and each minute (with the ping to Gmail) we subtract a fixed amount off of this. This allows us to sort on trending score and display recently voted on prods to users.

In addition to trending scores, each day we record a trenddata point for each user. We basically take a snapshot of their current total score and store them so that we (in a later version) we can show floor trend graphs of user performance. Currently we use the trend data to compute your 30 day trend score.

Future:

The latest release features a whole new set of commprods (thanks to Chris Post) so the site has data from 2005 to present. Max is working hard on a recommendation algorithm so users can be targeted with relevant prods (which is important as the dataset grows and spans many years). Some other new features that I have been working on include favoriting of prods and sorting by prods that contain media content as well as some internal admin tools and optimizations.

Ultimately our goal for the site is to be more than just a way to display and rate prods. We have a few people that want to help develop and we hope to continue working on the project during the school year. Some ideas that have bounced around for addition to the site are as follows:

Text alert system to notify the floor of happenings

Airport/Taxi management system

Floor *shopping* management system

Photo display for prospective MIT students to see what it’s like to be a Bomber

Donations page for alum to give back some love (by that I mean money 🙂 )

Currently the site is up at: burtonthird.com

Sorry, but it’s not public. If you want access (and are a bomber) let us know at comm.prod@mit.edu

Our code is up at, check it out: https://github.com/cantbloom/comm.prod

Leave a Reply