As I mentioned in the previous post, DoubleDibz was a craiglist-like community based market place website. I worked on it during my last year at UCLA but haven't maintained it since then. Now I want to open source the code since it could be useful for anyone working on similiar projects! If you are planning on building a social network on Flask, this is the project for you!

A couple highlight for the codebase

• A large and complete Flask application serving purely as API endpoints which powers a complex Backbone application in the frontend.
• A complete marketplace social network. This includes real-time notifications and messaging support.

The code

Come here to check out the entire source code.

What's included in the code?

Technology used

• Flask
• SqlALchemy + alembic for db migration
• a bunch of Flask libraries
• Backbone + Marionette js
• fabric for deployment
• celery
• Facebook SDK for login
• redis
• Nginx
• images stored on S3

Backend features

• Purely API endpoints.
• Chat.
• Post create. This is pretty straitforward. The highlight here is every single photo will be resized and cropped and saved on s3. See here.
• Comment on posts.
• hashtags on products. So every products could have category/hashtags. Users could explore products for a specific hashtag.
• Notification. DoubleDibz is completely social. User receives notifications when the products they subscribed to are sold or commented.
• celery workers for notification. I may have over-engineered the system. I was originally concerned that if a product has a lot of subscribers and someone comments on that product, it doesn't make sense to send notification to every subscriber in the same web request. So I introduced celery to handle those tasks asynchronously.
• Search. A really tricky part of the app is I have a database with product posts user created on DoubleDibz but I also have a separate server which pulls products from Facebook groups. The search aggregates results from both DBs. Paginations are supported as well.
• caching results with redis. To optimize for read, I cached user notifications, post data, search results, and user data. The code is also setup to invalidate cache upon write.

Frontend

The frontend is the most fun part of the project. I learned so much about how to structure a semi-large scale Backbone application to support all the features I wanted to make. All the things I mentioned before has a frontend counter part. You can upload photos to create a new product, look at home feed which is an infinite feed, then search for products and categories, look at product details page. You can message the seller for a product you are interested or comment on the product. You can even comment and tag your FB friends and your friends will receive notifications on FB!

The main thing I want to mention is Chat. This is the most complex part of the system. The challenge here is that every x seconds I'm pulling new messages from the system, and given those new messages, I need to update the correct Backbone view components. This includes updating the message count and also the message thread.

Deployment

• uwsgi configuration for the application server
• production and staging nginx configuration
• celery configuration
• Fabric script for running deployment (push)

Take a look! Maybe there's something you can use right away.

In the past couple weeks, I have been developing GoChat, a Facebook messenger bot that allows you to play a game of Go (Weiqi, Baduk) right in Facebook Messenger. It is essentially a Node server that accepts callbacks from Facebook whenever someone messages my Facebook page.

I have learned a ton from the project and I think what I used would be useful to y'all as well. I extrapolated a few things that I think any node projects improve any Node projects. Keep reading if you're working on a Node or Javascript project!

Source code

See source code for reference.

1. Install Babel

Everyone is using Babel and you should too. Babel is a Javascript compiler that allows you to write modern specification of Javascript (JS6 and proposed JS7) that the V8 engine or your browsers have yet to support. This page has a detailed list of nice features for ES6. There are tons of goodies like arrow functions, destructing objects and classes.

To take advantage of this right away, install the following. babel-cli is the command line interface for Babel. babel-preset-es2015 consists of all the plugins proposed for ES6.

> npm install --save-dev babel-preset-es2015
> npm install --save-dev babel-cli

Then create a .babelrc file in the root directory. This file will define the rules for the compiler.

{
  "presets": ["es2015"],
}

Now, put all your source code in .src/ directory. We will setup npm scripts that compiles your code to .build/ folder.

// package.json
"scripts": {
  "watch": "babel --watch=./src --out-dir=./build --source-maps inline --copy-files",
  "build": "babel ./src --out-dir=./build --source-maps inline --copy-files",
...
}

When you develop, have a separate session (tmux) open that runs npm run build. The script will then listen to any changes in the src folder and auto-compiles the code. To make the process even better, add this to package.json. This will restart the node server when anything in build changes.

> npm install --save-dev nodemon

// package.json
"scripts": {
  "server": "nodemon build/app.js --watch build",
  ...
}

Awesome! Now you can use any ES6 features in your Node app. Keep in mind that you can always include more babel plugins.

2. Flow typechecker

Like most scripting languages, Javascript is dynamically typed which means even tho you can code a bit faster but you would inevitably waste time catching stupid bugs. This is when Flow comes in. It uses type inference and type annotation to catch those stupid bugs.

> npm install --save-dev flow

Once you install flow, you need to specify which files you want flow to analyze. Add the following:

// @ flow

Then simply run the flow command to start checking your code!

You may notice that adding flow annotation is breaking babel compiling. No worries, simply install

> npm install --save-dev babel-plugin-transform-flow-strip-types

And add the babel plugin to the .babelrc file

{
  "presets": ["es2015"],
  "plugins": [
    "transform-flow-strip-types",
  ],
}

Not only does Flow help catch bugs, it also makes your code much more readable! Knowing precisely what the input format and the return type is are extremely helpful to create a mental model for the program. Imagine the following function from GoChat which renders a list of stone positions on a board image.

playStones(stones, board, cb) {  
  lwip.open(board, function(err, boardImage) {
    if(err) {
      error('Cannot open board image', err);
      cb(err);
    } else {
      _playStonesHelper(stones, boardImage, cb);
    }
  });
}

If I'm brand new to the code base this would be going through my head, "What are stones? Are they objects or array? What properties does it have? cb seems like callback but can I be exactly sure? Oh yeah okay it is..." As you see, the code could be interpreted in different ways and you are required to trace through other functions to have a better understanding.

Now, take a look at how this would look like using flow.

type StoneColor = 'black' | 'white';  
type Stone = {  
  x: number,
  y: number,
  color: StoneColor,
};

playStones(stones: Array<Stone>, board: string, cb: Function): void {  
  lwip.open(board, function(err, boardImage) {
    if(err) {
      error('Cannot open board image', err);
      cb(err);
    } else {
      _playStonesHelper(stones, boardImage, cb);
    }
  });
}

Much cleaner.

3. Linting with ESLint

Programmers are lazy and prone to make careless mistakes. And that's okay, as long as there are programs that clean up our messes. Linting tools are designed to do exactly that. JSLint and ESLint are both popular for javascript but the latter offers much better flexibility to add custom linting rules.

> npm install --save-dev eslint

Then add a .eslintrc file. Then you can define the rules you want from all the supported rules.

One tricky thing here is that since we are using Flow, our source code are not proper Javascript so ESLint will have parsing issues. Good news is that ESLint allows you to define custom parsers to transform into code that ESLint could understand.

> npm install --save-dev babel-eslint

Then in .eslintrc, define...

{
  "parser": "babel-eslint",
  "env": {
    // in CommonJS
    "node": true
  },
  "rules": {
    ...
  }
}

ESlint is great for maintaining consistent coding styles. Personally, I prefer using trailing commas and the rule comma-dangle has been wonders. By running eslint src --fix with the fix param, ESLint even auto fixes places where I don't have trailing commas.

If you ever feel overwhelmed and not sure what rules to you use, you can always download the standard rule sets and call it a day.

4. invariant

Invariant is a little handy module that I used very heavily in my app.

> npm install --save invariant

Without getting into too much details of its importance in Software Engineering, using invariants helps me keep track of the correctness and assumptions of my functions. I use invariants a lot to assert the type and data correctness of function arguments and also to verify an action is proper given the current application/user state. For example, I have a series of functions that are only applicable to users currently in a game, so I would throw an invariant in these functions:

async resignGame(user: User): Promise<void> {  
  invariant(user.isPlaying(), 'User should be playing');
  ...
}

This not only catches careless errors but is a great code documentation for me and my partner. Install now and try it for yourself!

5. async and await

Javascript is asynchronous and single-threaded. This means for many operations such as talking to the DB, making a http request, the application will kickoff the function and then process thread will "unblock" itself and run on something else. That means your code will most likely have a bunch of callbacks. A hell of nested callbacks!

function doSomething(callback): void {  
  doSomethingFirst((result1) => {
    okay(result1);
    doSomethingSecond((result2) => {
      more(result2);
      doSomethingThird((result3) => {
        callback(result3);
      });
    });
  });
}

There are methods to make the code cleaner when you want to avoid nested callback hells - avoid nesting functions, modularize the callbacks, etc. Personally, callbacks just make make my code much more unreadable and makes it difficult to handle exceptions. I prefer writing serial code and the framework should handle the asynchronous processing.

Fortunately, you can use await and async functions proposed in ES7 features. The above code will be equivalent to

async function doSomething(): Promise<void> {  
  const result1 = await doSomethingFirst();
  okay(result1);
  const result2 = await doSomethingSecond();
  more(result2);
  return await doSomethingThird();
}

Isn't that much nicer? It is very clear how the logic flows and the code is much more concise.

You can use it right away using Babel compiler. Under the hood, it will convert async/await to generators.

> npm install --save-dev babel-plugin-syntax-async-functions
> npm install --save-dev babel-plugin-transform-async-to-module-method

And update your .babel

{
  "presets": ["es2015"],
  "plugins": [
    ...
    "syntax-async-functions",
    ["transform-async-to-module-method", {
      "module": "bluebird",
      "method": "coroutine"
     }]
  ],
}

Finally, since v8 doesn't provide generator functions, you need to import the Babel polyfill environment which adds a bunch of ES2015 features to the global scope.

> npm install --save-dev babel-polyfill

Then add to app.js

import 'babel-polyfill';  

6. Test with Mocha

Always test your app. It's easy to forget about it and only focus on the business logic but trust me, at the end of the day, the only thing that makes me sleep at night is knowing that my code passes all the tests. When your app become more complicated, unit tests or integrations tests ensure any new changes don't result in regression or bugs.

For testing in Javascript there are plenty of options but I really enjoy using Mocha for its simplicaity.

> npm install --save-dev mocha
> npm install --save-dev chai 

A tricky thing I had was being able to use async/await for my unit tests. It's actually quite simple, when starting the mocha script, you could specify Babel as the compiler. It will then compile the code on the fly and run the tests.

// package.json
"scripts": {
  "test": "NODE_ENV=test mocha --compilers js:babel-register",
  ...
}

Let's see how to make a simple integration test that checks GET / returns 200. Note that I'm starting a new instance of server for each test case. This makes sure theres no order dependency and each test starts with a clean state.

// test/basicServerTest.js
import 'babel-polyfill';  
import {expect} from 'chai';  
import {mochaAsync} from './TestHelpers';  
import request from "supertest-as-promised";  
import makeServer from '../src/server';

describe('Basic server test', function() {  
  this.timeout(1000);
  var server;
  beforeEach(mochaAsync(async () => {
    server = await makeServer(true /* silent */);
  }));

  afterEach(function (done) {
    server.close(done);
  });

  it('Server healthy', mochaAsync(async () => {
    await request(server).get('/')
      .expect(200);
  }));
});

Then run the test.

> npm run test
npm run test

> Node_tutorial@1.0.0 test /Users/spchuang/github/GoChat-tutorial/6-things-to-add-to-your-node-projects
> NODE_ENV=test mocha --compilers js:babel-register



  Basic server test
    ✓ Server healthy (44ms)


  1 passing (212ms)

That's it!

Alright, that's it!

One last thing, I generally want to run flow, lint and tests together when I develop. This is the handy script I use:

// package.json
"scripts": {
  "check": "npm run lint && npm run flow && npm run test",
  ...
}

I hope you can try them out for your Node project because they have certain made my Node development a much smoother experience!

In the previous tutorial, I showed how to create the simplest possible Flask application - hello world. Unfortunately, the example application was too simple to show the full potential of the application structure, something I want to show in this post.

In this tutorial I will continue from the same code and develop a simple user authentication system. Users are the core to every modern-day application, and creating a user system is certainly the first step every develop takes on his or her journey to make the next world-changing application. You can start here.

To get a complete sense of the code base from before, please see the source code on Github.

Source code

You can download the source code on gibhub.

So what do I get in the end?

Good question. I will show how to create models with Flask-SQLAlchemy, do form validations using Flask-WTF, handling login sessions with Flask-Login and I would also show example usage of Flask-Script. There will also be code for the UI, but it will be quite minimal since this tutorial is mainly concerned with developing Flask. In the end, you'll be able to signup for a new user, login the user, refresh the page and you'll still be logged in, and then finally when you may logout, which will clear the user session from the server.

To be more specific, the following will happen during this tutorial:

• create a user table in the database (sqlite in this case). The table contains fields like first name, last name, email address, and user name and password. The password will be hashed for security reasons.
• create 2 WTForms, one for user logins and one for signup. They will be used for validating post data from logins and signup.
• create 4 authentication endpoints: /api/auth/verify_auth, /api/auth/login, /api/auth/logout, and /api/auth/signup.
• create a light UI in jquery to demonstrate the aforementioned authentication APIs.

Alright, that's too much talking and not enough coding. Lets get right to it.

Step 1. Install dependencies

Flask is a very flexible framework and offers plenty of different extensions for things like logging, form validation and database abstraction (ORM). In this tutorial, I'm going to show the use of 4 commonly-used Flask extensions:

• Flask-WTF : integration with WTForm, which offers useful features like form validation and CSRF handling to prevent XSS attacks.
• Flask-SQALAlchemy : Integration to SQLAlchemy, a very simple and powerful library for database abstraction.
• Flask-Script : Useful extension for creating external scripts in Flask.
• Flask-Login: provides user session management for Flask.

$ pip install Flask-Script
$ pip install Flask-SQLAlchemy
$ pip install Flask-WTF
$ pip install Flask-Login
$ pip freeze > requirements.txt

Step 2: Create the files

To create a user authentication system, we would create a a frontend module for serving the html pages and a user module (folder) for handling user validation and database access. Then we would add a another API auth.py for authentication endpoints like logging in and signing out. To make everything work together, we'll create a few helper functions along the way.

$ cd DoubleDibz
$ touch manage.py
$ mkdir app/users
$ mkdir app/static
$ mkdir app/frontend
$ touch app/frontend/__init__.py
$ touch app/frontend/controller.py
$ touch app/models.py
$ touch app/common/helpers.py
$ touch app/common/response.py
$ touch app/users/UserModels.py
$ touch app/users/UserConstants.py
$ touch app/users/UserForms.py
$ touch app/users/__init__.py
$ touch app/api/auth.py

Our current structure:

DoubleDibz  
├── app
│   ├── __init__.py
│   ├── api
│   │   ├── __init__.py
│   │   ├── auth.py
│   │   └── helloworld.py
│   ├── app.py
│   ├── common
│   │   ├── __init__.py
│   │   ├── constants.py
│   │   ├── helpers.py
│   │   └── response.py
│   ├── config.py
│   ├── extensions.py
│   ├── frontend
│   │   ├── __init__.py
│   │   └── controller.py
│   ├── models.py
│   ├── static
│   ├── templates
│   └── users
│       ├── UserConstants.py
│       ├── UserForms.py
│       ├── UserModels.py
│       └── __init__.py
├── manage.py
├── requirements.txt
└── run.py

I won't talk about every single line of code added or modified, such as app/common/helpers.py and app/common/response.py, but they are all crucial in the making of the system. So to get a sense of the bigger picture, take a look at the final code.

Step 3: Edit extensions.py and app.py

Initialize Flask-SQLAlchemy, Flask-WTF and Flask-Login extensions.

$ vim app/extensions.py

Place the following

# Flask-SQLAlchemy extension instance
from flask.ext.sqlalchemy import SQLAlchemy  
db = SQLAlchemy()

# Flask-Login
from flask.ext.login import LoginManager  
login_manager = LoginManager()

# Flask-WTF csrf protection
from flask_wtf.csrf import CsrfProtect  
csrf = CsrfProtect()  

These code would the extension objects. Next we would need to initialize the extension with the our Flask application.

$ vim app/app.py

Add this to top

from .models import User  
from .extensions import db, login_manager, csrf  

and add the following in the function configure_extensions()

def configure_extensions(app):  
   # flask-sqlalchemy
   db.init_app(app)

   # flask-login
   login_manager.login_view = 'frontend.index'
   login_manager.refresh_view = 'frontend.index'

   @login_manager.user_loader
   def load_user(id):
      return User.query.get(id)

   login_manager.setup_app(app)

   # flask-wtf
   csrf.init_app(app)

Step 4: create UserModels.py

Okay, now that we have initialized our extensions. We can start with the user model. In SQLAlchemy, each data model correspond one-to-one with a table in the database: User model will contain all the relevant information as a user and most importantly the credentials for logging in on the site.

from werkzeug import generate_password_hash, check_password_hash

from flask.ext.login import UserMixin  
from ..common.helpers import JsonSerializer, get_current_time  
from ..extensions import db

import UserConstants

class UserJsonSerializer(JsonSerializer):  
    __json_public__ = ['id', 'email', 'user_name']
    __json_modifiers__ = {
      'role_code' : ['role', (lambda code : UserConstants.USER_ROLE[code])]
    }

class User(db.Model, UserMixin, UserJsonSerializer):

   __tablename__ = "user"
   def __repr__(self):
      return '<User %r>' % (self.user_name)

   id            = db.Column(db.Integer, primary_key = True)
   first_name    = db.Column(db.String(UserConstants.STRING_LEN), nullable=False)
   last_name     = db.Column(db.String(UserConstants.STRING_LEN), nullable=False)
   user_name     = db.Column(db.String(UserConstants.STRING_LEN),  index = True, unique = True, nullable=False)
   email         = db.Column(db.String(UserConstants.STRING_LEN), index = True, unique = True, nullable=False)
   created_on    = db.Column(db.DateTime, nullable=False, default = get_current_time)
   role_code = db.Column(db.SmallInteger, default=UserConstants.USER, nullable=False)

   # User Password
   _password = db.Column('password', db.String(UserConstants.PW_STRING_LEN), nullable=False)

   def _get_password(self):
      return self._password

   def _set_password(self, password):
      self._password = generate_password_hash(password)

   password = db.synonym('_password',
                          descriptor=property(_get_password,
                                              _set_password))

   def check_password(self, password):
      if self.password is None:
         return False
      return check_password_hash(self.password, password)

   # methods
   @classmethod
   def authenticate(cls, user_name, password):
      user = User.query.filter(db.or_(User.user_name == user_name)).first()

      if user:
         authenticated = user.check_password(password)
      else:
         authenticated = False
      return user, authenticated

   @classmethod
   def is_user_name_taken(cls, user_name):
      return db.session.query(db.exists().where(User.user_name==user_name)).scalar()

   @classmethod
   def is_email_taken(cls, email_address):
      return db.session.query(db.exists().where(User.email==email_address)).scalar()

A couple things are worth talking about here.

• password field: notice how the password field is defined using SQLAlchemy's synonym and descriptor. password is essentially mirroring _password attribute and we use descriptor to define how getting password and setting password behaves, as defined in _get_password and _set_password respectively. When setting the password, instead of saving the user password string, we save the hashed password in the db using generate_password_hash.
• UserJsonSerializer mixin: inherited from JsonSerializerr (see app/common/helpers for definition). This handy mixin provides the to_json function which will deserialize an instance of the model to json format. Example (after setting up Flask-Script in step 5):

$ python manage.py shell
>>> user = models.User.query.all()[0];
>>> user.to_json()
{'role': 'user', 'user_name': u'spchuang', 'id': 1, 'email': u"test@gmail.com"}

• UserMixin Mixin: provides default implementation of what would be required for Flask-Login.

Step 5.1 : create the database with Flask-Script

Now that we have the user models, we can use it to create the actual database table. To do this, we're going to use Flask-script. This extension allows us to create handy scripts for things like initilizing the database and running a shell with the application context.

vim manager.py  

from flask.ext.script import Manager, Shell, Server  
from flask import current_app  
from app import create_app  
from app.extensions import db  
import app.models as Models  
from app.config import DefaultConfig  
import os

def create_my_app(config=DefaultConfig):  
  return create_app(config)

manager = Manager(create_my_app)

# runs Flask development server locally at port 5000
manager.add_command("runserver", Server(host="0.0.0.0", port=5000))

# start a Python shell with contexts of the Flask application
@manager.shell
def make_shell_context():  
   return dict(app=current_app, db=db, models=Models)

# init/reset database
@manager.command
def initdb():  
    db.drop_all(bind=None)
    db.create_all(bind=None)

    # add sample user
    user = Models.User(
            first_name=u'Sam',
            last_name=u'Chuang',
            user_name=u'spchuang',
            password=u'123456',
            email=u"test@gmail.com") 
    db.session.add(user)
    db.session.commit()


if __name__ == "__main__":  
    manager.run()

Now, by running the following command, you can initialize the database to create the User table and also an example user, which we'll use to test or login later.

$ python manage.py initdb

You should see a file being created in/tmp/test.db, as we configured SQLAlchemy to use a sqlite db at this path. To determine whether or not the user is actually created, you can take a use of Flask-Script's shell commands.

$ python manage.py shell
>>> users = models.User.query.all()
>>> print users
[<User u'spchuang'>]
>>> print users[0].password
pbkdf2:sha1:1000$x8kBSPNw$92b15d93720e001a1cf6343f2b589cff0b198024  

Notice how it shows the username of the user. This is defined in __repr__ in the User models object. Also, see how the password field is a hashed view. Besides using the manager shell, you can also examine db data using the sqlite3 interface:

$ sqlite3 /tmp/test.db 
SQLite version 3.8.3 2014-02-03 14:04:11  
Enter ".help" for instructions  
Enter SQL statements terminated with a ";"  
sqlite> select user_name,email from user;  
spchuang|test@gmail.com'  
sqlite> select * from sqlite_master where type='table';  
table|user|user|2|CREATE TABLE user (  
    id INTEGER NOT NULL, 
    first_name VARCHAR(64) NOT NULL, 
    last_name VARCHAR(64) NOT NULL, 
    user_name VARCHAR(64) NOT NULL, 
    email VARCHAR(64) NOT NULL, 
    created_on DATETIME NOT NULL, 
    password VARCHAR(80) NOT NULL, 
    role_code SMALLINT NOT NULL, 
    PRIMARY KEY (id)
)

Step 5.2: Testing the database and models

So far we have defined the User model and created the user table. Lets take a step back and do some testing to make sure everything is working correctly so far. It is good to break a meaty project into smaller, testable components. This allows us to establish baselines as we go.

$ python manage.py shell
>>> user, authenticated = models.User.authenticate('random_user', 'test')
>>> print user, authenticated
None False  
>>> user, authenticated = models.User.authenticate('spchuang', 'test')
>>> print user, authenticated
<User u'spchuang'> False  
>>> user, authenticated = models.User.authenticate('spchuang', '123456')
>>> print user, authenticated
<User u'spchuang'> True  
>>> print models.User.is_user_name_taken('spchuang')
True  
>>> print models.User.is_email_taken('test@gmail.com')
True  

Sweet, it seems like all the functions are working as we expected. As a general note, validating functions in shell is very handy and is a quick way to iterate on the code. But going forward, we'll probably want to add unit tests to cover those logics since it doesn't scale to hand test them everytime. Automated testings guarantees the code to work for all the test coverages and show whether new code has any regression (breaks test cases). Testing is outside the scope of this tutorial, but I'll probably find sometime to give an example how I could do testing in Flask.

Step 6: create UserForms.py

The next step is defining our form validation logic. Data checking is center to everything we do. Whenever we receive some data from the user. We want to make sure we have all the values needed to create a user and for logging in.

from flask_wtf import Form  
from wtforms import (BooleanField, TextField, HiddenField, PasswordField,  
   DateTimeField, validators, IntegerField, SubmitField)
import UserConstants

class LoginForm(Form):  
   login = TextField('user_name', [validators.Required()])
   password  = TextField('password',  [validators.Required()])
   remember_me = BooleanField('remember_me', default = False)

class SignupForm(Form):  
   user_name   = TextField('user_name',   [
      validators.Length(
         min = UserConstants.MIN_USERNAME_LEN, 
         max = UserConstants.MAX_USERNAME_LEN
      ),
      validators.Regexp(
         "^[a-zA-Z0-9]*$",
         message="Username can only contain letters and numbers"
      )
   ])
   first_name  = TextField('first_name', [validators.Required()])
   last_name   = TextField('last_name', [validators.Required()])
   email       = TextField('email', [validators.Required(), validators.Email()])
   password    = PasswordField(
      'New Password', 
      [validators.Length(min=UserConstants.MIN_PASSWORD_LEN,max=UserConstants.MAX_PASSWORD_LEN)]
   )
   confirm     = PasswordField('Repeat Password', [
      validators.Required(),
      validators.EqualTo('password', message='Passwords must match')
   ])

It is pretty straight forward. We define a form for login and another for signup. Flask-WTF allows us to define the fields and specific validations we want for each field. Some default validators are provided but we could also define our own. See the validation logic for username, in which I restrict it to only letters and number using regular expression.

Step 7 : create authentication endpoint api/auth.py

vim app/api/auth.py  

# Users API for authentication
'''  
   Users API for authentication
'''  
from flask import (Blueprint, render_template, current_app, request,  
                   flash, url_for, redirect, session, abort, jsonify)

from flask.ext.login import login_required, login_user, current_user, logout_user, confirm_login, login_fresh  
from ..common import Response  
from ..extensions import db

from ..users import User, SignupForm, LoginForm

auth = Blueprint('auth', __name__, url_prefix='/api/auth')

@auth.route('/verify_auth', methods=['GET'])
@login_required
def verify_auth():  
   return Response.make_data_resp(data=current_user.to_json())

@auth.route('/login', methods=['POST'])
def login():  
   """ POST only operation. check login form. Log user in """
   if current_user.is_authenticated():
      return Response.make_success_resp(msg="You are already logged in")

   form = LoginForm()
   if form.validate_on_submit():
      user, authenticated = User.authenticate(form.login.data,
                                              form.password.data)
      if user :
         if authenticated:
            login_user(user, remember=form.remember_me.data)
            return Response.make_data_resp(data=current_user.to_json(), msg="You have successfully logged in")
         else:
            return Response.make_error_resp(msg="Invalid username or password", type="Wrong Authentication", code=422)
      else:
         return Response.make_error_resp(msg="Username does not exist", type="Wrong Authentication", code=422)

   return Response.make_form_error_resp(form=form)


@auth.route('/logout', methods=['POST'])
@login_required
def logout():  
   """ logout user """
   session.pop('login', None)
   logout_user()
   return Response.make_success_resp(msg="You have successfully logged out")

@auth.route('/signup', methods=['POST'])
def signup():  
   if current_user.is_authenticated():
      return make_success_resp("You're already signed up")

   form = SignupForm()

   if form.validate_on_submit():
      # check if user_name or email is taken
      if User.is_user_name_taken(form.user_name.data):
         return Response.make_error_resp(msg="This username is already taken!", code=409)
      if User.is_email_taken(form.email.data):
         return Response.make_error_resp(msg="This email is already taken!", code=409)

      try:
         # create new user
         user = User()
         form.populate_obj(user)

         db.session.add(user)
         db.session.commit()

      except Exception as e:
         return Response.make_exception_resp(exception=e)

      # log the user in
      login_user(user)
      return Response.make_success_resp(msg="You successfully signed up! Please check your email for further verification.")

   return Response.make_form_error_resp(form=form)

Here we define 4 endpoints that we would use for authentication. You may wonder why do we need /api/auth/verify_auth. As I mentioned before, we are creating a pure API backend with frontend powered in javascript. Therefore, as the javascript application first start we need to determine whether the user is already logged in or not. This endpoint does just that - allows the frontend to bootstrap application state. You'll see a quick example in the UI section below.

The Form.validate_on_submit does what the name suggests - validate the input forms. It would make sure the every field passes the validation rules we defined for it and would fail otherwise.

By default, every POST request would have csrf protection on due to Flask-WTF settings. Cross-Site Request Forgery is unlike other hacking techniques where malicious users fake their credentials and behave on your behalf. Instead, hackers injects malicious scripts to trick your browser in doing an unwanted action on a website you're already logged in. In the website's perspective, the action is genuine and trust worthy since it came from you directly even though your browser did it unexpectedly. There are many common fixes for this - one of the which is CSRF protection. Flask-WTF will verify a hashed token for every POST request.

Update app.py to include the blueprint for our authentication API.

# in app/app.py
...
from .api import helloworld, auth  
...
DEFAULT_BLUEPRINTS = [  
   helloworld,
   auth,
]

Step 8: basic frontend

In this last part, I'll talk briefly on implementing a simple frontend that talks to the backend via jquery

First, Serve the static HTML with a new controller

$ vim app/frontend/controller.py

from flask import (Flask, Blueprint, render_template, current_app, request,  
                   flash, url_for, redirect, session, abort, jsonify, send_from_directory)

frontend = Blueprint('frontend', __name__)

@frontend.route('/')
@frontend.route('/<path:path>')
def index(path=None):  
   return render_template('app.html')

This controller is responsible for serving static pages, in this case, app/template/app.html.

Now Copy the app/templates and app/static folders from github. Open the file, and notice how we load two javascript files, jquery and our custom javascript file which will be used to communicate with the backend. We will use bootstrap for styles.

// app/templates/app.html
...

{% block js_btm %}
<!-- Put javascript here !-->

   <script src="/static/desktop/vendors/jquery.js"></script>
   <script src="/static/desktop/js/main.js"></script>

{% endblock %}

So our game plan is all in /static/desktop/js/main.js.

// setup csrf token for all ajax calls
var csrftoken = $('meta[name=csrf-token]').attr('content');  
$.ajaxSetup({
    beforeSend: function(xhr, settings) {
        if (!/^(GET|HEAD|OPTIONS|TRACE)$/i.test(settings.type)) {
         xhr.setRequestHeader("X-CSRFToken", csrftoken);
      }
    }
});

$(document).ready(function(){
   // initial check to see if user is logged in or not
   updateAuthStatus();



   // setup logged in view
   $('#logged-in-view button').click(function(){
      logout()
         .done(function(response){
            showLoggedout()
         }).fail(function(response){
            showLoggedIn();
         });
   });

   // setup signup view
   $('#signup-view #signup-form').submit(function(e) {
      e.preventDefault();
      var form = $(this);
      var signupData = extractFormInput(form);

      signup(signupData)
         .done(function(response){
            alert('You just created a new user');
            form.trigger('reset');
            updateAuthStatus();
         }).fail(function(response){
            alert('Something went wrong');
         });

   });
});

// helpers
function updateAuth() {  
   verifyAuth()
      .done(function(response){
         showLoggedIn(response.data.user_name)
      }).fail(function(response){
         showLoggedout()
      });
}
function extractFormInput(form) {  
   var inputs = form.serializeArray();
   var data = {};
   $.each(inputs, function(index, input) {
      data[input.name] = input.value;
   });
   return data;
}

function showLoggedIn(username) {  
   // show logged in view and show username
   $("#logged-in-view span").text(username);
   $("#logged-out-view").addClass('hidden');
   $("#logged-in-view").removeClass('hidden');
}

function showLoggedout() {  
   // show logged out view
   $("#logged-out-view").removeClass('hidden');
   $("#logged-in-view").addClass('hidden');
}

// API calls
function verifyAuth(callback) {  
   var url = '/api/auth/verify_auth';
   return $.get(url);
}

function login(loginData){  
   var url = '/api/auth/login';
   return $.post(url, loginData);
}

function logout() {  
   var url = '/api/auth/logout';
   return $.post(url);
}

function signup(signupData) {  
   var url = '/api/auth/signup';
   return $.post(url, signupData);
}

Login view

After logging in

Signup view

Thats it! Now you have created a simple user system in flask that could create new users and log existing users in. In a following post, I will work on the UI side of things and show example of using Backbone to power the user authentication.

In this tutorial, I will create a basic hello-world Flask application using the file structure from the previous tutorial. This application will simply create an api endpoint at api/helloworld/helloworld and return a message helloworld in json format.

Source code

You can download the source code on gibhub.

Setup the environment and install dependencies

Lets create the application folder first. For the purposes of this tutorial, I'll refer to our application as DoubleDibz.

$ cd [to where you want your application folder to be]
$ mkdir DoubleDibz

Before creating any more folders and files, we would need to setup the environment. Here I am assuming you guys have python, pip, and virtualenv installed already. If not, take a look here. It is generally a best practice to not install dependencies in the Python global packages, and instead install them in an isolated environment for each of your applications.

To create the virtual environment for DoubleDibz, do the following:

$ cd DoubleDibz
$ virtualenv .pyenv
New python executable in .pyenv/bin/python  
Installing setuptools, pip...done.  

Now the virtual environment resides in the .pyenv/ folder. Lets activate the environment!

$ source .pyenv/bin/activate

At this point, the name of the virtual environment (.pyenv) should appear on the left of your command prompt:

(.pyenv)UserName@Your-Computer:Some-folder/DoubleDibz $

From now ons, any package you install via pip will be installed locally in the .pyenv folder, separate from the global dependencies. Okay, lets go ahead and install what we need for this application.

$ pip install Flask
$ pip freeze > requirements.txt
$ cat requirements.txt
Flask==0.10.1  
Jinja2==2.8  
MarkupSafe==0.23  
Werkzeug==0.10.4  
itsdangerous==0.24  
wsgiref==0.1.2  

Now we have installed all the necessary dependencies for our application. You can see a list of all the packages and versions installed in the current environment with the command pip freeze. Notice how we save the output to requirements.txt. This allows you or other developers to recreate this same environment to ensure consistency in development and deployment (install dependencies on actual servers).

$ pip install -r requirements.txt

Remember to exclude the virtualenv (.pyenv or however you named it) from your version control by adding to the ignore list (.gitignore if youre using git).

Step 1: Create the file layout

mkdir app  
mkdir app/templates  
mkdir app/static  
mkdir app/frontend  
mkdir app/common  
touch run.py  
touch app/__init__.py  
touch app/config.py  
touch app/app.py  
touch app/extensions.py  
touch app/api/helloworld.py  
touch app/api/__init__.py  
touch app/common/constants.py  
touch app/common/__init__.py  

Our current structure:

DoubleDibz  
├── app
│   ├── __init__.py
│   ├── api 
│   │   ├── __init__.py
│   │   └── helloworld.py
│   ├── app.py
│   ├── common
│   │   ├── __init__.py
│   │   └── constants.py
│   ├── config.py
│   ├── extensions.py
│   ├── static
│   └── templates
└── run.py

Step 2: Edit app/app.py

Let's start first with the core of the application.

$ vim app/app.py

import os  
from flask import Flask

import config as Config  
from .common import constants as COMMON_CONSTANTS  
from .api import helloworld

# For import *
__all__ = ['create_app']

DEFAULT_BLUEPRINTS = [  
   helloworld
]

def create_app(config=None, app_name=None, blueprints=None):  
   """Create a Flask app."""

   if app_name is None:
     app_name = Config.DefaultConfig.PROJECT
   if blueprints is None:
     blueprints = DEFAULT_BLUEPRINTS

   app = Flask(app_name, instance_path=COMMON_CONSTANTS.INSTANCE_FOLDER_PATH, instance_relative_config=True)
   configure_app(app, config)
   configure_hook(app)
   configure_blueprints(app, blueprints)
   configure_extensions(app)
   configure_logging(app)
   configure_error_handlers(app)
   return app

def configure_app(app, config=None):  
   """Different ways of configurations."""

   # http://flask.pocoo.org/docs/api/#configuration
   app.config.from_object(Config.DefaultConfig)

   if config:
     app.config.from_object(config)
     return

   # get mode from os environment
   application_mode = os.getenv('APPLICATION_MODE', 'LOCAL')
   app.config.from_object(Config.get_config(application_mode))

def configure_extensions(app):  
   pass

def configure_blueprints(app, blueprints):  
   for blueprint in blueprints:
      app.register_blueprint(blueprint)

def configure_logging(app):  
    pass

def configure_hook(app):  
   @app.before_request
   def before_request():
      pass

def configure_error_handlers(app):  
   # example
   @app.errorhandler(500)
   def server_error_page(error):
      return "ERROR PAGE!"

app/app.py exposes the function create_app which sets up the Flask application with the desired configuration. The file abstracts out the process of configuring the Flask app in multiple stages:

• configure_app : initialize the Flask application with the correct configuration. I setup my application to have 3 modes of configs: local, staging, and producation (see app/config.py). This function will attempt to load the correct mode using either the passed-in config object or by identifying the mode from the environment variable (This is the most flexible for configuring staging and production servers).
• configure_blueprints: register the given list of Blueprint modules to the application. A common design pattern in Flask is factoring the application into multiple Blueprints modules to greatly simplify the complexity.
• configure_extensions: initializes Flask extensions for the application. We will see later how we use this function to initialize Flask-WTF, Flask-Script and Flask-SQLAlchemy.
• configure_logging : configure the way we want to log data. Depending on the environment, we would like to take different approach. For example, when testing locally, you might simply want to log data to stdout console, but for production servers you want to save in some log files.
• configure_hook: this is a bit less specific. In general, I use it to process requests and responses. For example, I could use it log every incoming requests or measure the time it takes for the server to generate a response for every request.

def configure_hook(app):  
   // log every incoming requests
   @app.before_request
      def before_request():
         app.logger.debug('Hitting %s' % request.url)

• configure_error_handlers: Sets up how we want to handle errors. For our purposes, we would return errors in API json format.

Step 3: Edit app/config.py

$ vim app/config.py

import os  
from common.constants import INSTANCE_FOLDER_PATH

class BaseConfig(object):

   PROJECT = "app"

   # Get app root path, also can use flask.root_path.
   PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))

   DEBUG = False
   TESTING = False

   ADMINS = ['youremail@yourdomain.com']

   # http://flask.pocoo.org/docs/quickstart/#sessions
   SECRET_KEY = 'secret key'

class DefaultConfig(BaseConfig):

   # Statement for enabling the development environment
   DEBUG = True

   # Secret key for signing cookies
   SECRET_KEY = 'development key'

class LocalConfig(DefaultConfig):  
   # config for local development
   pass

class StagingConfig(DefaultConfig):  
    # config for staging environment
    pass

class ProdConfig(DefaultConfig):  
    # config for production environment
    pass

def get_config(MODE):  
   SWITCH = {
      'LOCAL'     : LocalConfig,
      'STAGING'   : StagingConfig,
      'PRODUCTION': ProdConfig
   }
   return SWITCH[MODE]

The goal here is to have enough flexibility for different environment. As you can see, we have three configuration mode: LocalConfig, StagingConfig, and ProdConfig. All of which inherit from DefaultConfig which defines a set of standard settings such as project root directory and project name.

Step 4: Setup the helloworld endpoint

Now you have setup all the basics and the last thing you have to do is creating the API endpoint.

$ vim app/api/helloworld.py

"""
 Simple API endpoint for returning helloworld
"""
from flask import (Blueprint, render_template, current_app, request,  
                   flash, url_for, redirect, session, abort, jsonify, make_response)

helloworld = Blueprint('helloworld', __name__, url_prefix='/api/helloworld')

@helloworld.route('/helloworld', methods=['GET'])
def index():

   data = {
      'message' : "helloworld"
   }      
   return make_response(jsonify(data))

Step 5: Create __init__ files

For each folder we have, we would need to create a __init__.py file to mark the directory as a Python package. __init__.py is usually empty, but it can be used to expose selected portion of the package or rename them to more convenient/more specific names.

$ vim app/__init__.py
from app import create_app  

$ vim app/api/__init__.py
from .helloworld import helloworld  

This creates a api package that exposes the different API blueprints that we have without revealing the individual files and the implementation detail. This allows app.py to import the helloworld blueprint from the api/ directory level.

// in app.py
from .api import helloworld

// instead of loading from the specific api file
from .api.helloworld import helloworld  

Step 6: Edit constants.py

The app/common/ folder stores constants, functions and objects that are common for every modules we create (hence the name common). For now, we will only need a constants.py file.

$ vim app/common/constants.py

Place the following content that defines the directory path for the instance folder.

import os

# Instance folder path, make it independent.
INSTANCE_FOLDER_PATH = os.path.join('/tmp', 'instance')  

Step 7: Run and see!

When running the server locally, we'll use run.py

$ vim run.py

Place the contents:

import os  
from app import create_app

app = create_app()

if __name__ == '__main__':  
   port = int(os.environ.get("PORT", 5000))
   app.run(host='0.0.0.0', port=port, debug=True)

Running the file wil create a Flask server running at local port 5000.

$ python run.py
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat

Now you can hit your browser url : http://localhost:5000/api/helloworld/helloworld to see if the API endpoint is working properly. Alternatively, you can test with curl:

$ curl http://localhost:5000/api/helloworld/helloworld
{
  "message": "helloworld"
}

Yay, it works! Good job. Now you have successfully setup a basic Flask application with appropriate structure. Now lets get more advanced in the next section where we'll create a user authentication in Flask.

Flask is a minimalistic framework that doesn't provide an official way for organizing the application. In this post, I am going to show how we organize our DoubleDibz application directory . Our layout is based on fbone and there are other useful resources on this subject (1 , 2).

What makes DoubleDibz unique is that our backend is purely a API engine that powers the client. This simplifies the complexity in the backend by removing all the logic needed for templating. The backend is only concerned with receiving data, validating data, fetching new data from the db, performing some transformation, and finally returning the result to the client which will decide how to make use of it. If you are interested in creating a fully dynamic web client powered by Flask, read on!

Let's get started

As the home page shows, a hello-world Flask application is extremely simple.

// run.py
from flask import Flask  
app = Flask(__name__)

@app.route("/")
def hello():  
    return "Hello World!"

if __name__ == "__main__":  
    app.run()

This code is all nice and dandy but is only appropriate for a trivial application complicated enough to fit in one file. For a large Flask application, we need a more sophisticated structure for the growing complexity. We need to manage more complicated routing for various endpoints, being able to configure under different environments (staging vs. prod), abstracting out common logic, etc. Organizing the application in smaller modules goes a long way. It is like writing clean and modular code: both are simpler to understand and easier for other to maintain and extend.

Application layout

our Flask application is based on fbone with some tweaks. The biggest difference here is that the backend only serves as an API layer and has very minimal usage of templating. This is a generic look on our file layout:

├── app
│   ├── __init__.py
│   ├── api      
│   │   ├── __init__.py
│   │   ├── auth.py
│   │   └── ...
│   ├── app.py
│   ├── common
│   │   ├── __init__.py
│   │   ├── constants.py
│   │   ├── helpers.py
│   │   ├── response.py
│   │   └── ...
│   ├── config.py
│   ├── extensions.py
│   ├── frontend
│   │   ├── __init__.py
│   │   └── controllers.py
│   ├── static
│   │   └── ...
│   ├── templates
│   │   ├── app.html
│   │   └── ...
│   └── users
│       ├── UserConstants.py
│       ├── UserForms.py
│       ├── UserHelpers.py
│       ├── UserModels.py
│       └── __init__.py
├── alembic
|   ├── version
│   └── ...
├── tests
│   └── ...
├── fabfile.py
├── manage.py
├── requirements.txt
└── run.py

This may seem like a complicated mess. Lets take a look and see what each folder (or file) is used for.

app/

First and foremost, all the application logic lives under the app/ folder.

• app/app.py is the main entry point of the Flask application.
• app/config.py will be storing all the module configuration for different environments (local, staging and prod). You don't want to be messing with production data while developing locally! I'll talk more about how to setup different environment in later posts.
• app/extensions.py contains declaration of all the flask extensions
• app/static/ is the folder from which Flask will serve static files. If you want to use a different folder, see here
• app/common/ consists of common classes and functions that are useful for all the modules. Some examples are global constants, global helper functions.
• app/api/ contains all the controllers for every API endpoints, such as authentication (login/signup).
• app/frontend: as opposed to the API folder, the controllers here are serving non API endpoints. For example, it may define entry point to load the HTML file for your fully dynamic javascript application built in backbone, angular, react, or whatever fancy framework you indulge on.
• app/[module]/ represents a functional area in the application, such as users, posts, chat, and search.
  • app/[module]/[moduleModels] : database models (tables) declaration with Flask-SQLAlchemy
  • app/[module]/[moduleForms]: WTForm definition in Flask-WTF. Useful for form validation.
  • app/[module]/[moduleConstants] and app/[module]/[moduleHelpers] defines the necessary helper functions and constants for this module.

And stuff outside of app/?

The file outside of the application folder shouldn't contain any core business logic for your applications. Those files and scripts are generally for configuring and managing the application. Here I'm listing the scripts and libraries that I use for DoubleDibz.

• requirements.txt contains all the python dependencies for this application. Install using pip.

pip install -r requirements.txt`  

• manage.py contains useful scripts (run, initdb, testing, list routes, etc) using Flask-Script
• run.py will be used to launch the web server locally.
• fabfile.py: defining shell operations using Fabric (which is amazing). You can define command-line scripts for remote deployment (though it's generally not a good practice if you have multiple servers).
• tests/ : how can any application be created without any testing? With Flask-Testing, it is simple to write unit tests for models or integration tests for the different API endpoints.
• alembic/ : folder containing database migration information for SQLAlchemy using Alembic.

We are organizing the code based on different functional areas. Say you want to build a user authentication system in your new Uber for food website, you would group the models, forms, constants, and helper functions logic under the same folder. This structure makes it very clear that all the files under users/ are related for the same purpose. Understanding how authentication and signup works is as easy as browsing the files in this directory. Furthermore, you can expose certain objects and functions from this folder for the consumptions of other modules.

You may have noticed how we add a prefix User-' to the files in the users module, as opposed to simply using models or forms names. This was done deliberately to add more clarity to the meaning of the file just from the file name. Imagine you have a couple different modules, and you open some of the models file in your editor. It is unclear whether a how a models file is different from another models file without examining the code or the file path. We tried our best to set up a naming convention for files, functions and classes in our code in order to provide the most clarity even if it seems cumbersome.

And finally, I mentioned a bunch of Flask extensions and libraries earlier. For clarity sake, this is a list of them:

• Flask-Script
• Flask-SQLAlchemy
• Flask-WTF
• Flask-Testing
• Fabric
• Alembic

Though it's outside the scope of this post, I also want to list a couple other extensions and libraries that made my life so much easier.

• Flask-Admin
• Flask-Login
• Flask-Mail
• redis: in-memory store. Very useful for caching purposes.
• Celery: setting up async jobs
• Pillow: PIL fork. useful for image manipulation like resizing and cropping.
• boto: python interface to Amazon webservice. Useful for uploading files in S3.
• facebook-sdk : Facebook Platform Python SDK

Final note on styling and consistency

I think one of the most important thing about a code base is consistency. Sometimes, it's more important to be consistent with the rest of the code base than to do what is considered the right convention. Generally, this is what I follow:

• 2 space indent (no tab)
• prefer longer names for the sake of clarity and readability (UserModels over models)
• single quotation 'string' for string
• Naming convention: module_name, ClassName, method_name, function_name, CONSTANT_NAME, local_variable_name

Of course, I don't always follow my own convention, especially when I'm the only one pushing code. Even so, following some set of rules would guarantee some level of quality and makes it easier collaborating with others on the same code base. But most importantly, consistency is king. it's better to stick with whatever is already there than creating a hot mess.

Why did we created DoubleDibz

DoubleDibz was our attempt to catch on the trend of the rapidly growing usage of college for-sale groups on Facebook. This was before Facebook introduced the special "selling post". At the time, for sale groups were no different from any other groups, where members can post and others can comment; yet thousands of students started posting about selling textbooks, electronics, event tickets, used furnitures, and plenty of other types of items. It was extraordinary - more than half of the UCLA students are buying and selling on Facebook. I bet not even this ratio of students use Twitter.

As a frequent visitor of the group, I have witnessed the beginning of this trend and how fast it has grown. It's incredible how the market created its own fit on a platform not originally intended for this purpose. In my opinion, besides the fact that everyone already on Facebook and it has the best social features, the biggest driver for the trend is the sense of security and trust instituted in private Facebook groups. Unlike craiglist, here you feel comfortable buying, selling and meeting up with another student from the same school. This makes Facebook well positioned for this market.

At the time, a couple friends (Sean and Justin) and I really wanted to try doing a startup; we were half serious of course. We met once a week brainstorming ideas with the goal of applying to UCLA's summer startup accelerator. The exact plan wasn't very clear as Justin and I both have an internship lined up for the summer. We agreed that if we got in the accelerator we would push back our internship. It was a very thrilling plan. Unfortunately, the idea we came up with at the time didn't make the shot. I guess our pitch wasn't as enticing as we thought. In light of the news, we still decided to continue, and we came up with the idea of DoubleDibz (well, at the time it was called Palxchange).

It made sense. For sale groups are growing like crazy on Facebook, but the tools are basic and there are tons of problems, such as noise, spam, poor search functionality, unintuitive layout for browsing, lack of categories, and this list can go on. So why not build a private social network for buying and selling and we'll address all of these shortcomings Facebook has. The idea really made a lot of sense. Although we didn't think of a good way to monetize besides using the good old ads model but we thought we had a good chance to build our user base by providing a better buying and selling experience than Facebook.

It was a TON of work. Only Justin and I could code and at this early stage neither Sean nor Claudia (who joined later and helped us with design and advertising strategies) could really help. We knew that a MVP had to be created first before we can start iterating from users' feedbacks. We spent a good month building out the first MVP, which has the essential Facebook features - create selling posts, message sellers, and notifications. The backend was created in Flask and frontend was fully dynamic and built in Backbone. later we also added search functionality and categorization. These were the only screen shots I had from the time:

The supply and demand

Then we faced the chicken and egg problem. Sure, our product seemed to work fine and have addressed a couple problems we identified from Facebook. But now what? How do we get sellers to come posting when we don't have buyers using it. And why would potential buyers come to our site when nothing is being sold? I suppose we could start by selling random crap that I had in my apartment (it's funny how this was what Jack Ma did when he first started Taobao). We looked at Airbnb's growth hack in its early days to solve its supply and demand problem. By piggybacking on Craiglist, Airbnb was able to get a lot of traction really early. This was a no brainer for us. We'll piggyback from existing Facebook for sale groups.

This was why we created a service we called FBsync. Every 10 seconds, it pulls selling posts from different for sale group within the UCLA community (e.g. textbook, housing, clothing, etc) and this allows us to bootstrap from the existing posts on Facebook and present them with better browsing experience. It was our hope to solve the supply problem. In terms of the demand, we started deeper integration with Facebook. For example, if a selling post is created on our platform, we made it very easy to "share" the post back to specific Facebook for sale group.

One of the interesting integrations we experimented with is tagging Facebook friends on our platform, and sending those friends notification on Facebook about the item. Facebook is really good at connecting people, and we noticed a behavior that when you see something that you think your friends would be interested, tagging your friend is a no brainer. This is different from sharing on your friends wall or sending them messages, since this action is low friction and makes tagging multiple people easily. We wanted to replicate this experience on Doubledibz. It was kind of a hack since the link of the notification cannot bring the user offsite, so we had to create a canvas page of DoubleDibz on Facebook for viewing products from tagged notifications. The flow looks something like this:

The dawn

We tried to create everything we thought was necessary for a good buyer and seller experience on our platform. However, at the end, we couldn't take over Facebook's popularity. We faced a lot of friction trying to redirect people in joining a new platform. Even though Facebook wasn't intended for online C2C buying and selling, its default social features - private messaging, groups, posts, comments, and likes - already create a very intuitive and natural environment for this market. The last straw was knowing that Facebook is improving for sale groups. And we knew that Facebook groups would only become better, with more tools developed for specific types of collaboration. We finally decided to pull the trigger and call it the end. The convenience of Facebook simply outweighs most if not all the benefit of our standalone platform.

Now that I'm writing this, I feel quite sad. But looking back, it was an amazing experience - to be able to fully dedicate myself on an idea and working closely with Justin, Claudia, and Sean in a team dynamic was a great learning experience. I'm writing to share with you guys all the things that I have learned along the way. Since neither Justin and I had significant backend and frontend experience, we had to learn so much in order to create a website from scratch. What we have learned are likely very useful to anyone who is thinking of creating his or her own website. So here I am. I hope these tutorials could significantly reduce the amount of learning overhead so you can focus on what's really important for your own application.

Honestly, I'm not really a writer. Certainly not a clever one. I mean as a programmer, I know how to write beautify code but when it comes to real writing, the most I've ever done was for general education classes back in college. And yeah, it was a joke.

That's why I want to start this blog, somewhere i can practice writing. I'll be writing freely. I would write about the things I think about, things that inspire me (and hopefully that could inspire you), problems I recognize around me, and ideas that I have came up with for those issues. I don't necessarily intend on showing this to the entire world. Though it is public, it feels almost like writing in a diary. Hopefully, many years later, when I look at this post again, I would realize how much I have improved in writing.

Besides practice writing, a big motivation for this blog is to create my own platform for documenting my technical thoughts, ideas and work. This is technically a tech blog. I'm sure you know this unfortunate feeling - how, looking back, so many things you've done or have thought of disappeared in time, leaving no trace and meaning. I want to preserve them, taking snapshots of whatever I have done and I want to share them with anyone whom would potentially find them useful.

Right now, there are already a couple things that come to my mind that I want to write about - my past hackathon projects, freelancing projects, and random hacks that I enjoyed doing. The biggest among them has to be DoubleDibz, a social network for connecting buyers and sellers in private communities (Yes, similar to Facebook's for sale groups).

Doubledibz is by far the biggest project that I undertook and I have learned the most from. For the next couple weeks months, I want to write about our technical journey in making this application. I want to share with you guys all the twist and turns, and steps that we have taken to make a semi serious web application. Topics I have in mind include but not limited to server configuration, nginx setup, server scaling, db migration, setting up async batch jobs, and creating a beautifully dynamic frontend app. I haven't touched the code in almost 8 months (which means I don't remember a thing...), but I intend to go back to the code base and figure out anything valuable that I can leave behind to the community.

Stay tuned.