Contribute to django-calaccess-downloads-website

This walkthrough is for developers who want to contribute to django-calaccess-downloads-website, a open-source archive of campaign-finance and lobbying-disclosure data from the California Secretary of State’s CAL-ACCESS database.

It will show you how to install the source code of this application to fix bugs, develop new features and deploy an archive to the Internet using Amazon Web Services.

Preparing a development environment

In order to contribute you first need to set up a local development environment by installing the source code and configuring a few settings.

While not required, we recommend that development be done within a contained virtual environment.

One way to accomplish that is with a two related Python packages: virtualenv and virtualenvwrapper. If you have both of these installed, a new project can be easily created like so:

$ mkproject django-calaccess-downloads-website

That will jump into a new folder in your code directory, where you can fork our code repository from GitHub. Don’t know what that means? Read this.

Once you’ve created a fork, you should clone it to your computer.

$ git clone<YOUR-USERNAME>/django-calaccess-downloads-website.git .

Next, install the other Python libraries our code depends on, like the Django web framework.

$ pip install -r requirements.txt

Configuring settings

Many of the settings in this project can vary depending on where the code is being run. For instance, your local installation of the code will likely connect to a different database than the public website.

To keep these different environments straight and avoid including sensitive passwords in public repositories we have developed a system for storing many of the configuration options in a file named .env at the project’s root.

The file is excluded from Git’s version control system and needs to be created fresh each time the code is installed.

How .env works

The .env file is expected to contain a separate section for each environment, using the structure favored by Python’s ConfigParser module. Here’s a simple example:



By default, the source code will draw settings from a section name DEV. To configure it to use a different set of variables (like the``PROD`` section above), you must set the CALACCESS_WEBSITE_ENV environment variable.


If you are using virtualenv and virtualenvwrapper, you could add the above line of code to $VIRTUAL_ENV/bin/postactivate so that whenever you start the project’s virtual environment, the variable will be exported automatically.


You could also add the following line to your $VIRTUAL_ENV/bin/postdeactivate script to clear the variable whenever you deactivate the virtual environment:


Connecting to a local database

Unlike a typical Django project, this application only supports PostgreSQL version 9.6 and above as a database backend. This is because we enlist specialized tools to load the immense amount of source data more quickly than Django typically allows.

Create the database the PostgreSQL way.

$ createdb calaccess_website -U postgres

Creating an archive on Amazon S3

Even a development project that will run only on your computer needs an account with Amazon Web Services to store archived files in its S3 file service.

If you don’t already have an AWS account, make one now and request a key pair that lets you access its services via Python.

Then create a new S3 “bucket” to store files archived by this project.

Filling in .env for the first time

The development environment can be created in the .env file by running a Fabric task that will ask you to provide a value for all of this project’s settings.

$ fab createconfig

You will prompted to provide the project’s full list of settings, though some of them are only necessary when deploying the code and site with Amazon Web Services.

Setting Required in development Definition
db_name Yes Name of your database.
db_user Yes Database user.
db_password Yes Database password.
db_host Yes Database host location.
aws_access_key_id Yes Shorter secret key for accessing Amazon Web Services.
aws_secret_access_key Yes The longer secret key for accessing Amazon Web Services.
aws_region_name Yes Amazon Web Services region where you resources are located.
s3_archived_data_bucket Yes Amazon S3 bucket where archived CAL-ACCESS data will be stored.
s3_baked_content_bucket No Amazon S3 bucket where the public-facing website will be stored.
key_name No Name of the SSH .pem file associated with Amazon Web Services. Should be found in ~/.ec2.
ec2_host No Public address of website’s Amazon EC2 instance.
email_user No Gmail account for sending error emails.
email_password No Gmail password for sending error emails.

If necessary, you can overwrite a specific setting or append a new one:

$ fab setconfig:key=<new-variable-name>,value=<some-value>

You can also print your current app environment’s configuration:

$ fab printconfig

Or everything in the Fabric environment:

$ fab printenv

Bootstrapping the project

Now that everything is configured, create the database tables.

$ python migrate

Once everything is set up, the updatedownloadswebsite command will download the latest bulk data release from the Secretary of State’s website load it into your local database and archive the files on Amazon S3.

$ python updatedownloadswebsite


This will take a while. Go grab some coffee.

Exploring the site

Finally, start the development server and visit localhost:8000/admin/ in your browser to inspect the site.

$ python runserver

Preparing a production server

This section will walk you through deploy the downloads website on the Internet via Amazon Web Services. You will need to have completed the steps above.

Change your environment

As described above, the source code will draw settings from a section of the .env file named DEV.

To switch to configuring your project for a production environment, you should set the CALACCESS_WEBSITE_ENV environment variable to PROD.


If you are using virtualenv and virtualenvwrapper, you could add the above line of code to $VIRTUAL_ENV/bin/postactivate so that whenever you start the project’s virtual environment, this variable will be exported automatically whenever you use workon to begin work.

Creating an RDS database

You will need to create a hosted database to store the data and keep tabs on the archive over time. Our recommended method for doing this is using Amazon’s Relational Database Service.

You can spin up a PostgreSQL server there using our prepackaged Fabric commands. You’re only required to provide a name like download-website:

$ fab createrds:download-website

Then, wait several minutes while the server is provisioned.

By default, the new database server will have 100 GB of disk space allocated on a t2.large RDS class instance. If need be, you can override these settings:

$ fab createrds:download-website,block_gb_size=80,instance_type=db.m4.large

The address for the RDS host will automatically be added to the configuration for your current environment, which is stored in .env. If you already had an RDS host set for your current env, its address will be overwritten.

Create an EC2 Instance

Next you should create a new Ubuntu 14.04 server on Amazon’s Elastic Compute Cloud to host the Django project.

$ fab createec2

By default, the server will have 100 GB of disk space allocated on a c3.large class instance. If need be, you can override these settings:

$ fab createec2:block_gb_size=80,instance_type=c3.xlarge

You can also override our default Amazon Machine Image (AMI):

$ fab createec2:ami=<some-other-ami-id>

As with creating an RDS instance, the address for your new EC2 instance will automatically be added to the configuration for your current environment, which is stored in .env. If you already had an EC2 host set, its address will be overwritten.

Filling in .env for the second time

Now you’ll want to run our configuration command again, this time filling in the new details from your AWS account, database and server. You may want to create a new set of S3 buckets separate from your development buckets.

$ fab createconfig

Bootstrap the Django project

Finally, you’re ready to bootstrap the Django project on the Ubuntu server.

$ fab bootstrap

After connecting to your current EC2 instance, a framework called Chef and its dependencies, including Ruby, will be installed on the server. Chef is used to configure the server and install the downloads website’s code.

The bootstrap task also sets up a crontab job to execute run as command every six hours that will automate the collection, extraction and processing of the daily CAL-ACCESS database exports.

Wrapping up

And that’s it! You know have a live CAL-ACCESS archive running in the cloud.