Data Science Demos

Extract, transform, and load census data with Python

2016-02-08T03:50:00+00:00

Contents

Overview
Motivations
Example
Helpful links
Footnotes

Overview

The Census Bureau collects data from people in the United States through multiple survey programs. Federal, state, and local governments use the data to assess how constituents are represented and to allocate spending. The data are also made freely available to the public and have a wide range of uses.¹ In this post, I parse, load, and verify data from the Census Bureau's American Community Survey (ACS) 2013 5-year Public Use Microdata Sample (PUMS) for Washington DC.

Brief process:

Start with "Running an IPython Notebook on Google Compute Engine from Chrome".
- For additional storage, create and mount a disk to the instance.²
Download (and decompress):³ ⁴
- 2013 5-year PUMS data dictionary: PUMS_Data_Dictionary_2009-2013.txt (<1 MB)
- 2013 5-year PUMS person and housing records for Washington DC:
  - Person records: csv_pdc.zip (5 MB compressed, 30 MB decompressed)
  - Housing records: csv_hdc.zip (2 MB compressed, 13 MB decompressed)
- 2013 5-year PUMS estimates for user verification: pums_estimates_9_13.csv (<1 MB)
Load the files:
- Data dictionary TXT: dsdemos.census.parse_pumsdatadict (see dsdemos package below)
  This is a customized parser I wrote for PUMS_Data_Dictionary_2009-2013.txt. The data dictionary is inconsistently formatted, which complicates parsing.⁵ ⁶
- Person/housing records and user verification CSVs: pandas.read_csv ⁷ ⁸
Confirm the user verification estimates (see example below):⁹
- To calculate an estimate $X$ for a specific "characteristic" (e.g. "Age 25-34"), sum the column '[P]WGTP' of the filtered data ('PWGTP' for person records, 'WGTP' for housing records).¹⁰ '[P]WGTP' are the sample weights.
- To calculate the estimate's "direct standard error", use the ACS's modified root-mean-square deviation:
  
  $$\mathrm{SE}(X) = \sqrt{\frac{4}{80}\sum_{r=1}^{80}(X_r-X)^2}$$
  
  where each $X_r$ is the sum of the column '[P]WGTPr' of the filtered data. '[P]WGTP[1-80]' are the "replicate weights".
- To calculate the estimate's margin of error (defined by ACS at the 90% confidence level):
  
  $$\mathrm{MOE}(X) = 1.645\,\mathrm{SE}(X)$$

Source code:

For step-by-step, see the Jupyter Notebook (click the HTML export to render in-browser):
20160110-etl-census-with-python.ipynb
20160110-etl-census-with-python-full.html
This post uses dsdemos v0.0.3.¹¹

Motivations

Why am I using the American Community Survey (ACS)?

The ACS is a relevant data set. A future step is to predict an individual's household income, which is among the subjects that the ACS survey addresses.
The ACS is a reliable data set.¹² The ACS has quality controls to ensure that it is representative. The survey samples about 3 million addresses per year with a response rate of about 97%.
The ACS is a time-series data set. The survey sends questionnaires throughout the year and releases data once per year. A future step is to use the time series to forecast an individual's household income.
I recognize that using ACS data can be problematic. Data from the Census Bureau has been used for harm,¹³ and current ACS terminology asks respondents to identify by terms such as "race".¹⁴ For this project, I take data from the Census Bureau at face value, and I infer from it at face value. It's important to respect that these aren't simply data points; these are people.

Why am I using the ACS 5-year estimate?

As of Dec 2015, the ACS offers two windowing options for their data releases: 1-year estimates and 5-year estimates.¹⁵ Because the ACS 5-year estimates include data over a 5-year window, they have the largest sample size and thus the highest precision for modeling small populations. However, the sample size comes at the expense of currency. Forecasting the predictions from a 5-year window to be more relevant to a specific year is a future step.

Why am I using Python?

This project can be done using Python, R, SQL, and/or other languages.¹⁶ I'm using Python since a future step is to make a machine learning pipeline, which is a popular application of scikit-learn.

What about "big data"?

I'm starting with a data set small enough to be processed in memory (i.e. operated on in RAM), since the focus of many Python packages is in-memory operations on single machines.¹⁷ These packages often parallelize operations across the machine's processor cores. For operations that exceed the machine's available RAM (i.e. out-of-core computations), there's Dask for Python, and for operations that require a cluster of machines, there's Spark for Java, Scala, Python, and R. Scaling a pipeline to a large enough data set that requires a cluster is a future step.

Example

This is an abbreviated example of my ETL procedure in the Jupyter Notebook for this post (see links to source code above).

In [1]:

cd ~

/home/samuel_harrold

In [2]:

# Import standard packages.
import os
import subprocess
import sys
# Import installed packages.
import numpy as np
import pandas as pd
# Import local packages.
# Insert current directory into module search path.
# dsdemos version: https://github.com/stharrold/dsdemos/releases/tag/v0.0.3
sys.path.insert(0, os.path.join(os.path.curdir, r'dsdemos'))
import dsdemos as dsd

In [3]:

# File paths
# Base path to ACS:
#     http://www2.census.gov/programs-surveys/acs/
# 2013 5-year PUMS data dictionary:
#     tech_docs/pums/data_dict/PUMS_Data_Dictionary_2009-2013.txt
# 2013 5-year PUMS housing records for Washington DC (extracted from csv_pdc.zip):
#     data/pums/2013/5-Year/csv_pdc.zip
# 2013 5-year PUMS user verification estimates:
#     tech_docs/pums/estimates/pums_estimates_9_13.csv
path_acs = r'/mnt/disk-20151227t211000z/www2-census-gov/programs-surveys/acs/'
path_dtxt = os.path.join(path_acs, r'tech_docs/pums/data_dict/PUMS_Data_Dictionary_2009-2013.txt')
path_hcsv = os.path.join(path_acs, r'data/pums/2013/5-Year/ss13hdc.csv')
path_ecsv = os.path.join(path_acs, r'tech_docs/pums/estimates/pums_estimates_9_13.csv')

In [4]:

# Load and display the data dictionary.
ddict = dsd.census.parse_pumsdatadict(path=path_dtxt)
print("ddict, dfd: Convert the nested dict into a hierarchical data frame.")
tmp = dict() # tmp is a throwaway variable
for record_type in ddict['record_types']:
    tmp[record_type] = pd.DataFrame.from_dict(ddict['record_types'][record_type], orient='index')
dfd = pd.concat(tmp, names=['record_type', 'var_name'])
dfd.head()

`ddict`, `dfd`: Convert the nested `dict` into a hierarchical data frame.

Out[4]:

		length	description	var_codes	notes
record_type	var_name
HOUSING RECORD	ACR	1	Lot size	{'b': 'N/A (GQ/not a one-family house or mobil...	NaN
	ADJHSG	7	Adjustment factor for housing dollar amounts (...	{'1086032': '2009 factor', '1068395': '2010 fa...	[Note: The values of ADJHSG inflation-adjusts ...
	ADJINC	7	Adjustment factor for income and earnings doll...	{'1085467': '2009 factor (0.999480 * 1.0860317...	[Note: The values of ADJINC inflation-adjusts ...
	AGS	1	Sales of Agriculture Products (Yearly sales)	{'b': 'N/A (GQ/vacant/not a one-family house o...	[Note: No adjustment factor is applied to AGS.]
	BATH	1	Bathtub or shower	{'b': 'N/A (GQ)', '1': 'Yes', '2': 'No'}	NaN

In [5]:

# Load and display the housing records.
print("dfh: First 5 housing records and first 10 columns.")
dfh = pd.read_csv(path_hcsv)
dfh.iloc[:, :10].head()

`dfh`: First 5 housing records and first 10 columns.

Out[5]:

	insp	RT	SERIALNO	DIVISION	PUMA00	PUMA10	REGION	ST	ADJHSG	ADJINC
0	600	H	2009000000403	5	102	-9	3	11	1086032	1085467
1	NaN	H	2009000001113	5	103	-9	3	11	1086032	1085467
2	480	H	2009000001978	5	103	-9	3	11	1086032	1085467
3	NaN	H	2009000002250	5	105	-9	3	11	1086032	1085467
4	2500	H	2009000002985	5	101	-9	3	11	1086032	1085467

In [6]:

# Load and display the verification estimates.
# Select the estimates for Washington DC then for the
# characteristic 'Owner occupied units (TEN in 1,2)'.
dfe = pd.read_csv(path_ecsv)
tfmask_dc = dfe['state'] == 'District of Columbia'
dfe_dc = dfe.loc[tfmask_dc]
print("dfe_dc: This example verifies these quantities.")
dfe_dc.loc[[310]]

`dfe_dc`: This example verifies these quantities.

Out[6]:

	st	state	characteristic	pums_est_09_to_13	pums_se_09_to_13	pums_moe_09_to_13
310	11	District of Columbia	Owner occupied units (TEN in 1,2)	110,362	1363	2242

In [7]:

# Verify the estimates following
# https://www.census.gov/programs-surveys/acs/
#     technical-documentation/pums/documentation.2013.html
#     tech_docs/pums/accuracy/2009_2013AccuracyPUMS.pdf
# Define the column names for the housing weights.
# Select the reference verification data for the characteristic,
# and select the records for the characteristic.
hwt = 'WGTP'
hwts = [hwt+str(inum) for inum in range(1, 81)] # ['WGTP1', ..., 'WGTP80']
char = 'Owner occupied units (TEN in 1,2)'
print("'{char}'".format(char=char))
tfmask_ref = dfe_dc['characteristic'] == char
tfmask_test = np.logical_or(dfh['TEN'] == 1, dfh['TEN'] == 2)
# Calculate and verify the estimate ('est') for the characteristic.
# The estimate is the sum of the sample weights 'WGTP'.
col = 'pums_est_09_to_13'
print("    '{col}':".format(col=col), end=' ')
ref_est = int(dfe_dc.loc[tfmask_ref, col].values[0].replace(',', ''))
test_est = dfh.loc[tfmask_test, hwt].sum()
assert np.isclose(ref_est, test_est, rtol=0, atol=1)
print("(ref, test) = {tup}".format(tup=(ref_est, test_est)))
# Calculate and verify the "direct standard error" ('se') of the estimate.
# The direct standard error is a modified root-mean-square deviation
# using the "replicate weights" 'WGTP[1-80]'.
col = 'pums_se_09_to_13'
print("    '{col}' :".format(col=col), end=' ')
ref_se = dfe_dc.loc[tfmask_ref, col].values[0]
test_se = ((4/80)*((dfh.loc[tfmask_test, hwts].sum() - test_est)2).sum())0.5
assert np.isclose(ref_se, test_se, rtol=0, atol=1)
print("(ref, test) = {tup}".format(tup=(ref_se, test_se)))
# Calculate and verify the margin of error ('moe') at the
# 90% confidence level (+/- 1.645 standard errors).
col = 'pums_moe_09_to_13'
print("    '{col}':".format(col=col), end=' ')
ref_moe = dfe_dc.loc[tfmask_ref, col].values[0]
test_moe = 1.645*test_se
assert np.isclose(ref_moe, test_moe, rtol=0, atol=1)
print("(ref, test) = {tup}".format(tup=(ref_moe, test_moe)))

'Owner occupied units (TEN in 1,2)'
    'pums_est_09_to_13': (ref, test) = (110362, 110362)
    'pums_se_09_to_13' : (ref, test) = (1363, 1363.1910174293257)
    'pums_moe_09_to_13': (ref, test) = (2242, 2242.449223671241)

In [8]:

# Export ipynb to html
path_static = os.path.join(os.path.expanduser(r'~'), r'stharrold.github.io/content/static')
basename = r'20160110-etl-census-with-python'
filename = r'example'
path_ipynb = os.path.join(path_static, basename, filename+'.ipynb')
for template in ['basic', 'full']:
    path_html = os.path.splitext(path_ipynb)[0]+'-'+template+'.html'
    cmd = ['jupyter', 'nbconvert', '--to', 'html', '--template', template, path_ipynb, '--output', path_html]
    print(' '.join(cmd))
    subprocess.run(args=cmd, check=True)
    print()

jupyter nbconvert --to html --template basic /home/samuel_harrold/stharrold.github.io/content/static/20160110-etl-census-with-python/example.ipynb --output /home/samuel_harrold/stharrold.github.io/content/static/20160110-etl-census-with-python/example-basic.html

jupyter nbconvert --to html --template full /home/samuel_harrold/stharrold.github.io/content/static/20160110-etl-census-with-python/example.ipynb --output /home/samuel_harrold/stharrold.github.io/content/static/20160110-etl-census-with-python/example-full.html

Helpful links

Some links I found helpful for this blog post:

American Community Survey (ACS):
- ACS About.
- ACS Guidance for Data Users describes how to get started with ACS data.
- ACS Technical Documentation links to resources for learning how to work with ACS data.
- ACS Methodology includes design details, sample sizes, coverage estimates, and past questionnaires.
- ACS Library has a collection of reports and infographics using ACS data.
Python:
- Learning Python, 5th ed. (2013, O'Reilly) was my formal introduction to Python.
- Python for Data Analysis (2012, O'Reilly) introduced me to pandas.
- Python Cookbook, 3rd ed. (2013, O'Reilly) has a collection of optimized recipes.
- From the well-documented Python 3.5 standard library, I used collections, functools, os, pdb, subprocess, sys, and time for this post.
- Likewise, the documentation for numpy and pandas is thorough and invaluable.
- Of IPython's convenient "magic" commands, within this post's Jupyter Notebooks, I used %pdb, %reload_ext, and the extension %autoreload.
- StackOverflow "How to get line count cheaply in Python" (not as performant as wc -l but still neat).
Git:
- GitHub Guides are where I started with git.
- Git documentation answers a lot of questions.
- Git-flow streamlines my repository management with this branching model.
- StackOverflow "Download a specific tag with git".
dsdemos v0.0.3 (browse code):
- To design the package file structure, I used Learning Python, 5th ed. (2013, O'Reilly), Part V, "Modules and Packages"; and Python Cookbook, 3rd ed. (2013, O'Reilly), Ch. 10, "Modules and Packages".
- I use Google-style docstrings adapted from the example by the Napoleon extension to Sphinx (a Python documentation generator, not yet used by dsdemos).
- Pytest for testing.
- Semantic Versioning for version numbers.

Footnotes

Some use cases for the Census Bureau's American Community Survey with data access recommendations: ACS Which Data Tool ↩
For this post, I mounted a single disk for storage to a single instance and loaded the data in RAM. A scaled version of this pipeline on Google Cloud Platform could include integrated services such as Cloud Storage and Big Query. ↩
To download ACS data via FTP:
$ sudo curl --remote-name <url>
Decompress the ZIP files with unzip. ↩
I'm downloading the data files rather than using the Census Bureau's API because this project requires one-time access to all data rather than dynamic access to a subset of the data. ↩
StackOverflow "regex to parse well-formatted multi-line data dictionary". ↩
With dsdemos v0.0.3 above, you can export the data dictionary to JSON format with the json Python library. See example from dsdemos tests. ↩
Docs for pandas.read_csv ↩
Pandas 0.17.1 has a compatibility issue with Python 3.5. See GitHub pandas issue 11915 for a temporary fix. The issue should be resolved in pandas 0.18. ↩
For the formulas to calculate the estimates, the direct standard error, and the margin of error, as well as for example calculations, see 2013 5-year PUMS Accuracy, section 7, "Measuring Sampling Error". The sample weights '[P]WGTP' mitigate over/under-representation and control agreement with published ACS estimates. The accuracy PDF describes two methods of calculating the error (i.e. uncertainty) associated with an estimate of a characteristic:
1. Calculate the "generalized standard error" of the estimate using "design factors" from the survey. This method does not use columns '[P]WGTP[1-80]' and requires looking up a design factor for the specific characteristic (e.g "population by tenure"). See the accuracy PDF for the design factors.
2. Calculate the "direct standard error" of the estimate using the "replicate weights", which are the columns '[P]WGTP[1-80]'. This method is extensible to many kinds of characteristics (e.g. population by tenure by age).
Note: Controlled estimates of characteristics like "total population" have 0 direct standard error from replicate weights. Use the generalized standard error for these estimates. ↩
There are several ways to select rows by filtering on conditions using pandas. I prefer creating a pandas.Series with boolean values as true-false mask then using the true-false mask as an index to filter the rows. See the docs for pandas.DataFrame.loc.
Example query: Select columns 'AGEP' and 'WGTP' where values for 'AGEP' are between 25 and 34.
tfmask = np.logical_and(25 <= df['AGEP'], df['AGEP'] <= 34)
df_subset = df.loc[tfmask, ['AGEP', 'WGTP']] ↩
To download dsdemos and checkout v0.0.3 for following the example above:
$ cd ~
$ git clone https://github.com/stharrold/dsdemos.git
$ cd dsdemos
$ git checkout tags/v0.0.3
If you already have a clone of dsdemos, update your local repository then checkout the version's tag:
$ cd dsdemos
$ git pull
$ git checkout tags/v0.0.3 ↩
ACS Methodology includes design details, sample sizes, coverage estimates, and past questionnaires. ↩
Data from the Census Bureau was used to identify Japanese communities as part of the internment of US citizens and residents with Japanese ancestry during World War II. See the ACLU's FAQ section about census data and the Wikipedia article "Internment of Japanese Americans". ↩
"Race" is a problematic term with historical connotations and conflicts between self-identification and labeling by others. The 2015 ACS questionnaire refers to "race" and "ethnicity" separately. The American Anthropological Association recommended in 1997 that questions about "race" and "ethnicity" are ambiguous given the historical context and would be better phrased as about "race/ethnicity". For this project, I refer to "race" and "ethnicity" as "race/ethnicity". The following links are also helpful:
↩
The ACS 3-year estimates are discontinued; 2013 is the last year included in the 3-year estimates. For guidance in choosing, accessing, and using a data set, see ACS Guidance for Data Users. ↩
StackExchange Programmers "R vs Python for data analysis". ↩
See the PyData stack for a collection of performant Python packages. ↩

Running an IPython Notebook on Google Compute Engine from Chrome

2016-02-08T02:45:00+00:00

Contents

Overview
Motivations
First-time setup
Helpful links
Acknowledgements
Footnotes

Overview

In October 2015, I bought a Chromebook with the intent of learning how to move my data-science development environment to the cloud. In exchange for an extra 5 minutes of setup, I now have a flexible infrastructure that can scale with the task. This setup is cross-platform in that it can be used on any laptop with Chrome, not just a Chromebook.

Brief setup:

Start a Google Compute Engine virtual machine instance.
Start a Jupyter Notebook server on the instance:
$ jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser &
$ disown 1234 (where 1234 is the process ID)
Create an SSH tunnel to forward a local port to the server's port on the instance:
$ ssh -f -N -L localhost:8888:0.0.0.0:8888 samuel_harrold@123.123.123.123
For Chrome Secure Shell, omit -f to keep the tunnel open (see screenshot).
View the server at http://localhost:8888
I use the Cloud9 IDE and connect the instance as an SSH workspace.

Motivations

Why did I move to the cloud?

I wanted to save money.
- With a cloud-based platform, I only need a laptop as a web browser. I spent $170 on my ASUS C201 Chromebook with 4GB RAM.
- I spend about $20 per month on Google Cloud services.¹
- I spend $10 per month on a Cloud9 micro plan for SSH workspaces.
I wanted a reproducible environment. Snapshots can serve as simple backups of instances. For more complex platform managment, there's the Google Cloud Shell.
I wanted to test the scalability of a pipeline. On a cloud platform, I can mount disks with large data sets and change the instance size to test how efficiently algorithms use CPUs and memory. Connecting other cloud services expands the possibilities.²

Why do I use Google Cloud?

Between Google Cloud and Amazon Web Services, I chose Google Cloud for its intuitive UI. SSH within the browser is very convenient.

Why do I use IPython Notebooks?

IPython (Jupyter) Notebooks are an important part of my development process since they enable me to prototype quickly and to share my work in-progress. The notebook serves as a top-level script, the parts of which I eventually modularize as components of installable packages. I prefer the Continuum Analytics Anaconda Python distribution for its Conda package manager. I'm using Python 3.5.

Why do I use Cloud9?

I saw that Cloud9 is popular and has good documentation.³ I wanted a cloud-based IDE since I didn't want to spend resources on my Chromebook or on my instances to run the IDE.

First-time setup

There are many ways to run a Jupyter Notebook server on a virtual machine instance. This is one example setup working from my Chromebook with details for newcomers:

Create a Google Compute Engine virtual machine instance and SSH keys:
- Make a project in the Google Developers Console.
- Configure an instance:
  - Machine type: Start with the smallest machine type.⁴
  - Boot disk: Start with the default boot disk (Debian, 10GB).⁵
  - Firewall: Allow HTTP and HTTPS connections to use curl and wget.
  - Project access: Reserve an external IP address ("Networking" > "External IP"). Other settings can be left at default.⁶ For this example, I give 123.123.123.123 as my instance's static external IP address.
- Connect to the instance, e.g. with Google's in-browser SSH.
- Update the Debian system:
  $ sudo apt-get update && sudo apt-get dist-upgrade
- Generate an SSH key pair for the instance and might as well connect to GitHub.⁷
Start a Jupyter Notebook server on the instance from the in-browser SSH:
- Install Python on the instance.
- Start a Jupyter Notebook server:
  $ jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser &
  $ disown 1234 (where 1234 is the process ID)⁸
Create an SSH tunnel to forward a local port to the server's port on the instance:
- Generate an SSH key pair for the Chromebook as above⁹ and add the Chromebook's public key to the instance's authorized_keys.¹⁰
- Within Chrome, install Chrome Secure Shell and forward a port (see screenshot above):
  Username: samuel_harrold (in the instance's shell, run whoami)
  Hostname: 123.123.123.123 (the instance's external IP address)
  Port: 22
  Identity: id_rsa ¹¹
  SSH Arguments: -N -L localhost:8888:0.0.0.0:8888 ¹² ¹³
- View the server at http://localhost:8888.
For an IDE, connect a Cloud9 remote SSH workspace to the instance:
- Install Node.js on the instance.
- Create a Cloud9 SSH workspace, copy the public SSH key from Cloud9 to the instance's authorized_keys as above, then open the workspace:¹⁴
  Username: samuel_harrold (in the instance's shell, run whoami)
  Hostname: 123.123.123.123 (the instance's external IP address)
  Initial path: /home/samuel_harrold
  Port: 22
  Node.js binary path: /usr/bin/nodejs (in the instance's shell, run which nodejs)
To shutdown the instance:
- Close the Jupyter Notebook and the Chrome Secure Shell tabs. Kill the Jupyter Notebook server.¹⁵
- Close the Cloud9 workspace tab.
- "Stop" the instance in the Developers Console.
For a simple backup of the instance, create a snapshot from the Developers Console. This can be done while the instance is running.
To change the instance's machine type or disk size:
- Shutdown the instance as above.
- Create a snapshot of the instance.
- Clone the instance but set the new boot disk to the new snapshot and...
  - ...if changing the machine type, set the new machine type.
  - ...if changing the disk size, set the new disk size.
- Reassign the external IP address to the new instance.¹⁶
- Start the Jupyter Notebook server on the instance and create an SSH tunnel as above.¹⁷
- Open the Cloud9 workspace.

Helpful links

Some links I found helpful for this blog post:

Chrome:
- Chrome app Secure Shell.
- Chrome app cookies.txt to export cookies from Chrome for wget.
- To inspect system resources, chrome://system in Chrome's address bar.
- To inspect RAM usage by Chrome, chrome://memory-redirect in Chrome's address bar (from the Chrome Task Manager).
Chromebook:
- Chromebook 2GB vs 4GB Demo. I had to upgrade from a 2GB RAM Chromebook to a model with 4GB RAM since I typically use about 2.5GB RAM while working.
- Using a Chromebook as a developer. I've had my Chromebook for 2 months and haven't yet needed developer mode.
- Low-profile flash drive to expand the Chromebook's storage.
- Chromebook app SFTP.
- Chomebook app Caret text editor.
Linux:
- Classic Shell Scripting (2005, O'Reilly) was my formal introduction to Bash and Linux/Unix.
- Choosing between Debian and Ubuntu.
- Updating a Debian system.
- SSH login without password.
- SSH port forwarding (tunnels) explained.
- Google's recommended best practices for securing communications with Compute Engine instances.
- wget vs curl with examples
- StackExchange Unix and Linux download from Google Drive with wget
- Download from Kaggle with wget
- disown examples.
IPython (Jupyter) Notebooks:
- Running an IPython Notebook from Amazon Web Services.
- Hosting a password-protected IPython Notebook on Google Compute Engine.

Acknowledgements

Thanks to John and Julie for their early reviews.

John Jardel: http://thewannabedatascientist.blogspot.com
Julie Hollek: https://github.com/jkru

Footnotes

As of Dec 2015 on Google Compute Engine, running a 1-core shared virtual CPU instance with 0.6GB RAM costs about $4.50 per month. Running a 32-core virtual CPU instance with 120GB RAM costs about $1.12 per hour. ↩
There are also hosted services like Continuum Analytics Wakari, Google Cloud Datalab, Cloud9 hosted workspaces, and Digital Ocean. ↩
As of Dec 2015, Cloud9 doesn't support debugging in Python. However, this hasn't been a problem for me since I use pdb for debugging and pytest for testing. I use the IDE mostly for code navigation, autocomplete, and managing packages with many files. ↩
Determine if more RAM is necessary by using free -m to display the free memory (RAM) in MB. Use the Developers Console to determine the CPU utilization. ↩
Determine if more disk space is necessary by using df -B MB to display the free disk space in MB. ↩
Reassigning a static external IP address to a new instance when changing instances is often more convenient than changing an ephemeral IP address in all connections to the instance, e.g. in Chrome Secure Shell and Cloud9. ↩
For Google's in-browser SSH, xclip does not function. Copy the public key from less. ↩
Disowning a background process (the control operator &) from the shell allows a process to continue running in the background when the shell is closed. ↩
To create an SSH key pair for the Chromebook without going into the laptop's developer mode, generate an extra pair of keys on the instance as above then move them to the Chromebook. I save mine under Downloads/ssh (no dot-file access without developer mode). Transfer the keys by copy-paste using less from instance's in-browser SSH and a text editor app for Chromebook or download them from a connected Cloud9 SSH workspace: right-click the file > "Download". ↩
To append a local public SSH key, e.g. id_rsa.pub, to a remote machine's authorized_keys file, in the instance's in-browser shell:
$ cat >> ~/.ssh/authorized_keys
[Ctrl+V to paste the local public key, then Enter]
[Ctrl+D to signal end of file] ↩
Select both of the Chromebook's private and public keys, id_rsa and id_rsa.pub, to import as a pair. ↩
Omit the -f option to keep Chrome Secure Shell's tunnel open. Pin the tab in Chrome (right-click the tab > "Pin tab") to keep Chrome Secure Shell open and minimized in the browser. ↩
To paste the password for the Chromebook's SSH key, use Chrome's paste function ("Customize and control" > "Edit" > "Paste"; using Ctrl+V will input ^v as the password). In place of ssh-add on my Chromebook, I use LastPass to manage passwords. ↩
If the Cloud9 workspace fails to connect to the instance, e.g. the terminal within the workspace doesn't receive input, run the Cloud9 dependency installation script then reopen the workspace:
curl -L https://raw.githubusercontent.com/c9/install/master/install.sh | bash
(requires HTTPS traffic allowed in the instance's firewall settings) ↩
In the instance's in-browser SSH:
$ lsof -i:8888 (list process IDs filtered by port)
$ kill 1234 (send a termination signal to the process ID)
(install lsof with sudo apt-get install lsof) ↩
In the Developers Console, manage IP addresses under "Products & services" > "Networking". ↩
Because the external IP address was reassigned to a new instance, a warning will appear that the remote host identification has changed. To remove the offending ECDSA key from known_hosts, in Chrome, open the JavaScript console (Ctrl+Shift+J) and run term_.command.removeKnownHostByIndex(idx) where idx is the given line number in known_hosts, e.g. from the warning line Offending ECDSA key in /.ssh/known_hosts:1, idx=1. ↩

Test page and template

2016-02-08T02:45:00+00:00

Contents

Overview
Heading2
- Heading3
  - Heading4
    - Heading5
    - Heading6
Motivations
Body
Helpful links
Footnotes

Overview

I use this page for testing how structures are rendered on the blog and for templating new posts. I leave this page openly visible as a resource for others who are interested in building their own blog.

Source code:

20151030-test.ipynb
20151030-test-full.html

Heading2

Heading3

Heading4

Heading5

Heading6

Motivations

Why am I doing this?

Why I'm doing this.

Body

Text with a footnote.¹

A link:
GitHub link
A link with code as text: python docs

Text with always-rendered dollar signs $1.12.

Text with a non-breaking space.

Text with a referenced portion.

Text referencing above.

Text with an equation $\mathrm{SE}(X) = \sqrt{(\sum_{i=1}^{n}(X_i-X)^2)/n}$ inline.

Text with an equation

$$\mathrm{SE}(X) = \sqrt{\frac{\sum_{i=1}^{n}(X_i-X)^2}{n}}$$

as a paragraph.

Text that is collapsible:

Short answer.

Long answer.

A plot included via Markdown:

plot.jpg

A screenshot included via HTML:

A table:

	c0	c1	c2
0	a	A	10
1	b	B	11
2	c	C	12

table.html

A code snippet:

def my_print(string:str) -> None:
  r"""My docstring.

  Args:
    string (str): String to print.

  Returns:
    None

  """
  print('my '+string)
  return None

A Jupyter Notebook:

TODO: Add exported notebook by embedding HTML rather than copy-paste (issue 5).

Copy-pasted from 20151030-test-basic.html (all cells included):

20151030-test¶

Heading2¶

Heading3¶

Heading4¶

Heading5¶

Heading6¶

Description.

Initialization¶

Imports¶

In [1]:

cd ~

/home/samuel_harrold

In [2]:

# Import standard packages.
import os
import pdb # Debug with pdb.
import subprocess
import sys
import time
# Import installed packages.
import matplotlib.pyplot as plt
import pandas as pd
# Import local packages.
# Append current directory to module search path.
# Autoreload local packages after editing.
# dsdemos version: https://github.com/stharrold/dsdemos (master branch)
sys.path.insert(0, os.path.join(os.path.curdir, r'dsdemos'))
%reload_ext autoreload
%autoreload 2
import dsdemos as dsd
%matplotlib inline

In [3]:

print("Datestamp:")
print(time.strftime(r'%Y-%m-%dT%H:%M:%S%Z', time.gmtime()))
print()
print("Python version:")
print(sys.version_info)

Datestamp:
2016-01-04T16:44:26GMT

Python version:
sys.version_info(major=3, minor=5, micro=1, releaselevel='final', serial=0)

Globals¶

In [4]:

# File paths
path_static = os.path.join(os.path.expanduser(r'~'), r'stharrold.github.io/content/static')
basename = r'20151030-test'
filename = basename

Section 1¶

Description of section 1.

In [5]:

df = pd.DataFrame([['a', 'A', 10], ['b', 'B', 11], ['c', 'C', 12]], columns=['c0', 'c1', 'c2'])
path_html = os.path.join(path_static, basename, r'table.html')
df.to_html(buf=path_html)
print(path_html)
df

/home/samuel_harrold/stharrold.github.io/content/static/20151030-test/table.html

Out[5]:

	c0	c1	c2
0	a	A	10
1	b	B	11
2	c	C	12

In [6]:

plt.plot([0,1], [0,1])
path_jpg = os.path.join(path_static, basename, r'plot.jpg')
plt.savefig(path_jpg)
print(path_jpg)

/home/samuel_harrold/stharrold.github.io/content/static/20151030-test/plot.jpg

Export ipynb to html¶

In [7]:

# Export ipynb to html
path_ipynb = os.path.join(path_static, basename, filename+'.ipynb')
for template in ['basic', 'full']:
    path_html = os.path.splitext(path_ipynb)[0]+'-'+template+'.html'
    cmd = ['jupyter', 'nbconvert', '--to', 'html', '--template', template, path_ipynb, '--output', path_html]
    print(' '.join(cmd))
    subprocess.run(args=cmd, check=True)
    print()

jupyter nbconvert --to html --template basic /home/samuel_harrold/stharrold.github.io/content/static/20151030-test/20151030-test.ipynb --output /home/samuel_harrold/stharrold.github.io/content/static/20151030-test/20151030-test-basic.html

jupyter nbconvert --to html --template full /home/samuel_harrold/stharrold.github.io/content/static/20151030-test/20151030-test.ipynb --output /home/samuel_harrold/stharrold.github.io/content/static/20151030-test/20151030-test-full.html

Helpful links

Some links I found helpful for this blog post:

Some links here.

Footnotes

Text as a footnote. ↩