Empty Pipes

Selectable zoomable force directed graph in D3v4

2017-04-29T00:00:00+00:00

A previous post described how to use d3.js to create a force-directed graph we can zoom in to and select nodes from. Such a tool is useful for displaying and arranging larger networks. My colleagues and I personally used it to create a small web application for displaying RNA secondary structure.

Since that post, a new version of D3 was released. D3 V4 introduced a lot of useful new features (see Irene Ros’s excellent overview of the differences between v3 and v4). Unfortunately, however, it did not maintain backward compatiblity with previous versions of d3. This means that the previous selectable zoomable force directed graph example could not be used with new code written with the latest version of the D3 library. Until now.

As in the previous example, this graph provides the following selection behavior:

Clicking on a node selects it and de-selects everything else.
Shift-clicking on a node toggles its selection status and leaves all other nodes as they are.
Shift-dragging toggles the selection status of all nodes within the selection area.
Dragging on a selected node drags all selected nodes.
Dragging an unselected node selects and drags it while de-selecting everything else.

Upgrading the selectable zoomable force directed graph implementation to D3 v4 required a few minor and not-so-minor changes.

The new brush in v4 captures the shift, alt and meta keys to perform some actions by default. To get around this, I forked d3-brush and modified it so that it doesn’t capture the shift events. The new version (d3-brush-lite) can be found on github. There is an open github issue to disable this behavior in d3-brush.
Because the d3-drag behavior consumes all events in v4, it is no longer necessary to stop propagation.
The brush creates its own overlay which catches all events meaning that we don’t need to turn the zoom behavior off when the shift key is pressed.
Whether a node is fixed is specified by the .fx and .fy parameters. This eliminates the need to set the .fixed parameter on each node.
The force layout in v4 lets us specify an accessor for the nodes that a link connects. This lets us use ids for a link’s endpoint and makes the graph specification JSON easier to read:

{
  "nodes": [
    {"id": "Myriel", "group": 1},
    {"id": "Napoleon", "group": 1},
    {"id": "Mlle.Baptistine", "group": 1},
    ...
  ],
  "links": [
    {"source": "Napoleon", "target": "Myriel", "value": 1},
    {"source": "Mlle.Baptistine", "target": "Myriel", "value": 8},
    ...
  ]
}

The source code for this example can be found as a github gist or on bl.ocks.org.

D3v4 event filtering example

2017-04-18T00:00:00+00:00

D3 behaviors, such as d3.zoom, work by responding to events which pass through the element on which they are called. If the element has children, the behavior will be called as long as the children don’t block events’ propagation. This is often beneficial. If we want to be able to zoom on a populated SVG, we need only call the zoom behavior on the root node and we’ll be able to pan and zoom even if we drag and scroll on the child elements.

There are times, however, when we may want to ignore certain elements without having the block the propagation of the event. For this, there is event filtering. By filtering events, we can let them pass through without having to block or process them. This can be seen in the example below where dragging the background leads to panning, while dragging the circles has no effect.

The crux of the code for this example is a simple check to see that handled events have not passed though an element with a no-zoom class.

var zoom = d3v4.zoom()
    .filter(() => { return !d3v4.event.path[0].classList.contains('no-zoom') })
    .on('zoom', function(d) { g.attr('transform', d3v4.event.transform); });

A bl.ock of this example can be found here.

Fast ES6 Development Using webpack-dev-server

2016-12-06T00:00:00+00:00

Summary

Switching from gulp and webpack-stream to webpack-dev-server reduces the rebuild time for a 5500-line javascript project from ~11s to ~1.3 seconds.

Details

Whenever I create a javascript project, I do it using a very uniform directory structure and configuration, as outlined in a previous blog post. With this configuration, all the source files are transpiled using babel and bundled using the webpack-stream module as part of a step in the build process managed by gulp.

This is great because then I can run gulp serve and have it recompile and reload the resulting web page whenever I make any changes to the source code in app/scripts.

This works like a charm until the source code and dependencies get to any appreciable size. As more and more files need to be transpiled, the process gets slower and slower until at about ~10 seconds, it starts to get annoying:

[BS] 3 files changed (main.js, playground.js, worker.js)
[08:31:20] Finished 'scripts' after 11 s

So how can this be sped up? Easy, stop using gulp and webpack-stream and switch to the…

Webpack dev server

The webpack dev server runs in its own terminal and watches the source files listed in its config file (webpack.config.js). When one of the files changes, it recreates the output files specified in its config and reloads the web page. I run it using the following command line:

webpack-dev-server --content-base app --display-exclude --profile --inline | grep -v "\\[\\d\*\\]"

The grep at the end is to filter out some of the [overly] verbose output that webpack produces. So how long does it take to regenerate the code when a source file is changed?

Version: webpack 1.12.15
Time: 1296ms
chunk    {0} main.js (main) 4.61 MB

This is about 10x faster than the configuration using gulp and webpack-stream.

The resulting web page can be found at http://localhost:8080/index.html

The only thing I needed to change in my webpack.config.js file was to add output: { publicPath: '/scripts/'}. This is because my index.html file loads the compiled scripts from the scripts directory:

<script src='scripts/playground.js'></script>

Below is the entire webpack.config.js for this project. Notice that there’s multiple different targets being built including a worker script that can be used in a web worker to do compute intensive tasks off of the main UI thread.

Other notable sights include the devtool: "cheap-source-map" entry to make sure we can easily see the source code when debugging.

var path = require('path');
var webpack = require('webpack');

module.exports = {
  context: __dirname + '/app',
  entry: {
      playground: ['./scripts/playground.jsx'],
      main: ['./scripts/main.jsx'],
      worker: ['./scripts/worker.js']
  },
  devtool: "cheap-source-map",
  output: {
    path: __dirname + '/build',
    publicPath: '/scripts/',
    filename: '[name].js',
    libraryTarget: 'umd',
    library: '[name]'
  },
  module: {
    loaders: [
      {
        test: /\.jsx?$/,
        //exclude: /node_modules/,
        include: [path.resolve(__dirname, 'app/scripts')],
        loader: 'babel-loader',
        query: {
          presets: ['es2015', 'react']
        }
      }, {
        test: /\.css$/,
        loader: 'style!css'
      }
    ],
    postLoaders: [
        {
            include: path.resolve(__dirname, 'node_modules/pixi.js'),
            loader: 'transform?brfs'
        }
    ],
    externals: {

               },
    resolve: {
      extensions: ['.js', '.jsx']
    }
  }
};

Cambridge Tree Map

2016-11-16T00:00:00+00:00

The city of Cambridge, MA maintains a wealth of geographic information on its GIS website. One of the more unconvential datasets is the list of trees lining the streets of the city. It contains the position and species of around 30 thousand trees and can be explored using the Cambridge Tree Walk) application. While this app is incredily detailed and useful at high resolution, it loses all information at low resolution. The identities and positions of the trees are lost. And if they weren’t they would be too dense to display in a meaningful and intelligible manner.

To provide a different view, I calculated which tree species is most common on each block in Cambridge and plotted the results using D3.js. The analysis shows that Maple trees dominate the landscape in Cambridge. Further down the list are Oaks, Lindens and Pears. While these are the most common trees on most street blocks in Cambridge, there are a few which are dominated by less common species. This isn’t to say that those species are only found on those blocks, just that those are the only blocks where those species are in the majority.

How useful is this map? I don’t know. But it was fun to make and will hopefully serve as a decent example for introducing how D3.js can be used for cartography at Maptime Boston. A tutorial describing how this map is made is available on the GitHub page for the project. It’s also avilable as a block.

Panning and Zooming with D3v4

2016-07-03T00:00:00+00:00

All that’s necessary for panning and zooming is a translation [t_x, t_y] and a scale factor k. When a zoom transform is applied to an element at position [x₀, y₀], its new position becomes [t_x + k × x₀, t_y + k × y₀]. That’s it. Everything else is just sugar and spice on top of this simple transform.

The major difference between zooming in D3v3 and and D3v4 is that the behavior (dealing with events) and the transforms (positioning elements) are more separated. In v3, they used to be part of the behavior whereas in v4, they’re part of the element on which the behavior is called.

To illustrate, let’s plot 4 points. The rest of this post will only deal with data in one dimension. It should be trivial to expand to two dimensions. The points will represent the values 1, 1010, 1020 and 5000:

    var xScale = d3.scaleLinear()
        .domain([0,5000])
        .range([100,500])

    var dataPoints = [1,1010,1020,5000];

    gMain.selectAll('circle')
    .data(dataPoints)
    .enter()
    .append('circle')
    .attr('r', 7)
    .attr('cx', function(d) { return xScale(d); });

We can see that two of the points, 1010 and 1020, are virtually on top of each other. Using our xScale, we can determine that they’re less than 1 pixel apart.

    xScale(1010) //180.8
    xScale(1020) //181.6

What if we want to zoom in so that they’re 10 pixels apart? We’ll first need to calculate the scale factor, k:

    var k = 10 / (xScale(1020) - xScale(1010))  //~ 12.5 

Let’s say we want the point 1010 to be positioned at pixel 200. We need to determine t_x such that 200 = t_x + k × xScale(1010)

    var tx = 200 - k * xScale(1010) //-2600

When we apply this to our plot.

    var k = 10 / (xScale(1020) - xScale(1010))
    var tx = 200 - k * xScale(1010)
    var t = d3.zoomIdentity.translate(tx, 0).scale(k)

    gMain.selectAll('circle')
    .data(dataPoints)
    .enter()
    .append('circle')
    .attr('r', 7)
    .attr('cx', function(d) { return t.applyX(xScale(d)); });

We get two lovely separated circles.

Fantastic, right? But notice that the top axis still refers to the old domain. This is because we never changed it. In the old version of D3, we would attach the axis to the zoom behavior, set the translate and scale properties and be done with it. In v4, we have to rescale our linear scale manually and use the rescaled version to create the axis:

    var xNewScale = t.rescaleX(xScale)

    var xTopAxis = d3.axisTop()
    .scale(xNewScale)
    .ticks(3)

The examples above demonstrate how the zoom transforms work, but they don’t actually use the zoom behavior. For that we need to create a behavior and attach it to an element:

    var circles = svg.selectAll('circle');
    var zoom = d3.zoom().on('zoom', zoomed);

    function zoomed() {
        var transform = d3.event.transform;

        // rescale the x linear scale so that we can draw the top axis
        var xNewScale = transform.rescaleX(xScale);
        xTopAxis.scale(xNewScale)
        gTopAxis.call(xTopAxis);

        // draw the circles in their new positions
        circles.attr('cx', function(d) { return transform.applyX(xScale(d)); });
    }

    gMain.call(zoom)

Here we recompute the zoom transform every time there is a zoom event and reposition each circle. We also rescale the x-scale so that we can use it to create an axis. The astute observer will note that transform.applyX(xScale(d)) is actually equivalent to xNewScale(d). Automatic rescaling was possible using v3 by calling zoom.x(xScale), but this has been done away with in favor of explicit rescaling using transform.rescaleX(xScale).

The code above works but if we had programmatically zoomed in beforehand (as we did in the previous section by applying the transform), then applying the zoom behavior would remove that transform as soon as we start zooming.

Why?

Because in the zoomed function we obtain a transform from d3.event.transform. In previous versions of D3, this would come from the zoom behavior itself (zoom.translate and zoom.scale). In v4, it comes from the element on which the zoom behavior is called (gMain). To programmatically zoom in and then apply the zoom behavior starting from there, we need to set the zoom transform of the gMain element before we call the behavior:

var k = 10 / (xScale(1020) - xScale(1010))
var tx = 200 - k * xScale(1010)
var t = d3.zoomIdentity.translate(tx, 0).scale(k)

gMain.call(zoom.transform, t);
gMain.call(zoom)

Now we start with an already zoomed in view and can zoom in and out using the mouse.

To wrap up this post, let’s combine these techniques to create a figure which automatically zooms between random data points (a la M. Bostock’s Zoom Transitions Block). How do we do this?

First, we need a function to call every time we want to jump to a point:

    let targetPoint = 1010;

    function transition(selection) {
        let n = dataPoints.length;
        let prevTargetPoint = targetPoint;

        // pick a new point to zoom to
        while (targetPoint == prevTargetPoint) {
            let i = Math.random() * n | 0
            targetPoint = dataPoints[i];
        }

        selection.transition()
        .delay(300)
        .duration(2000)
        .call(zoom.transform, transform)
        .on('end', function() { circles.call(transition); });
    }

    circles.call(transition);

This function picks a random point (targetPoint) and calls a transition on the selection. In our case, the selection will be the circles. When the transition is over, we simply call the function again to start it over.

Second, we need a transform to center the view on the target point:

    function transform() {
        // put points that are 10 values apart 20 pixels apart
        var k = 20 / (xScale(10) - xScale(0))
        // center in the middle of the visible area
        var tx = (xScale.range()[1] + xScale.range()[0])/2 - k * xScale(targetPoint)
        var t = d3.zoomIdentity.translate(tx, 0).scale(k)
        return t;
    }

And that’s all. Just remember, when zooming and panning the position of the transformed point [x₁,y₁] = [t_x + k × x₀, t_y + k × y₀]. Everything else is just window dressing.

Isochrone Driving Maps of the World

2016-03-04T00:00:00+00:00

One of my favorite things about maps is the context they provide for overlayed information. This can range from the mundane and orthodox (such as roads and boundaries) to the esoteric and abstract. Nearly a year ago I wrote a blog post which overlayed travel time (isochrone) data on top of a map of Europe. It showed how long it would take somebody to travel from a given city to any other point in Europe using only public transportation. In this post, I do the exact same thing using driving times.

Driving times starting in Vienna, Austria. Notice how the frontier of the contours is more rounded than in Lincoln, NE.

Driving times starting in Lincoln, Nebraska.

The wonderful thing about portraying driving times is that it's possible to make such maps for cities from all over the world. In doing so, we can see the how the transportation infrastructure of a region meshes with the natural features to create a unique accessibility profile. Lincoln (Nebraska) is centered in the USA and has a characteristic diamond shaped travel time profile. Why? Anyone that has looked at a map of the region will certainly have noticed that the roads are arranged in a grid pattern. Thus it takes much longer to travel along the diagonal than to travel straight north and south. Vienna, Austria, on the other hand, has a more circular accessibility profile due to the abundance of roads going in all directions.

The Andes mountains block driving to the east of Santiago, Chile

The Darien Gap separates Central America from South America

Looking at the isochrone map of Santiago, Chile, one can clearly see the how the Andes mountains block travel east of the city. Southeast of panama city, the odd fact that you can’t drive between North / Central America and South America becomes clear. Zooming in at identical levels, you can compare cities and see the difference in accessibility of the relatively wealthy South Africa with that of the less developed, wilder Congo. The differences in accessibility between different cities of the world can range from the trivial (Denver, CO vs Lincoln, NE) to the substantial (Perth, Australia vs. Sydney, Australia). Individual cities can have a wide automobile-reachible area (Moscow, Russia) or a narrow, geography, politics and infrastructure-constrained area (Irkutsk, Russia).

The accessibility profile of Kinshasa, Democratic Republic of the Congo is rugged and discontinuous.

Cape Town, South Africa has good links to the interior of the country as well as to Namibia in the north.

Whatever the case, the places that are most interesting are always those that are also most familiar. For this reason, I’ve provided overlays for most major cities around the world. The map currently shows Vienna, but clicking any of the links below will open a map for that particular city. The travel times were obtained using GraphHopper and OpenStreetMap data so don’t be surprised if they differ from those of Google maps.

Other Cities

South America

Asuncion	Cartagena	Manaus	Santiago
Bogota	Cayenne	Montevideo	Sao Paolo
Brasilia	Georgetown	Paramaribo	Ushuaia
Buenos Aires	La Paz	Quito
Caracas	Lima	Rio De Janeiro

Australia and Oceania

Adelaide	Christchurch	Noumea	Sydney
Apia	Hobart	Perth	Wellington
Auckland	Honiara	Port Moresby
Brisbane	Melbourne	Suva

Errata / Disclaimer

Everything is an estimate. Rounding errors abound. Don’t use this for anything but entertainment and curiosity. But you already know that.

Some data may be missing. There may be faster routes. Google Maps certainly finds some better ones. If you find issues, please let me know and I’ll do my best to fix them.

How it’s made, technically

For each starting city, an area encompassing 30 degrees north, east, south and west was subdivided into a .1 degree grid. Directions from the starting city to each point were calculated using graphhopper and the OSM map files. From this, contours were extracted using matplotlib’s contourf function. These contour files are stored as JSONs and loaded as a D3.js overlay on top of a leaflet map.

All of the code used to create these maps is on github. If you have any questions, feel free to ask.

Background Information and Motivation

This project was a logical extension of the isochrone train maps of Europe project.

ES6 Modules using Webpack and Gulp

2016-03-02T00:00:00+00:00

Overview

One of the most common scenarios I run into when creating javascript applications is the following.

I start working on some application (let’s call it foo).
I start working on a different program (let’s call it bar) and need some of the functionality that I already implemented in foo.

In python, accomplishing this is trivial by placing the source files in the same directory and saying import foo. In javascript, it’s a little more complicated. Rather than recapitulate a lot of existing documentation about the different ways that one can implement modules in javascript (CommonJS, AMD, etc…), I’ll just dive into how I use gulp and webpack to take advantage of the ECMAScript 6 syntax for modules to create reusable javascript components.

Goal

The goal of this project is to create two components: foo and bar. foo will export a function called drawCircle that we can import into bar. bar will then draw a rectangle on top of the circle.

Here’s the code for app/scripts/bar.js

import d3 from 'd3';
import {drawCircle} from 'foo';

export function drawRectangleAndCircle(divName) {
    drawCircle(divName);

    d3.select('svg')
    .append('rect')
    .attr('x', 75)
    .attr('y', 75)
    .attr('width', 100)
    .attr('height', 100)
    .classed('bar-rectangle', true)
    .attr('opacity', 0.7)
}

And the relevant code from app/index.html which uses bar.js:

 <!-- build:js scripts/bar.js -->
  <script src='scripts/bar.js'></script>
  <!-- endbuild -->

  <script type='text/javascript'>

  bar.drawRectangleAndCircle('#circle');

  </script>

In order for all of this to function, we need some tools to manage which modules we have installed (npm), resolve the import statements (webpack), translate the es6 to more universally understandable es5 (babel) and to build everything into one package (gulp). The easiest way to get started is to use yeoman to scaffold out the application. To install yeoman, simply use npm:

npm install -g yo

Then, install the generator for this tutorial:

npm install -g generator-gulp-webpack-es6

Setting up foo

Create a directory to host the foo module and scaffold it out:

mkdir foo
cd foo
yo gulp-webpack-es6

When yeoman asks for Your project name:, make sure to enter foo:

You can just use the default values for the other fields. The default values for author name and email address are taken from your ~/.gitconfig if you have one.

That’s it. We now have a module which simply draws a circle in the browser. Witness it in all its glory by running:

gulp serve

And pointing your browser to 127.0.0.1:9000. The result should look like this:

Now, we want to install this package locally so that we can import it from our soon-to-be-created package bar. To do this we need to make one minor change to our webpack.config.js by adding the following lines:

  externals: {
      "d3": "d3"
  },

These are necessary to tell webpack not to include d3 in the packaged file. Under normal circumstances this wouldn’t be necessary but if I omit it and use npm link as described below, I get the following error:

Uncaught TypeError: Cannot read property 'document' of undefined

It’s seems like it happens either because webpack adds a use strict; somewhere in the bundled foo package, or because d3 is included in both foo and bar. In either case, specifying that it’s an external package seems to resolve the problem.

To build and install foo so that it can be imported by other modules, just run the following two commands.

gulp build
npm link

The first will compile the code into a single javascript file and the second will install it as a global module on the local computer. From now on, whenever we need to make changes to foo we just need to run gulp build and they will be reflected in every local package that uses foo.

Setting up bar

To create the calling package bar we follow a very similar procedure:

mkdir bar
cd bar
yo gulp-webpack-es6

When yeoman asks for Your project name:, make sure to enter bar. You can use the default values for the other fields.

Now because we want to use the functionality provided in foo, we need to install it:

npm link foo

Adn then we need to modify app/scripts/bar.js and have it implement its own functionality:

import d3 from 'd3';
import {drawCircle} from 'foo';

export function drawRectangleAndCircle(divName) {
    drawCircle(divName);

    d3.select('svg')
    .append('rect')
    .attr('x', 75)
    .attr('y', 75)
    .attr('width', 100)
    .attr('height', 100)
    .classed('bar-rectangle', true)
    .attr('opacity', 0.7)
}

And finally we need to change the <script> code inside index.html to call the newly created function from bar:

<script type='text/javascript'>

bar.drawRectangleAndCircle('#circle');

</script>

If everything worked as it should, running gulp serve should show a lovely circle / square combination:

Generator directory structure

The directory structure of the componenets generated using gulp-webpackes6 is organized so that the configuration files are in the root directory, all of the code is in the app subdirectory, unit tests are in test and the compiled javascript is in dist:

.
├── app
│   ├── index.html
│   ├── scripts
│   │   ├── foo.js
│   │   └── helper_module.js
│   └── styles
│       └── foo.css
├── bower.json
├── dist
│   ├── index.html
│   └── scripts
│       └── foo.js
├── gulpfile.babel.js
├── package.json
└── webpack.config.js

Changing the module’s name

The default module is created so that the main file is in app/scripts/module_name.js where module_name is the name given in the generator questionnaire. To change it to something else, you can either re-run the generator or make the following changes:

Rename app/scripts/module_name.js
Change occurences of module_name in app/index.html
Change the entry line in webpack.config.js

Largest Ski Areas On Each Continent

2015-11-29T00:00:00+00:00

The chart below shows the five largest ski areas on each continent (except Antarctica). Africa only has three entries because those are the only ski areas I found with annotated ski lifts and slopes. Everything is calculated according to some reasonable yet arbitrary heuristics so please take the rankings with a grain of salt. If you click on a ski area, it will be displayed in the map below.

The Map

All the ski areas in the chart above are annotated on the map below. Clicking on an area above centers the map below.

How it’s made

Each ski area is defined by something like an alpha shape, calculated over all the lifts and slopes. Each “alpha shape” -like area is calculated by first creating a Delaunay triangulation of all ski lift and slope waypoints. Every triangle which has a side greater than 1km is then removed. The result is a set of disconnected shapes all composed of multiple triangles. Each aggregation of triangles is what I consider a ski area.

In some cases, this can lead to two ski resorts being connected even though you might have to walk up to 1km to get from one lift to another (e.g. Kitzbühel / SkiWelt above). In the opposite case (e.g. Oukaimeden), a long lift may not be counted simply because the points defining it create degenerate triangles. Nevertheless, in most cases the shapes created reflect the ski areas they represent quite well.

Here’s the recipe to re-create the graphic:

Download OpenStreetMap dumps of each continent
Filter for lifts and slopes
Sample points along long segments to make sure they’re not split
Calculate a Delaunay triangulation of all the OSM nodes and the sampled points
Exclude all triangles which have an edge longer than 1km (concave hull of sorts)
Polygonize
Google the towns near the resorts to figure out what the resorts are called
Convert to topojson and downsample to reduce the size
Make table using d3.js
Create zoomable map using leaflet.js and a d3.js overlay

Questions? Comments? Twitter (@pkerpedjiev) or email (see about page).

Never Take The First Option

2015-10-21T00:00:00+00:00

The King’s Jewels Problem

A long time ago, the story goes, a hero rescues the king’s life. The elated king promises the hero one of his crown jewels as a reward. He takes out his bag of treasures and tells the hero to reach into the bag and randomly choose one. If he likes it, he can keep it. If he doesn’t, he can throw it aside and lose it forever. The hero, greedy as he is heroic, naturally wants to pick the best jewels. Looking at the bag, he judges there to be about 10 jewels inside. He stops to ponder for a second.

How should the hero proceed to maximize his chances of scoring the best jewel in the bag?

Should he just take the first one he picks or should he look at a few before deciding on one to keep? If so, how many?

We could try to answer this question analytically, but since “data science” is all the rage, why not try and answer it stochastically? If we simulate the hero’s choices, we should be able to come up with some numerically-grounded recommendation for what he should do.

Let’s assume a simple scenario. The hero first looks at n jewels and then takes the next best jewel. If none of the jewels that come up after the first n are better, he must settle for the last.

The results are striking! If he doesn’t look at any jewels and always pick the first one, he’s equally likely to get any of the jewels. That is, the chances of picking the best one are 1/9. If he looks at one, discards it and then pick the next jewel which is better than any of the discarded, the chances of picking the best jewel increase dramatically to almost 4 / 10! In fact, the entire distribution gets skewed right and the chances of picking the second and third best jewels also significantly increase.

But the hero is after the best. From the histogram above, it’s hard to see how many he has to see and discard to maximize the chances of picking the best one. For that we can either run a simulation with lots of jewels and lots of iterations. Or we can run simulations for increasing numbers of jewels and plot how many one has to look at, to maximize his/her chances of getting the best:

What this chart shows is that the number one needs look at and throw out to maximize the chances of finding the best grows linearly with the number of jewels. The slope of the line (0.36), indicates the fraction of the total number of jewels that we should examine before picking the next best.

The red line shows how many we one should look at to maximize the average value of the kept jewel. If we try and maximize the chances of bagging the best jewel, we also increase the chances that we end up with the worst. When trying to get the highest value on average, one should look at fewer options before deciding to stick with the next best.

What if the jewels are…?

But what if the values of the jewels are not uniformly distributed? What if most jewels are average in value and only a few are either precious or worthless? Or what if most jewels have a low value with only a few true gems? The next two sections show the results when the jewels are normally distributed (mostly average) and exponentially distributed (mostly low-value).

Normally Distributed Jewels

Exponentially Distributed Jewels

Moral of the Story

So what should we tell our king-saving friend?

As long as there’s more than one option and you want to get the best reward possible, always discard the first third (or 1/e‘th, where e is Euler’s Number, to be more precise) of the options and then pick the next one which is better than any you have seen so far. If you want something better than average but not necessarily the best, discard fewer. Or, to put it more succintly:

NEVER take the first offer when you have more than one choice!

Acknowledgements

Huge thanks to Steven Rudich for introducing me to the King's Jewels problem a long time ago during a summer lecture at the Andrew's Leap program at CMU.
Thanks to Trent Richardson for the javascript implementation of a linear regression function.

Graph Proportions and Label Sizes

2015-10-18T00:00:00+00:00

The example below shows how to create a graph that is both aesthetically pleasing and sensibly proportioned. I used these parameters to generate the majority of the figures in my PhD thesis. The savefig function can also be used to save a pdf or an svg file for modification in your favorite image editor (e.g. inkscape).

The code to generate this plot uses the seaborn library in an IPython notebook:

%load_ext autoreload
%autoreload 2
%pylab inline

import numpy as np
import seaborn as sns

rc('text', usetex=True)
plt.rc('font', family='Palatino')
sns.set_style('white')
sns.set_context("notebook", font_scale=1.0, rc={"lines.linewidth": 2})
rc('text', usetex=True)    # use latex in the labels
pylab.rcParams['figure.figsize'] = (1.3,1.0)


font = {'family' : 'serif',
        'serif': 'Palatino',
        'weight' : 'bold',
        'size'   : 11}

matplotlib.rc('font', **font)

# create the dummy data
x = np.linspace(0, 2*math.pi,100)
y = sin(x)
z = cos(x)

# plot that sucker
fig, ax = plt.subplots()
ax.plot(x, y, label='sin')
ax.plot(x, z, label='cos')

ax.set_xlabel('x')
ax.set_ylabel('y')
ax.axvline(x=3, color='red', ls='dashed')
ax.set_title('Trigonometric Functions', y=1.08)

ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

pyplot.locator_params(nbins=7)
for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
         ax.get_xticklabels() + ax.get_yticklabels()):
    item.set_fontsize(10)

handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels, bbox_to_anchor=(1.8, 0.85))

plt.savefig('img/trigonometric_functions.png', dpi=500, bbox_inches="tight")

Atlanta	Juneau	Ottawa	Tegucigalpa
Belmopan	Lincoln	Panama City	Toronto
Charlotte	Los Angeles	Philadelphia	Vancouver
Chicago	Managua	Phoenix
Dallas	Mexico City	Pittsburgh	Wichita
Denver	Miami	San Diego	Winnipeg
Edmonton	Minneapolis	San Francisco	Yellowknife
Fairbanks	Montreal	San Jose
Guatemala City	New York	San Salvador
Houston	Nome	Seattle

Amsterdam	Budapest	London	Rome
Antwerp	Copenhagen	Luxembourg	Sofia
Athens	Dublin	Madrid	Stockholm
Barcelona	Frankfurt	Minsk	Tallinn
Belgrade	Geneva	Oslo	Vienna
Berlin	Helsinki	Paris	Vilnius
Bratislava	Jena	Podgorica	Warsaw
Brussels	Lisbon	Prague	Zagreb
Bucharest	Ljubljana	Riga	Zurich

Abidjan	Conakry	Lagos	Monrovia
Accra	Dakar	Libreville	N Djamena
Algiers	Dar Es Salaam	Lilongwe	Nairobi
Antananarivo	Freetown	Lome	Niamey
Asmara	Gaborone	Luanda	Nouakchott
Banjul	Johannesburg	Lusaka	Porto Novo
Bissau	Juba	Maputo	Rabat
Brazzaville	Kampala	Marrakesh	Tripoli
Bujumbura	Khartoum	Maseru	Tunis
Cairo	Kigali	Mbabane	Windhoek
Cape Town	Kinshasa	Mogadishu	Yaounde

Abu Dhabi	Dubai	Moscow	St Petersburg
Almaty	Dushanbe	Mosul	Tashkent
Amman	Hanoi	Mumbai	Tbilisi
Ankara	Irkutsk	Muscat	Tehran
Ashgabat	Islamabad	Naypyidaw	Tel Aviv
Astana	Jakarta	New Delhi	Thimphu
Baghdad	Jeddah	Novosibirsk	Tianjin
Baku	Jerusalem	Phnom Penh	Tokyo
Bangkok	Kabul	Pyongyang	Ulaanbaatar
Beijing	Karachi	Riyadh	Vientiane
Beirut	Kathmandu	Sanaa	Vladivostok
Bishkek	Kuwait City	Seoul	Yerevan
Colombo	Lahore	Shanghai
Damascus	Manama	Singapore
Doha	Manila	Sochi