Empty Pipes



Fast ES6 development using webpack-dev-server

  • 06 Dec 2016
  • |
  • javascript
  • es6
  • |

Summary

Switching from gulp and webpack-stream to webpack-dev-server reduces the rebuild time for a 5500-line javascript project from ~11s to ~1.3 seconds.

Details

Whenever I create a javascript project, I do it using a very uniform directory structure and configuration, as outlined in a previous blog post. With this configuration, all the source files are transpiled using babel and bundled using the webpack-stream module as part of a step in the build process managed by gulp.

This is great because then I can run gulp serve and have it recompile and reload the resulting web page whenever I make any changes to the source code in app/scripts.

This works like a charm until the source code and dependencies get to any appreciable size. As more and more files need to be transpiled, the process gets slower and slower until at about ~10 seconds, it starts to get annoying:

[BS] 3 files changed (main.js, playground.js, worker.js)
[08:31:20] Finished 'scripts' after 11 s

So how can this be sped up? Easy, stop using gulp and webpack-stream and switch to the…

Webpack dev server

The webpack dev server runs in its own terminal and watches the source files listed in its config file (webpack.config.js). When one of the files changes, it recreates the output files specified in its config and reloads the web page. I run it using the following command line:

webpack-dev-server --content-base app --display-exclude --profile --inline | grep -v "\\[\\d*\\]"

The grep at the end is to filter out some of the [overly] verbose output that webpack produces. So how long does it take to regenerate the code when a source file is changed?

Version: webpack 1.12.15
Time: 1296ms
chunk    {0} main.js (main) 4.61 MB

This is about 10x faster than the configuration using gulp and webpack-stream.

The resulting web page can be found at http://localhost:8080/index.html

The only thing I needed to change in my webpack.config.js file was to add output: { publicPath: '/scripts/'}. This is because my index.html file loads the compiled scripts from the scripts directory:

<script src='scripts/playground.js'></script>

Below is the entire webpack.config.js for this project. Notice that there’s multiple different targets being built including a worker script that can be used in a web worker to do compute intensive tasks off of the main UI thread.

Other notable sights include the devtool: "cheap-source-map" entry to make sure we can easily see the source code when debugging.

var path = require('path');
var webpack = require('webpack');

module.exports = {
  context: __dirname + '/app',
  entry: {
      playground: ['./scripts/playground.jsx'],
      main: ['./scripts/main.jsx'],
      worker: ['./scripts/worker.js']
  },
  devtool: "cheap-source-map",
  output: {
    path: __dirname + '/build',
    publicPath: '/scripts/',
    filename: '[name].js',
    libraryTarget: 'umd',
    library: '[name]'
  },
  module: {
    loaders: [
      {
        test: /\.jsx?$/,
        //exclude: /node_modules/,
        include: [path.resolve(__dirname, 'app/scripts')],
        loader: 'babel-loader',
        query: {
          presets: ['es2015', 'react']
        }
      }, {
        test: /\.css$/,
        loader: 'style!css'
      }
    ],
    postLoaders: [
        {
            include: path.resolve(__dirname, 'node_modules/pixi.js'),
            loader: 'transform?brfs'
        }
    ],
    externals: {

               },
    resolve: {
      extensions: ['.js', '.jsx']
    }
  }
};

Cambridge Tree Map

  • 16 Nov 2016
  • |
  • javascript
  • d3.js
  • maps
  • |

The city of Cambridge, MA maintains a wealth of geographic information on its GIS website. One of the more unconvential datasets is the list of trees lining the streets of the city. It contains the position and species of around 30 thousand trees and can be explored using the Cambridge Tree Walk) application. While this app is incredily detailed and useful at high resolution, it loses all information at low resolution. The identities and positions of the trees are lost. And if they weren’t they would be too dense to display in a meaningful and intelligible manner.

To provide a different view, I calculated which tree species is most common on each block in Cambridge and plotted the results using D3.js. The analysis shows that Maple trees dominate the landscape in Cambridge. Further down the list are Oaks, Lindens and Pears. While these are the most common trees on most street blocks in Cambridge, there are a few which are dominated by less common species. This isn’t to say that those species are only found on those blocks, just that those are the only blocks where those species are in the majority.



How useful is this map? I don’t know. But it was fun to make and will hopefully serve as a decent example for introducing how D3.js can be used for cartography at Maptime Boston. A tutorial describing how this map is made is available on the GitHub page for the project. It’s also avilable as a block.


Panning and Zooming with D3v4

  • 03 Jul 2016
  • |
  • javascript
  • d3.js
  • zooming
  • |

All that’s necessary for panning and zooming is a translation [tx, ty] and a scale factor k. When a zoom transform is applied to an element at position [x0, y0], its new position becomes [tx + k × x0, ty + k × y0]. That’s it. Everything else is just sugar and spice on top of this simple transform.

The major difference between zooming in D3v3 and and D3v4 is that the behavior (dealing with events) and the transforms (positioning elements) are more separated. In v3, they used to be part of the behavior whereas in v4, they’re part of the element on which the behavior is called.

To illustrate, let’s plot 4 points. The rest of this post will only deal with data in one dimension. It should be trivial to expand to two dimensions. The points will represent the values 1, 1010, 1020 and 5000:

    var xScale = d3.scaleLinear()
        .domain([0,5000])
        .range([100,500])

    var dataPoints = [1,1010,1020,5000];

    gMain.selectAll('circle')
    .data(dataPoints)
    .enter()
    .append('circle')
    .attr('r', 7)
    .attr('cx', function(d) { return xScale(d); });

We can see that two of the points, 1010 and 1020, are virtually on top of each other. Using our xScale, we can determine that they’re less than 1 pixel apart.

    xScale(1010) //180.8
    xScale(1020) //181.6

What if we want to zoom in so that they’re 10 pixels apart? We’ll first need to calculate the scale factor, k:

    var k = 10 / (xScale(1020) - xScale(1010))  //~ 12.5 

Let’s say we want the point 1010 to be positioned at pixel 200. We need to determine tx such that 200 = tx + k × xScale(1010)

    var tx = 200 - k * xScale(1010) //-2600

When we apply this to our plot.

    var k = 10 / (xScale(1020) - xScale(1010))
    var tx = 200 - k * xScale(1010)
    var t = d3.zoomIdentity.translate(tx, 0).scale(k)

    gMain.selectAll('circle')
    .data(dataPoints)
    .enter()
    .append('circle')
    .attr('r', 7)
    .attr('cx', function(d) { return t.applyX(xScale(d)); });

We get two lovely separated circles.

Fantastic, right? But notice that the top axis still refers to the old domain. This is because we never changed it. In the old version of D3, we would attach the axis to the zoom behavior, set the translate and scale properties and be done with it. In v4, we have to rescale our linear scale manually and use the rescaled version to create the axis:

    var xNewScale = t.rescaleX(xScale)

    var xTopAxis = d3.axisTop()
    .scale(xNewScale)
    .ticks(3)

The examples above demonstrate how the zoom transforms work, but they don’t actually use the zoom behavior. For that we need to create a behavior and attach it to an element:

    var circles = svg.selectAll('circle');
    var zoom = d3.zoom().on('zoom', zoomed);

    function zoomed() {
        var transform = d3.event.transform;

        // rescale the x linear scale so that we can draw the top axis
        var xNewScale = transform.rescaleX(xScale);
        xTopAxis.scale(xNewScale)
        gTopAxis.call(xTopAxis);

        // draw the circles in their new positions
        circles.attr('cx', function(d) { return transform.applyX(xScale(d)); });
    }

    gMain.call(zoom)

Here we recompute the zoom transform every time there is a zoom event and reposition each circle. We also rescale the x-scale so that we can use it to create an axis. The astute observer will note that transform.applyX(xScale(d)) is actually equivalent to xNewScale(d). Automatic rescaling was possible using v3 by calling zoom.x(xScale), but this has been done away with in favor of explicit rescaling using transform.rescaleX(xScale).

The code above works but if we had programmatically zoomed in beforehand (as we did in the previous section by applying the transform), then applying the zoom behavior would remove that transform as soon as we start zooming.

Why?

Because in the zoomed function we obtain a transform from d3.event.transform. In previous versions of D3, this would come from the zoom behavior itself (zoom.translate and zoom.scale). In v4, it comes from the element on which the zoom behavior is called (gMain). To programmatically zoom in and then apply the zoom behavior starting from there, we need to set the zoom transform of the gMain element before we call the behavior:

var k = 10 / (xScale(1020) - xScale(1010))
var tx = 200 - k * xScale(1010)
var t = d3.zoomIdentity.translate(tx, 0).scale(k)

gMain.call(zoom.transform, t);
gMain.call(zoom)

Now we start with an already zoomed in view and can zoom in and out using the mouse.

To wrap up this post, let’s combine these techniques to create a figure which automatically zooms between random data points (a la M. Bostock’s Zoom Transitions Block). How do we do this?

First, we need a function to call every time we want to jump to a point:

    let targetPoint = 1010;

    function transition(selection) {
        let n = dataPoints.length;
        let prevTargetPoint = targetPoint;

        // pick a new point to zoom to
        while (targetPoint == prevTargetPoint) {
            let i = Math.random() * n | 0
            targetPoint = dataPoints[i];
        }

        selection.transition()
        .delay(300)
        .duration(2000)
        .call(zoom.transform, transform)
        .on('end', function() { circles.call(transition); });
    }

    circles.call(transition);

This function picks a random point (targetPoint) and calls a transition on the selection. In our case, the selection will be the circles. When the transition is over, we simply call the function again to start it over.

Second, we need a transform to center the view on the target point:

    function transform() {
        // put points that are 10 values apart 20 pixels apart
        var k = 20 / (xScale(10) - xScale(0))
        // center in the middle of the visible area
        var tx = (xScale.range()[1] + xScale.range()[0])/2 - k * xScale(targetPoint)
        var t = d3.zoomIdentity.translate(tx, 0).scale(k)
        return t;
    }

And that’s all. Just remember, when zooming and panning the position of the transformed point [x1,y1] = [tx + k × x0, ty + k × y0]. Everything else is just window dressing.