Shawn Dahlen

Periodic updates on my software startup endeavor

4dashes - the Build System

| Comments

This is the first in a series of posts discussing the implementation of the 4dashes productivity tool. It covers the project structure and build system supporting a rapid developer workflow and creation of production assets. It is a continuation on the groundwork discussed here representing the state of the product at launch.

4dashes is a web-based product built on Node.js and Angular (deviating from the earlier setup using Backbone). Like many Node.js projects nowadays, the build system is based on Grunt. Before discussing the Gruntfile mechanics, its beneficial to understand the project file structure. Here is the top-level:

1
2
3
4
5
6
7
8

4dashes/
  app/
  lib/
  test/
  public/
  Gruntfile.js
  package.json
  server.js

The server.js file loads configuration, sets up logging, and mounts RESTful endpoints defined within modules in the lib/ directory. The details of this will be discussed in a future article. The app/ directory contains the client-side assets that comprise the Angular-based web application and which are ultimately built to the public/ directory to serve for production.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

4dashes/
  app/
      components/
          timer/
              timer.html
              timer.less
              timer.js
          charts/
          task-list/
          ...
      images/
      fonts/
          icons.woff
      index.html
      app.html
      app.less
      app.js

Unlike other Angular projects, 4dashes was organized around the functional components that comprise the application (similar to what was recently proposed by the Angular team.) This approach aligns well with the upcoming web components technology. Each component is contained within its own directory with a file for template markup, styles, and script. The app.* files represent the entry point for the application and dynamically reference files for each component. When a new component is created (or removed), the build system updates the appropriate references. This is accomplished with the grunt-file-blocks plugin.

The plugin works by updating replacement blocks within a source file with references based on file matching patterns. app.html contains replacement blocks for third-party dependencies and component scripts:

app.html 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

<html>
  <body>
      <header>...</header>
      <div id="content" ng-view></div>
      <footer>...</footer>

    <!-- build:js app.js -->
      <!-- fileblock:js dependencies -->
      <script src="bower_components/angular/angular.min.js"></script>
      <script src="bower_components/moment/moment.js"></script>
      ...
      <!-- endfileblock -->

      <!-- fileblock:js components -->
      <script src="components/api/api.js"></script>
      <script src="components/avatar/avatar.js"></script>
      <script src="components/timer/timer.js"></script>
      <script src="components/templates.js"></script>
      ...
      <!-- endfileblock -->

      <script src="app.js"></script>
    <!-- endbuild -->
  </body>
</html>

app.less follows the same pattern replacing a comment block with @import statements for components styles:

app.less 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// import base bootstrap styles
@import 'bower_components/bootstrap/less/bootstrap.less';

// define color palette
@base-color: hsl(45, 15%, 99%);
@base-dark-color: hsl(45, 15%, 89%);

// additional common styles
...

/* fileblock:less components */
@import 'components/avatar/avatar.less';
@import 'components/timer/timer.less';
...
/* endfileblock */

Finally, component template markup is pre-processed and compiled into a single templates.js file using the grunt-angular-templates plugin. While this wasn’t technical required for a development workflow it did simplify references within unit testing code.

The supporting Grunt configuration for these two plugins is as follows:

Gruntfile.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

module.exports = function (grunt) {
  require('matchdep').filterDev('grunt-*').forEach(grunt.loadNpmTasks)

  var dependencies = [ ... ]

  grunt.initConfig({
      fileblocks: {
          options: {
              removeFiles: true,
              templates: {
                  less: '@import \'${file}\';'
              }
          },
          js: {
              src: 'app/app.html',
              blocks: {
                  dependencies: { cwd: 'app', src: dependencies },
                  components: { cwd: 'app', src: 'components/**/*.js' }
              }
          },
          less: {
              src: 'app/app.less',
              blocks: {
                  components: { cwd: 'app', src: 'components/**/*.less' }
              }
          }
      },
      ngtemplates: {
          options: {
              prefix: 'template/',
              standalone: true,
              htmlmin: { collapseWhitespace: true }
          },
          templates: {
              cwd: 'app/components',
              src: '**/*.html',
              dest: 'app/components/templates.js'
          }
      }
  })
}

This setup eliminated the need to manually update referenced assets and provided the basis for a rapid development workflow and a streamlined production build.

During development, assets are served from the app/ directory. A custom Grunt task was registered to start the server and reload on both client- and server-side changes. For client-side changes, less files and angular templates are processed and the browser is refreshed using LiveReload. For server-side changes, the Node.js server process is restarted. During development, the developer simply runs grunt dev on the command line, points a browser at http://localhost:8080, and edits/saves code to view the changes.

The following Grunt configuration supports this workflow. It leverages grunt-contrib-watch and grunt-nodemon to watch and reload client- and server-side resources respectively. grunt-concurrent allows these two plugins to run concurrently.

Gruntfile.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

module.exports = function (grunt) {
  require('matchdep').filterDev('grunt-*').forEach(grunt.loadNpmTasks)

  var dependencies = [ ... ]

  grunt.initConfig({
      concurrent: {
          dev: {
              tasks: ['nodemon', 'watch'],
              options: {
                  logConcurrentOutput: true
              }
          }
      },
      nodemon: {
          dev: {
              options: {
                  file: 'server.js',
                  ignoredFiles: ['app', 'test', 'node_modules'],
                  watchedExtensions: ['js']
              }
          }
      },
      watch: {
          app: {
              options: { livereload: true },
              files: [
                  'app/app.*',
                  'app/components/**/*.js',
                  'app/images/*'
              ]
          },
          templates: {
              tasks: ['ngtemplates'],
              files: ['app/components/**/*.html']
          },
          less: {
              tasks: ['less'],
              files: ['app/app.less', 'app/components/**/*.less']
          }
      },
      less: {
          build: {
              files: { 'app/app.css': 'app/app.less' }
          }
      }
  })

  grunt.registerTask('dev', [
      'ngtemplates',
      'fileblocks:js',
      'fileblocks:less',
      'less',
      'concurrent'
  ])

For production, the build system uses the excellent grunt-usemin plugin to replace non-optimized scripts and stylesheets in app.html with a reference to a single optimized app.js and app.css file. This is combined with the grunt-rev plugin for asset revisioning to support cache busting when changes are made. Ultimately, the application consists of three core files, app.html, app.js, and app.css with the later two served with an explicit cache header.

To round out the production build, grunt-ngmin is used to pre-minify Angular scripts so that minifers appropriately handle the dependency injection style used by the framework.

Gruntfile.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

module.exports = function (grunt) {
  require('matchdep').filterDev('grunt-*').forEach(grunt.loadNpmTasks)

  var dependencies = [ ... ]

  grunt.initConfig({
      clean: {
          build: ['public'],
          templates: ['app/templates/**/*.js'],
          less: ['app/styles/*.css'],
          test: ['test/**/*.log']
      },
      copy: {
          fonts: {
              expand: true,
              cwd: 'app/fonts',
              src: '**',
              dest: 'public/fonts/',
              flatten: true,
              filter: 'isFile'
          },
          sounds: {
              expand: true,
              cwd: 'app/sounds',
              src: '**',
              dest: 'public/sounds/',
              flatten: true,
              filter: 'isFile'
          }
      },
      useminPrepare: {
          options: { dest: 'public' },
          html: 'app/*.html'
      },
      usemin: {
          html: 'public/*.html',
          css: ['public/*.css', 'public/index.html'],
          js: ['public/*.js'],
          options: {
              patterns: {
                  js: [[
                      /["']([^:"']+\.(?:png|gif|jpe?g|css|js))["']/img,
                      'Update JavaScript with assets in strings'
                  ]]
              }
          }
      },
      htmlmin: {
          build: {
              files: [{
                  expand: true,
                  cwd: 'app',
                  src: '*.html',
                  dest: 'public'
              }]
          }
      },
      imagemin: {
          build: {
              files: [{
                  expand: true,
                  cwd: 'app/images',
                  src: '*.{jpg,png}',
                  dest: 'public/images'
              }]
          }
      },
      ngmin: {
          build: {
              files: { 'public/app.js': 'public/app.js' }
          }
      },
      rev: {
          build: {
              src: ['public/**/*.{js,css,png,jpg,woff}']
          }
      }
  })

  grunt.registerTask('build', [
      'clean',
      'fileblocks:js',
      'fileblocks:less',
      'less',
      'copy',
      'useminPrepare',
      'ngtemplates',
      'concat',
      'htmlmin',
      'imagemin',
      'cssmin',
      'ngmin',
      'uglify',
      'rev',
      'usemin'
  ])

To conclude, a new production server can be run with the following commands:

1
2
3
4
5
6

$ git clone 4dashes.git
$ cd 4dashes
$ npm install --production
$ bower install --production
$ grunt build
$ NODE_ENV=production node server.js

Comments