XUtils

minify

Fast minifiers for HTML, CSS, JS, XML, JSON and SVG file formats.

Golang
https://github.com/tdewolff/minify

Roadmap

  • [ ] Use ASM/SSE to further speed-up core parts of the parsers/minifiers
  • [x] Improve JS minifiers by shortening variables and proper semicolon omission
  • [ ] Speed-up SVG minifier, it is very slow
  • [x] Proper parser error reporting and line number + column information
  • [ ] Generation of source maps (uncertain, might slow down parsers too much if it cannot run separately nicely)
  • [ ] Create a cmd to pack webfiles (much like webpack), ie. merging CSS and JS files, inlining small external files, minification and gzipping. This would work on HTML files.

Prologue

Minifiers or bindings to minifiers exist in almost all programming languages. Some implementations are merely using several regular expressions to trim whitespace and comments (even though regex for parsing HTML/XML is ill-advised, for a good read see Regular Expressions: Now You Have Two Problems). Some implementations are much more profound, such as the YUI Compressor and Google Closure Compiler for JS. As most existing implementations either use JavaScript, use regexes, and don’t focus on performance, they are pretty slow.

This minifier proves to be that fast and extensive minifier that can handle HTML and any other filetype it may contain (CSS, JS, …). It is usually orders of magnitude faster than existing minifiers.

Docker

If you want to use Docker, please see https://hub.docker.com/r/tdewolff/minify.

$ docker run -it tdewolff/minify --help

Testing

For all subpackages and the imported parse package, test coverage of 100% is pursued. Besides full coverage, the minifiers are fuzz tested using github.com/dvyukov/go-fuzz, see the wiki for the most important bugs found by fuzz testing. These tests ensure that everything works as intended and that the code does not crash (whatever the input). If you still encounter a bug, please file a bug report!

Performance

The benchmarks directory contains a number of standardized samples used to compare performance between changes. To give an indication of the speed of this library, I’ve ran the tests on my Thinkpad T460 (i5-6300U quad-core 2.4GHz running Arch Linux) using Go 1.15.

name                              time/op
CSS/sample_bootstrap.css-4          2.70ms ± 0%
CSS/sample_gumby.css-4              3.57ms ± 0%
CSS/sample_fontawesome.css-4         767µs ± 0%
CSS/sample_normalize.css-4          85.5µs ± 0%
HTML/sample_amazon.html-4           15.2ms ± 0%
HTML/sample_bbc.html-4              3.90ms ± 0%
HTML/sample_blogpost.html-4          420µs ± 0%
HTML/sample_es6.html-4              15.6ms ± 0%
HTML/sample_stackoverflow.html-4    3.73ms ± 0%
HTML/sample_wikipedia.html-4        6.60ms ± 0%
JS/sample_ace.js-4                  28.7ms ± 0%
JS/sample_dot.js-4                   357µs ± 0%
JS/sample_jquery.js-4               10.0ms ± 0%
JS/sample_jqueryui.js-4             20.4ms ± 0%
JS/sample_moment.js-4               3.47ms ± 0%
JSON/sample_large.json-4            3.25ms ± 0%
JSON/sample_testsuite.json-4        1.74ms ± 0%
JSON/sample_twitter.json-4          24.2µs ± 0%
SVG/sample_arctic.svg-4             34.7ms ± 0%
SVG/sample_gopher.svg-4              307µs ± 0%
SVG/sample_usa.svg-4                57.4ms ± 0%
SVG/sample_car.svg-4                18.0ms ± 0%
SVG/sample_tiger.svg-4              5.61ms ± 0%
XML/sample_books.xml-4              54.7µs ± 0%
XML/sample_catalog.xml-4            33.0µs ± 0%
XML/sample_omg.xml-4                7.17ms ± 0%

name                              speed
CSS/sample_bootstrap.css-4        50.7MB/s ± 0%
CSS/sample_gumby.css-4            52.1MB/s ± 0%
CSS/sample_fontawesome.css-4      61.2MB/s ± 0%
CSS/sample_normalize.css-4        70.8MB/s ± 0%
HTML/sample_amazon.html-4         31.1MB/s ± 0%
HTML/sample_bbc.html-4            29.5MB/s ± 0%
HTML/sample_blogpost.html-4       49.8MB/s ± 0%
HTML/sample_es6.html-4            65.6MB/s ± 0%
HTML/sample_stackoverflow.html-4  55.0MB/s ± 0%
HTML/sample_wikipedia.html-4      67.5MB/s ± 0%
JS/sample_ace.js-4                22.4MB/s ± 0%
JS/sample_dot.js-4                14.5MB/s ± 0%
JS/sample_jquery.js-4             24.8MB/s ± 0%
JS/sample_jqueryui.js-4           23.0MB/s ± 0%
JS/sample_moment.js-4             28.6MB/s ± 0%
JSON/sample_large.json-4           234MB/s ± 0%
JSON/sample_testsuite.json-4       394MB/s ± 0%
JSON/sample_twitter.json-4        63.0MB/s ± 0%
SVG/sample_arctic.svg-4           42.4MB/s ± 0%
SVG/sample_gopher.svg-4           19.0MB/s ± 0%
SVG/sample_usa.svg-4              17.8MB/s ± 0%
SVG/sample_car.svg-4              29.3MB/s ± 0%
SVG/sample_tiger.svg-4            12.2MB/s ± 0%
XML/sample_books.xml-4            81.0MB/s ± 0%
XML/sample_catalog.xml-4          58.6MB/s ± 0%
XML/sample_omg.xml-4               159MB/s ± 0%

Whitespace removal

The whitespace removal mechanism collapses all sequences of whitespace (spaces, newlines, tabs) to a single space. If the sequence contained a newline or carriage return it will collapse into a newline character instead. It trims all text parts (in between tags) depending on whether it was preceded by a space from a previous piece of text and whether it is followed up by a block element or an inline element. In the former case we can omit spaces while for inline elements whitespace has significance.

Make sure your HTML doesn’t depend on whitespace between block elements that have been changed to inline or inline-block elements using CSS. Your layout should not depend on those whitespaces as the minifier will remove them. An example is a menu consisting of multiple <li> that have display:inline-block applied and have whitespace in between them. It is bad practise to rely on whitespace for element positioning anyways!

JS

The JS minifier typically shaves off about 35% – 65% of filesize depending on the file, which is a compression close to many other minifiers. Common speeds of PHP and JS implementations are about 100-300kB/s (see Uglify2, Adventures in PHP web asset minimization). This implementation is orders of magnitude faster at around ~25MB/s.

The following features are implemented:

  • remove superfluous whitespace
  • remove superfluous semicolons
  • shorten true, false, and undefined to !0, !1 and void 0
  • rename variables and functions to shorter names (not in global scope)
  • move var declarations to the top of the global/function scope (if more than one)
  • collapse if/else statements to expressions
  • minify conditional expressions to simpler ones
  • merge sequential expression statements to one, including into return and throw
  • remove superfluous grouping in expressions
  • shorten or remove string escapes
  • convert object key or index expression from string to identifier or decimal
  • merge concatenated strings
  • rewrite numbers (binary, octal, decimal, hexadecimal) to shorter representations

Options:

  • KeepVarNames keeps variable names as they are and omits shortening variable names
  • Precision number of significant digits to preserve for numbers, 0 means no trimming
  • Version ECMAScript version to use for output, 0 is the latest

Comparison with other tools

Performance is measured with time [command] ran 10 times and selecting the fastest one, on a Thinkpad T460 (i5-6300U quad-core 2.4GHz running Arch Linux) using Go 1.15.

  • minify: minify -o script.min.js script.js
  • esbuild: esbuild --minify --outfile=script.min.js script.js
  • terser: terser script.js --compress --mangle -o script.min.js
  • UglifyJS: uglifyjs --compress --mangle -o script.min.js script.js
  • Closure Compiler: closure-compiler -O SIMPLE --js script.js --js_output_file script.min.js --language_in ECMASCRIPT_NEXT -W QUIET --jscomp_off=checkVars optimization level SIMPLE instead of ADVANCED to make similar assumptions as do the other tools (do not rename/assume anything of global level variables)

Compression ratio (lower is better)

All tools give very similar results, although UglifyJS compresses slightly better.

Tool ace.js dot.js jquery.js jqueryui.js moment.js
minify 53.7% 64.8% 34.2% 51.3% 34.8%
esbuild 53.8% 66.3% 34.4% 53.1% 34.8%
terser 53.2% 65.2% 34.2% 51.8% 34.7%
UglifyJS 53.1% 64.7% 33.8% 50.7% 34.2%
Closure Compiler 53.4% 64.0% 35.7% 53.6% 34.3%

Time (lower is better)

Most tools are extremely slow, with minify and esbuild being orders of magnitudes faster.

Tool ace.js dot.js jquery.js jqueryui.js moment.js
minify 49ms 5ms 22ms 35ms 13ms
esbuild 64ms 9ms 31ms 51ms 17ms
terser 2900s 180ms 1400ms 2200ms 730ms
UglifyJS 3900ms 210ms 2000ms 3100ms 910ms
Closure Compiler 6100ms 2500ms 4400ms 5300ms 3500ms

JSON

Minification typically shaves off about 15% of filesize for common indented JSON such as generated by JSON Generator.

The JSON minifier only removes whitespace, which is the only thing that can be left out, and minifies numbers (1000 => 1e3).

Options:

  • Precision number of significant digits to preserve for numbers, 0 means no trimming
  • KeepNumbers do not minify numbers if set to true, by default numbers will be minified

SVG

The SVG minifier uses these minifications:

  • trim and collapse whitespace between all tags
  • strip comments, empty doctype, XML prelude, metadata
  • strip SVG version
  • strip CDATA sections wherever possible
  • collapse tags with no content to a void tag
  • minify style tag and attributes with the CSS minifier
  • minify colors
  • shorten lengths and numbers and remove default px unit
  • shorten path data
  • use relative or absolute positions in path data whichever is shorter

TODO:

  • convert attributes to style attribute whenever shorter
  • merge path data? (same style and no intersection – the latter is difficult)

Options:

  • Precision number of significant digits to preserve for numbers, 0 means no trimming

XML

The XML minifier uses these minifications:

  • strip unnecessary whitespace and otherwise collapse it to one space (or newline if it originally contained a newline)
  • strip comments
  • collapse tags with no content to a void tag
  • strip CDATA sections wherever possible

Options:

  • KeepWhitespace preserve whitespace between inline tags but still collapse multiple whitespace characters into one

New

Retrieve a minifier struct which holds a map of mediatype → minifier functions.

m := minify.New()

The following loads all provided minifiers.

m := minify.New()
m.AddFunc("text/css", css.Minify)
m.AddFunc("text/html", html.Minify)
m.AddFunc("image/svg+xml", svg.Minify)
m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

You can set options to several minifiers.

m.Add("text/html", &html.Minifier{
	KeepDefaultAttrVals: true,
	KeepWhitespace: true,
})

From reader

Minify from an io.Reader to an io.Writer for a specific mediatype.

if err := m.Minify(mediatype, w, r); err != nil {
	panic(err)
}

From bytes

Minify from and to a []byte for a specific mediatype.

b, err = m.Bytes(mediatype, b)
if err != nil {
	panic(err)
}

From string

Minify from and to a string for a specific mediatype.

s, err = m.String(mediatype, s)
if err != nil {
	panic(err)
}

To reader

Get a minifying reader for a specific mediatype.

mr := m.Reader(mediatype, r)
if _, err := mr.Read(b); err != nil {
	panic(err)
}

To writer

Get a minifying writer for a specific mediatype. Must be explicitly closed because it uses an io.Pipe underneath.

mw := m.Writer(mediatype, w)
if mw.Write([]byte("input")); err != nil {
	panic(err)
}
if err := mw.Close(); err != nil {
	panic(err)
}

Middleware

Minify resources on the fly using middleware. It passes a wrapped response writer to the handler that removes the Content-Length header. The minifier is chosen based on the Content-Type header or, if the header is empty, by the request URI file extension. This is on-the-fly processing, you should preferably cache the results though!

fs := http.FileServer(http.Dir("www/"))
http.Handle("/", m.Middleware(fs))

Custom minifier

Add a minifier for a specific mimetype.

type CustomMinifier struct {
	KeepLineBreaks bool
}

func (c *CustomMinifier) Minify(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {
	// ...
	return nil
}

m.Add(mimetype, &CustomMinifier{KeepLineBreaks: true})
// or
m.AddRegexp(regexp.MustCompile("/x-custom$"), &CustomMinifier{KeepLineBreaks: true})

Add a minify function for a specific mimetype.

m.AddFunc(mimetype, func(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {
	// ...
	return nil
})
m.AddFuncRegexp(regexp.MustCompile("/x-custom$"), func(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {
	// ...
	return nil
})

Add a command cmd with arguments args for a specific mimetype.

m.AddCmd(mimetype, exec.Command(cmd, args...))
m.AddCmdRegexp(regexp.MustCompile("/x-custom$"), exec.Command(cmd, args...))

Mediatypes

Using the params map[string]string argument one can pass parameters to the minifier such as seen in mediatypes (type/subtype; key1=val2; key2=val2). Examples are the encoding or charset of the data. Calling Minify will split the mimetype and parameters for the minifiers for you, but MinifyMimetype can be used if you already have them split up.

Minifiers can also be added using a regular expression. For example a minifier with image/.* will match any image mime.

Examples

Common minifiers

Basic example that minifies from stdin to stdout and loads the default HTML, CSS and JS minifiers. Optionally, one can enable java -jar build/compiler.jar to run for JS (for example the ClosureCompiler). Note that reading the file into a buffer first and writing to a pre-allocated buffer would be faster (but would disable streaming).

package main

import (
	"log"
	"os"
	"os/exec"

	"github.com/tdewolff/minify/v2"
	"github.com/tdewolff/minify/v2/css"
	"github.com/tdewolff/minify/v2/html"
	"github.com/tdewolff/minify/v2/js"
	"github.com/tdewolff/minify/v2/json"
	"github.com/tdewolff/minify/v2/svg"
	"github.com/tdewolff/minify/v2/xml"
)

func main() {
	m := minify.New()
	m.AddFunc("text/css", css.Minify)
	m.AddFunc("text/html", html.Minify)
	m.AddFunc("image/svg+xml", svg.Minify)
	m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

	if err := m.Minify("text/html", os.Stdout, os.Stdin); err != nil {
		panic(err)
	}
}

External minifiers

Below are some examples of using common external minifiers.

Closure Compiler

See Closure Compiler Application. Not tested.

m.AddCmdRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"),
    exec.Command("java", "-jar", "build/compiler.jar"))

UglifyJS

See UglifyJS.

m.AddCmdRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"),
    exec.Command("uglifyjs"))

esbuild

See esbuild.

m.AddCmdRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"),
    exec.Command("esbuild", "$in.js", "--minify", "--outfile=$out.js"))

Custom minifier

Custom minifier showing an example that implements the minifier function interface. Within a custom minifier, it is possible to call any minifier function (through m minify.Minifier) recursively when dealing with embedded resources.

package main

import (
	"bufio"
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/tdewolff/minify/v2"
)

func main() {
	m := minify.New()
	m.AddFunc("text/plain", func(m *minify.M, w io.Writer, r io.Reader, _ map[string]string) error {
		// remove newlines and spaces
		rb := bufio.NewReader(r)
		for {
			line, err := rb.ReadString('\n')
			if err != nil && err != io.EOF {
				return err
			}
			if _, errws := io.WriteString(w, strings.Replace(line, " ", "", -1)); errws != nil {
				return errws
			}
			if err == io.EOF {
				break
			}
		}
		return nil
	})

	in := "Because my coffee was too cold, I heated it in the microwave."
	out, err := m.String("text/plain", in)
	if err != nil {
		panic(err)
	}
	fmt.Println(out)
	// Output: Becausemycoffeewastoocold,Iheateditinthemicrowave.
}

ResponseWriter

Middleware

func main() {
	m := minify.New()
	m.AddFunc("text/css", css.Minify)
	m.AddFunc("text/html", html.Minify)
	m.AddFunc("image/svg+xml", svg.Minify)
	m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
	m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

	fs := http.FileServer(http.Dir("www/"))
	http.Handle("/", m.MiddlewareWithError(fs))
}

func handleError(w http.ResponseWriter, r *http.Request, err error) {
    http.Error(w, err.Error(), http.StatusInternalServerError)
}

In order to properly handle minify errors, it is necessary to close the response writer since all writes are concurrently handled. There is no need to check errors on writes since they will be returned on closing.

func main() {
	m := minify.New()
	m.AddFunc("text/html", html.Minify)
	m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)

	input := `<script>const i = 1_000_</script>` // Faulty JS
	req := httptest.NewRequest(http.MethodGet, "/", nil)
	rec := httptest.NewRecorder()
	m.Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/html")
		_, _ = w.Write([]byte(input))

		if err = w.(io.Closer).Close(); err != nil {
			panic(err)
		}
	})).ServeHTTP(rec, req)
}

ResponseWriter

func Serve(w http.ResponseWriter, r *http.Request) {
	mw := m.ResponseWriter(w, r)
	defer mw.Close()
	w = mw

	http.ServeFile(w, r, path.Join("www", r.URL.Path))
}

FAQ

Newlines remain in minified output

While you might expect the minified output to be on a single line for it to be fully minified, this is not true. In many cases, using a literal newline doesn’t affect the file size, and in some cases it may even reduce the file size.

A typical example is HTML. Whitespace is significant in HTML, meaning that spaces and newlines between or around tags may affect how they are displayed. There is no distinction between a space or a newline and they may be interchanged without affecting the displayed HTML. Remember that a space (0x20) and a newline (0x0A) are both one byte long, so that there is no difference in file size when interchanging them. This minifier removes unnecessary whitespace by replacing stretches of spaces and newlines by a single whitespace character. Specifically, if the stretch of white space characters contains a newline, it will replace it by a newline and otherwise by a space. This doesn’t affect the file size, but may help somewhat for debugging or file transmission objectives.

Another example is JavaScript. Single or double quoted string literals may not contain newline characters but instead need to escape them as \n. These are two bytes instead of a single newline byte. Using template literals it is allowed to have literal newline characters and we can use that fact to shave-off one byte! The result is that the minified output contains newlines instead of escaped newline characters, which makes the final file size smaller. Of course, changing from single or double quotes to template literals depends on other factors as well, and this minifier makes a calculation whether the template literal results in a shorter file size or not before converting a string literal.

Articles

  • coming soon...