Weekend Website Experiment

As you may know if you read this blog via Planet GNOME, the GNOME project is busy switching to GitLab for its code hosting and bug tracking. I like GitLab! It’s a large step up from Bugzilla, which was what GNOME used for the last 20 years. Compared to GitHub, GitLab is about equal, with a few nicer things and a few less nice things.

The one thing that I miss from Bugzilla is a dashboard showing the overall status of the bugs for your project. I thought it would not be too hard to use the GitLab API to do some simple queries and plop them on a web page. So, last weekend I gave it a try. The final result is here. Click the button to log into GitLab, and you’ll be redirected back to the page where you’ll get the results of the queries.

I’d like to write up what I did because I learned a new thing, and I think more writeups illustrating the trial and error of learning a new thing are always good.

My goals were to write something without being too meticulous, and write something that was not intended to scale. (Both are things that I normally disapprove of. It’s good to try the other side once in a while.) So, I was just going to mix the HTML, CSS, and JS all in one file. I used the base CSS and overall page structure from my personal website.

I decided to use GitHub Pages for hosting. My personal website is already hosted there. GitLab also offers GitLab Pages, which is very similar, but GNOME hasn’t enabled it yet on their GitLab instance. So, I created a fork of GNOME/gjs on GitHub, and created a gh-pages branch. Whatever you commit to that branch shows up as your project’s GitHub Pages site on the web.

The first thing I figured out is that you have to be authenticated to query issues in the API, even if the same information is publicly available on the web. That’s too bad! My little project suddenly went from “easy” to “figuring out something I’ve never done before.” But that’s also exciting.

First, I got the queries running on a local webpage, using a temporary personal GitLab access token. Each item looked kind of like this:

Number of crashers: 16
Number of bugs: 25
etc.

Next, I decided to tackle the authentication problem. I did some searches on variations of “gitlab authentication plugin”, “add gitlab authentication to webpage”, etc., to see if there was something ready-made I could drop in. No such luck. I did find NodeJS modules that I could have used, had I been writing the site using NodeJS. I weighed the unknown cost of implementing the authentication in plain old browser JS against the unknown cost of setting up tools that I was unfamiliar with in order to use the ready-made module (I wasn’t even sure what tools I would have to use — Webpack, I think?) and decided to keep looking.

I next searched for things like “gitlab oauth2 in browser”, “gitlab oauth2 example”, since I knew that the login used the OAuth2 protocol. Eventually I landed on this page and figured out that the magic words I wanted were “implicit flow” or “implicit grant”:

Implicit Flow – This flow is designed for user-agent only apps (e.g. single page web application running on GitLab Pages).

That sounded like exactly my use case, so I read further. You have to send the user to a certain page on the GitLab site, and send a redirect URL which the user will be sent back to with the authorization token in the URL hash. I managed to keep everything on the same webpage. In pseudocode, the flow looks like this:

if we have the authorization token:
    fetch the numbers with the API calls
else if there is a hash in the URL:
    token = get the token from the hash
    store the token
    fetch the numbers with the API calls
else:
    show a button that links to the authorization URL on GitLab

For storing the token so that you don’t have to log in every time, I used localStorage. I have no idea if that’s good practice or not, but from what I could read online it seems that it’s at least not bad practice. It’s quite easy to retrieve the token, but only if you have access to the browser where it is stored. I don’t think localStorage can leak out over the web, but with the recently discovered vulnerabilities who knows…

Last, I made it look nicer. I had a pretty good idea of what I wanted it to look like: I wanted the numbers to be large, in colored boxes with rounded corners and thick borders. I tried a few things with floating <div>s before giving up and using a CSS flex layout. This makes the page probably unviewable on older browsers, but I was seriously done with CSS positioning.

The code is here, or just “View Source” while you’re on the page.

What I would do differently

Writing HTML, CSS, and JS directly for the web is tedious and repetitive. I wish my younger self had used some sort of framework like Bootstrap to make my personal site. Failing that, I wish I had decided not to reuse components from my personal site to do this, and instead started with a fresh site using a framework. Bootstrap or Semantic UI are two that I know of, and maybe should have tried out. The code ended up being 263 lines of HTML, CSS, and JS, much of it just repeated items in order to do the boxes in different colors.

Reuse of this code

You may notice I did not release this code under an open source license. That’s because it’s probably full of bad practices, so I don’t want people to copy it. If you can convince me that it’s done right or tell me what I did wrong, then I’ll (fix it and) open-source it, and other GitLab maintainers might find it useful.

New Year, New Blogs

Here are three clear thinkers whose blogs I’ve enjoyed discovering over the past year. Maybe you might like them too.

Tech

Julia Evans takes a complicated topic that she’d like to learn about, and just… goes and learns about it. Then she posts her findings to her blog, written in a really accessible way. She does this mainly for technical topics, but sometimes also tech leadership skills.

Completely true to form, she’s currently on a sabbatical from her job to write a profiler for Ruby. At the time of this writing, she’s publishing a weekly post on what she learned each week doing this project.

She also produces zines: short, handwritten, comic-book style explanations of technical subjects. I managed to get my hands on a paper copy of So You Want To Be A Wizard, which is a collection of tips about building up your problem-solving skills as a software engineer. The zines are also available to read for free on her website.

Julia Evans’ writing style is really what I aspire to on this blog, I just never knew it before. She takes complicated topics and demystifies them, and reading about them really makes you feel like you too can get your head around difficult things if you can just conquer your hesitation and dive in.

Politics

Benjamin Studebaker is politically a lot farther to the left than I am, and actually has written a certain number of articles that I strongly disagree with. There’s nothing that’s not well-thought-out, though, and sometimes it’s good to read things you disagree with.

However, I’ve learned a few things from this blog. One is what he calls “the core left-wing premise”: People’s actions are shaped by conditions. In other words, the left-wing approach to fighting poverty is to ask the question “How can we change the conditions in our society to make it possible for poor people to have the opportunities they need?” whereas the right-wing approach is to ask “How can we make poor people take responsibility for themselves?”

The most thought-provoking thing I’ve read here is the need to apply the core left-wing premise consistently — even to realize that we need to change the systems in our society that cause people to find various -isms (such as racism) attractive, and the -ists themselves will follow, whereas an aggressive approach will only cause the -ists to entrench their views. In Benjamin Studebaker’s words:

[W]hen we tell racists to “educate themselves” we’re no different from the conservatives who tell the homeless guy they see on the corner to “get a job”.

Media

Mike Caulfield has a blog that defies categorization. I’m calling it “media” because that seems to be the common thread. He writes a lot about one topic for a while, then moves on to another topic. (I’m actually cheating a bit because I got into this blog a few years ago when he was writing about Federated Wiki, then he moved on to the garden model versus the feed model, and on to shared resources. But it’s like a whole different blog every year!)

This past year he’s moved on to the topic of fact-checking and polarization on social media. It’s really worth going back and reading posts from the beginning of 2017, since there are too many good ones to put in just a list of highlights. The short of it is that he has written a lot about both the technical and social aspects about why ultra-polarized fake news is taking over social media platforms, why the companies behind these platforms have no incentive to change that, and the skills that we as consumers need to protect ourselves from falling into the fake news trap. One thing I especially appreciate is that he tries hard to be apolitical by including examples of fake news from all over the political spectrum.

He recently published a post of “Predictions for 2018” that in turn make me predict that his topic for 2018 will be clickbait content generated by machine learning algorithms…

Javascript news from GNOME 3.24

Welcome back to the latest news on GJS, the Javascript engine that powers GNOME Shell, Endless OS, Polari, GNOME Documents, and many other apps.

GNOME 3.24 has been released for about three weeks now, and with it went GJS 1.48.0. Here’s what’s new!

Javascript upgrade!

First of all, we have a more modern Javascript engine. GJS is based on Mozilla’s SpiderMonkey, the same Javascript engine that runs in the Firefox browser. Back in GNOME 3.22, GJS was based on version 24, which was released in September 2013. Now we’ve moved to version 38, which although still old, was released almost two years later in May 2015.

(The number of each SpiderMonkey release increases by 7 each time, because they make a standalone SpiderMonkey release for each Extended Support Release of Firefox, which is one out of every 7. That’s why you might also hear them referred to as “ESR 38”, etc.)

This brings a lot of new Javascript language features with it. Here are some of the ones I’m most excited about.

Promises

Promises allow you to do asynchronous operations (like reading files, or waiting, or fetching things from the network) in a much more intuitive way. With Promises, the code reads from top to bottom as if it were synchronous, instead of from nested level to nested level (often called “callback hell“.)

Here’s an example, a Promises version of examples/gio-cat.js that’s included in GJS’s source distribution:

	const GLib = imports.gi.GLib;
	const Gio = imports.gi.Gio;
	function wrapPromise(obj, cancellable, asyncName, finishName, …inArgs) {
	return new Promise((resolve, reject) => {
	let callerStack = new Error().stack
	.split('\n')
	.filter(line => !line.match(/<Promise>|wrapPromise/))
	.join('\n');
	obj[asyncName](…inArgs, cancellable, (obj, res) => {
	try {
	let results = obj[finishName](res);
	resolve(results);
	} catch (e) {
	e.stack += '— Called from: —\n' + callerStack;
	reject(e);
	}
	});
	});
	}
	function loadContents(file, cancellable=null) {
	return wrapPromise(file, cancellable, 'load_contents_async',
	'load_contents_finish');
	}
	// ————————-
	let loop = GLib.MainLoop.new(null, false);
	function cat(filename) {
	let f = Gio.File.new_for_commandline_arg(filename);
	loadContents(f)
	.then(([ok, contents]) => {
	print(contents);
	})
	.catch(e => {
	logError(e);
	})
	.then(() => loop.quit());
	loop.run();
	}
	if (ARGV.length != 1) {
	printerr("Usage: gio-cat.js filename");
	} else {
	cat(ARGV[0]);
	}

view raw

gio-cat.js

hosted with ❤ by GitHub

This is much longer than the original program, but only the lower part of the program is actually the equivalent of the old callback-based code. The top part would ideally be provided by GJS itself. I’m still figuring out what is the best API for wrapPromise but it’s definitely a candidate for including in a future version of GJS.

This code calls loadContents, prints the contents, and exits the main loop. If an exception is thrown anywhere in the chain before .catch, then the function provided to the catch call will log the error message. In any case, no matter whether the operation succeeded or not, the last then call will make sure the main loop exits.

Template literals

Template literals will change your life if you work with text in your GJS program. They are regular strings in backticks, with interpolation. Say goodbye to this:

const Format = imports.format;
String.prototype.format = Format.format;
log("%s, %s!".format(greeting, name));

Also say goodbye to this:

log(greeting + ", " + name + "!");

Instead, from now on you’ll do this:

log(`${greeting}, ${name}!`);

It’s a lot more readable and intuitive.

Template literals can also cover more than one line, and they do real interpolation of expressions too, not just variable names:

const CSS = `
label {
    font-size: ${fontdesc.get_size()};
}`;

You can also “tag” templates which is out of scope of this blog post, but there is one built-in tag which serves the same purpose as r'' string literals do in Python:

String.raw`I'm writing some \LaTeX\ code here
and I \textbf{don't} want to deal with escaping it:
\[ E = mc^2 \]`

Generators

Generators are a great addition to the Javascript toolbox. They were actually already available in GJS, but only in Mozilla’s nonstandard extension form. They are introduced with the function* keyword instead of function, and they work a lot like Python’s generators. Here’s an example, implementing the xrange() function similar to the one in Python using a generator:

function* xrange(limit) {
    for(let count = 0; count < limit; count++)
        yield count;
}

The yield statement returns control back to the caller, while preserving the state of the generator until the next call. You can get all the values one by one, calling a generator’s next() method, but for...of loops will also deal with generators:

for (let ix of xrange(5))
    print(`Counting from 0 to 4: ${ix}`);

If you want to empty a generator into an array, you can also use the spread operator: [...xrange(5)] will give you an array of numbers from 0 to 4.

Here’s a more complicated example showing the yield* statement which allows you to compose more than one generator:

	const Gio = imports.gi.Gio;
	function* leafnodes(file) {
	let enumerator = file.enumerate_children('standard::*', 0, null);
	let info;
	while ((info = enumerator.next_file(null))) {
	let child = enumerator.get_child(info);
	if (info.get_file_type() === Gio.FileType.DIRECTORY)
	yield* leafnodes(child);
	else
	yield child.get_basename();
	}
	}
	let file = Gio.File.new_for_commandline_arg(ARGV[0]);
	for (let leaf of leafnodes(file))
	print(leaf);

view raw

leafnode.js

hosted with ❤ by GitHub

This code prints looks at the directory that it’s given, and prints all the files in it that are not themselves directories (the “leaf nodes”.) If one of the files is a directory, it will descend into that directory and repeat the process, thanks to yield*.

Want to know more?

Since there’s a lot more than I can cover in a comfortably readable blog post, I made a slide deck. I tried to put it together in such a way that you can use it as reference material.

For more information on all of these cool things, I highly recommend this “ES6 Explained” series of posts from the Mozilla Hacks blog. Some of these features, such as classes and modules, are still to come in GJS.

Maintainer life

The Javascript engine upgrade was the major feature, but I also spent some time on making things easier for myself as the maintainer. A well-tended garden will hopefully attract more gardeners. Happily, some other people joined in for this part.

I cleaned up the build system, using more modern and concise Autotools code. I also spent some time cleaning up compiler warnings, both on GCC and Clang. Now the build and test runs are faster, and the cleaner output makes it much easier to see when something goes wrong. I also made sure that GJS builds on macOS, or at least it did until my Apple hardware broke down. Chun-wei Fan made some improvements that ensure GJS builds on Windows with MSVC. Claudio André implemented continuous integration in a Docker container, with the intention to run it on Travis CI, but sadly we do not have permission to flip the bit to get Travis to build it.

Having written Jasmine GJS in order to bring some of that convenient unit testing technology from the Node world into GJS applications, I also wanted to use it for writing GJS’s own unit tests. I couldn’t use it directly because that would have been a circular dependency, of course, but I embedded a copy of upstream Jasmine plus a very stripped-down version of Jasmine GJS, and called it “Minijasmine”. It’s now a lot easier, and dare I say less of a drag, to write unit tests for GJS. Accordingly, we’ll now try to cover every bug fix with a regression test.

And I worked on getting the bug tracker down to a less daunting number of bugs. It was fun to make the bug chart in my last post, so here’s another one: this is the number of open bugs during the release cycle from 1.46.0 to 1.48.0.

You can definitely see that November Bug Squash Month had an effect

Unfortunately the chart will not look like this again next time around. The big drop was me closing all the obsolete or already-fixed bugs during November Bug Squash Month. We are down from about 160 to about 100 bugs, but those were all the easy ones; there are only hard ones left now.

Thanks

Thanks to everyone who participated to bring GJS to GNOME 3.24: Chun-wei Fan, Claudio André, Florian Müllner, Alexander Larsson, Iain Lane, Jonh Wendell, and Lionel Landwerlin.

As well, this release incorporated a lot of patches that people contributed a long time ago, even up to 8 years, that for various reasons had not been reviewed yet. (Many from emeritus GJS maintainers!) Thanks to those people for participating in the past, and I’m glad we were able to finally bring your contributions into the project: Giovanni Campagna, Jasper St. Pierre, Sam Spilsbury, Havoc Pennington, Joe Shaw, Paolo Borelli, Shawn Walker, and Tim Lunn.

Luke Jones and Hussam Al-Tayeb identified a serious memory leak right before the final 1.48.0 release and without their contribution, it would have been a different and much sadder story. As it was, 1.48.0 still contained another serious bug that made GNOME Shell quite unusable for an unlucky few people. Thanks to Georges Stavracas for rewriting a happy ending for 1.48.2.

Special thanks to Cosimo Cecchi, for reviewing almost every single line of the code I wrote for this release: about 20000 lines, many of them boring and repetitive.

Thanks also to my employer Endless which sponsored most of the Javascript engine upgrade, and a good chunk of miscellaneous bug fixing time.

Looking forward

My next post will be about what’s to come in GJS for GNOME 3.26.