GJS: What’s next?

In my last post, I went into detail about all the new stuff that GJS brought to GNOME 3.24. Now, it’s time to talk about the near future: what GJS will bring to GNOME 3.26.

Javascript engine

The highest priority is to keep upgrading the Javascript engine. At the time of writing, I’ve got SpiderMonkey 45 almost, but not quite, working, and Mozilla is on the verge of releasing the standalone version of SpiderMonkey 52. If we can get there, then we’ll finally be on a supported release which means we can have a closer collaboration with the Mozilla team. (During the past six months, they’ve been patient with me asking questions about old, unsupported releases, but it’s not fair to ask them to continue doing that.)

I plan to upgrade to 45 but not merge it, and then immediately continue upgrading to 52 on the same branch, then merge it all in at the same time. That way, we won’t have an interregnum where everyone has to build SpiderMonkey 45 in JHBuild and Continuous and the Flatpak SDK. Subscribe to bug 781429 and its offshoots if you want to follow along.

The main language features that this will bring in are: classes (45) and ES7 async/await statements (52). At that point, the only major ES6 feature that we will still be missing is modules.

ES6 Classes

After that is done, I will refactor GJS’s class system (Lang.Class and GObject.Class). I believe this needs to be done before GNOME 3.26. That’s because in SpiderMonkey 45, we gain ES6 classes, and I want to avoid the situation where we have two competing, and possibly incompatible, ways to write classes.

ES6 Modules

Full ES6 module support is still missing in SpiderMonkey 52, but at least some parts of it are implemented. I’ll need to investigate if it’s possible to enable them in GJS already. Although, we will definitely not enable them yet if there’s no way to keep the existing modules working; we don’t want to break everyone’s code.

Developer tools

Next comes a debugger. There are not one, not two, but three existing implementations of a GJS debugger sitting unattended in Bugzilla or a Git branch. None of them will apply to the codebase as is, so my task will be to fix them up, evaluate the merits of each one, and hopefully come up with one patchset to rule them all.

Christian Hergert is planning to add a profiler, so that you can profile your Javascript code with Sysprof, inside Builder.

Documentation

I would very much like to get the GJS documentation browser back online. I hosted it on EC2, but I have run out of free hosting. If you have a server where it can be parked, let me know! (It’s a web app, not static pages, so I can’t just put it on GitHub Pages.) If you want to run the web app locally yourself, you can find instructions here for how I set it up on EC2, on a RHEL 7 box.

Misc.

All that is probably more than I’ll have time for, but here are some of the things that I’d like to get done after that:

  • Update the tutorials on developer.gnome.org to use more modern GJS
  • Better integration with Builder
  • Use structured logging to clean up the “debug topics” mechanism
  • Reduce the list of unreviewed patches in Bugzilla down to 0
  • Find ways to bring in some of the conveniences that Node developers are used to

Chun-wei Fan is working on converting some of the codebase to use C++ smart pointers so that we get the memory leak safety advantages of g_autoptr without losing portability to MSVC.

Build system

The question is inevitable: are we going to switch the build system to Meson? I’m looking forward to it, but no, not until Meson is more mature and some of the open questions about distribution and autobuilding have been answered.

Help!

I think it’s great that once I started contributing, other people soon started contributing too. The 1.48.0 release had way more patches and contributors than 1.46.0, even if you don’t count all the stale patches that I souped up. GNOME’s #javascript IRC channel is starting to be a lively place, compared to how deserted it was last year.

What I’d most like to encourage is for more people to contribute major features so that the above list doesn’t read like a to-do list that’s mostly for me. I’m happy to provide guidance. I think it would be great for GJS to become a more competitive development language for apps using the GNOME technology stack1 and we won’t get there with just me.

Another way you can help is by using the development version of GJS while developing your apps or GNOME Shell, thereby helping to try out the new features. We had some serious bugs up to, or even past, the last minute in GNOME 3.24, and this seems like the best way to prevent that.

Finally, you can help by sharing your experiences with GJS: good and bad. Talk on the mailing list or IRC, or file a bug on bugzilla.gnome.org if there’s something wrong.


[1] In that regard I’d love to prove wrong Michael Catanzaro’s opinion about using GJS: “there’s no way to change the reality that JavaScript is a terrible language. It has close to zero redeeming features, and many confusing ones.” There is a way! In my opinion ES6 and ES7 have gone a long way towards filling in those potholes. To name just a few, arrow functions mean you can almost always stop caring about the pitfall of what this refers to, and the prospect of doing asynchronous I/O with Promises instead of callbacks actually makes me want to use JS. Of course, in-browser JS is still a terrible language because it has to support the lowest common denominator of Javascripts so that people who haven’t upgraded their browser since Internet Explorer 8 can still visit your website, and that’s why modern web developers preprocess and transpile it to high heaven. But we don’t have to care about all those browser users!

Javascript news from GNOME 3.24

Welcome back to the latest news on GJS, the Javascript engine that powers GNOME Shell, Endless OS, Polari, GNOME Documents, and many other apps.

GNOME 3.24 has been released for about three weeks now, and with it went GJS 1.48.0. Here’s what’s new!

Javascript upgrade!

First of all, we have a more modern Javascript engine. GJS is based on Mozilla’s SpiderMonkey, the same Javascript engine that runs in the Firefox browser. Back in GNOME 3.22, GJS was based on version 24, which was released in September 2013. Now we’ve moved to version 38, which although still old, was released almost two years later in May 2015.

(The number of each SpiderMonkey release increases by 7 each time, because they make a standalone SpiderMonkey release for each Extended Support Release of Firefox, which is one out of every 7. That’s why you might also hear them referred to as “ESR 38”, etc.)

This brings a lot of new Javascript language features with it. Here are some of the ones I’m most excited about.

Promises

Promises allow you to do asynchronous operations (like reading files, or waiting, or fetching things from the network) in a much more intuitive way. With Promises, the code reads from top to bottom as if it were synchronous, instead of from nested level to nested level (often called “callback hell“.)

Here’s an example, a Promises version of examples/gio-cat.js that’s included in GJS’s source distribution:

This is much longer than the original program, but only the lower part of the program is actually the equivalent of the old callback-based code. The top part would ideally be provided by GJS itself. I’m still figuring out what is the best API for wrapPromise but it’s definitely a candidate for including in a future version of GJS.

This code calls loadContents, prints the contents, and exits the main loop. If an exception is thrown anywhere in the chain before .catch, then the function provided to the catch call will log the error message. In any case, no matter whether the operation succeeded or not, the last then call will make sure the main loop exits.

Template literals

Template literals will change your life if you work with text in your GJS program. They are regular strings in backticks, with interpolation. Say goodbye to this:

const Format = imports.format;
String.prototype.format = Format.format;
log("%s, %s!".format(greeting, name));

Also say goodbye to this:

log(greeting + ", " + name + "!");

Instead, from now on you’ll do this:

log(`${greeting}, ${name}!`);

It’s a lot more readable and intuitive.

Template literals can also cover more than one line, and they do real interpolation of expressions too, not just variable names:

const CSS = `
label {
    font-size: ${fontdesc.get_size()};
}`;

You can also “tag” templates which is out of scope of this blog post, but there is one built-in tag which serves the same purpose as r'' string literals do in Python:

String.raw`I'm writing some \LaTeX\ code here
and I \textbf{don't} want to deal with escaping it:
\[ E = mc^2 \]`

Generators

Generators are a great addition to the Javascript toolbox. They were actually already available in GJS, but only in Mozilla’s nonstandard extension form. They are introduced with the function* keyword instead of function, and they work a lot like Python’s generators. Here’s an example, implementing the xrange() function similar to the one in Python using a generator:

function* xrange(limit) {
    for(let count = 0; count < limit; count++)
        yield count;
}

The yield statement returns control back to the caller, while preserving the state of the generator until the next call. You can get all the values one by one, calling a generator’s next() method, but for...of loops will also deal with generators:

for (let ix of xrange(5))
    print(`Counting from 0 to 4: ${ix}`);

If you want to empty a generator into an array, you can also use the spread operator: [...xrange(5)] will give you an array of numbers from 0 to 4.

Here’s a more complicated example showing the yield* statement which allows you to compose more than one generator:

This code prints looks at the directory that it’s given, and prints all the files in it that are not themselves directories (the “leaf nodes”.) If one of the files is a directory, it will descend into that directory and repeat the process, thanks to yield*.

Want to know more?

Since there’s a lot more than I can cover in a comfortably readable blog post, I made a slide deck. I tried to put it together in such a way that you can use it as reference material.

For more information on all of these cool things, I highly recommend this “ES6 Explained” series of posts from the Mozilla Hacks blog. Some of these features, such as classes and modules, are still to come in GJS.

Maintainer life

The Javascript engine upgrade was the major feature, but I also spent some time on making things easier for myself as the maintainer. A well-tended garden will hopefully attract more gardeners. Happily, some other people joined in for this part.

I cleaned up the build system, using more modern and concise Autotools code. I also spent some time cleaning up compiler warnings, both on GCC and Clang. Now the build and test runs are faster, and the cleaner output makes it much easier to see when something goes wrong. I also made sure that GJS builds on macOS, or at least it did until my Apple hardware broke down. Chun-wei Fan made some improvements that ensure GJS builds on Windows with MSVC. Claudio André implemented continuous integration in a Docker container, with the intention to run it on Travis CI, but sadly we do not have permission to flip the bit to get Travis to build it.

Having written Jasmine GJS in order to bring some of that convenient unit testing technology from the Node world into GJS applications, I also wanted to use it for writing GJS’s own unit tests. I couldn’t use it directly because that would have been a circular dependency, of course, but I embedded a copy of upstream Jasmine plus a very stripped-down version of Jasmine GJS, and called it “Minijasmine”. It’s now a lot easier, and dare I say less of a drag, to write unit tests for GJS. Accordingly, we’ll now try to cover every bug fix with a regression test.

And I worked on getting the bug tracker down to a less daunting number of bugs. It was fun to make the bug chart in my last post, so here’s another one: this is the number of open bugs during the release cycle from 1.46.0 to 1.48.0.

Graphical report results

You can definitely see that November Bug Squash Month had an effect

Unfortunately the chart will not look like this again next time around. The big drop was me closing all the obsolete or already-fixed bugs during November Bug Squash Month. We are down from about 160 to about 100 bugs, but those were all the easy ones; there are only hard ones left now.

Thanks

Thanks to everyone who participated to bring GJS to GNOME 3.24: Chun-wei Fan, Claudio André, Florian Müllner, Alexander Larsson, Iain Lane, Jonh Wendell, and Lionel Landwerlin.

As well, this release incorporated a lot of patches that people contributed a long time ago, even up to 8 years, that for various reasons had not been reviewed yet. (Many from emeritus GJS maintainers!) Thanks to those people for participating in the past, and I’m glad we were able to finally bring your contributions into the project: Giovanni Campagna, Jasper St. Pierre, Sam Spilsbury, Havoc Pennington, Joe Shaw, Paolo Borelli, Shawn Walker, and Tim Lunn.

Luke Jones and Hussam Al-Tayeb identified a serious memory leak right before the final 1.48.0 release and without their contribution, it would have been a different and much sadder story. As it was, 1.48.0 still contained another serious bug that made GNOME Shell quite unusable for an unlucky few people. Thanks to Georges Stavracas for rewriting a happy ending for 1.48.2.

Special thanks to Cosimo Cecchi, for reviewing almost every single line of the code I wrote for this release: about 20000 lines, many of them boring and repetitive.

Thanks also to my employer Endless which sponsored most of the Javascript engine upgrade, and a good chunk of miscellaneous bug fixing time.

Looking forward

My next post will be about what’s to come in GJS for GNOME 3.26.

November Bug Squash Month: GJS

Here’s what happened during November Bug Squash Month in GJS.

First off, I didn’t really get on the ball to promote Bug Squash Month and I didn’t take pictures of any bug squashing activity… which I regret. I hope this post can make up for some of that.

During November I finally took the leap and offered to become a maintainer of GJS. My employer Endless has been sponsoring work on bugs 742249 and 751252, porting GJS’s Javascript engine from SpiderMonkey 24 to SpiderMonkey 31. But aside from that I had been getting interested in contributing more to it, and outside of work I did a bunch of maintenance work modernizing the Autotools scripts and getting it to compile without warnings. From there it was a small step to officially volunteering.

With not much of November remaining and a holiday and family visit coming up (life always is more important than bug squashing!) I decided to start out my bug-squashing campaign with what would get me the most results for the time spent: going through GJS’s bug tracker and closing obsolete or invalid bugs. This I managed to do, closing about 1/4 of all open bugs!

Then I made a list of all open bugs with attached patches and intended to review them to see if they still applied and why they hadn’t been committed yet. I got through a few, and had the dubious distinction of fixing up and committing patches from a 7 year old bug yesterday. But as you can see in the list, there are still 54 remaining. A good to-do list for the next Bug Squash Month, or whenever I feel like working on GJS but don’t know what to work on!

Did you know Bugzilla could generate graphs? I didn’t! Here’s a graph of the total bug count in GJS during November Bug Squash Month:

chart

The clunkiness of this chart kills me though…

My plans now that Bug Squash Month is over are to concentrate on fixing things that make it more pleasant to use and contribute to GJS:

  • Find an active co-maintainer so that we can review each other’s patches (could this be you?)
  • Make ES6 Promises available (this work is also being sponsored by Endless)
  • Rework the test suite to use an embedded copy of Jasmine so that writing automated tests becomes less of a pain
  • Find ways to bring in some of the conveniences that Node developers are used to

I’ve also decided to try an experiment: I’ve just made the Trello board public on which I keep track of what I’m working on and what I’d like to work on. Let me know if this is interesting to you and what features you might like to see on there! (It’s made possible by a Chrome extension, Bug 2 Trello.)

All in all November Bug Squash Month was a success, though next time I will get started earlier in the month. Come join me next time!

gnome_bugsquash_nov_2016

 

GJS and Autoconf

Here’s a leftover from the GNOME Developer Experience Hackfest that I participated in back in January: in my last post about it, I mentioned I had worked on some Autoconf macros for GJS that got added to the Autoconf Archive.

My plan was to port an existing GNOME application written in GJS to use the new macros, as an example for other projects to follow. I did so for GNOME Documents a while ago, but then forgot all about it as the patches sat spoiling on my hard drive. Recently I fixed that up and submitted a patch as I should have done long ago.

Here’s the guide to using the new macros in your project:

  • Replace any AC_PATH_PROG([GJS], [gjs]) (or, possibly, pkg-config --variable=gjs_console gjs-1.0; I might add this to the macro) with the shorter AX_PROG_GJS.
  • Use AX_CHECK_GIRS_GJS to check that you have the correct API version of each introspected dependency that you import in your code. Easy rule of thumb in a one-liner: git grep imports\.gi\. | cut -d: -f2 | awk '{$1=$1};1' | sort | uniq will pick out all the GIRs that you import. (Take special note for Cairo: its GIR name starts with a lowercase letter!)
  • Since GIRs don’t have a concept of versions other than the API version, use AX_CHECK_GIR_SYMBOLS_GJS to check for APIs that you use that aren’t available in all versions. For example, if you use the new GtkShortcutsWindow in your code, don’t try to check for GTK 3.20; instead check for the ShortcutsWindow symbol using this macro. (Use this sparingly, of course; there’s no need to check exhaustively for every new API that was added since 3.0, only the latest that you use!)

You might ask “why should I check these dependencies at build-time when they are only necessary at runtime?” You would be correct, it’s not necessary to check them at build time. However, these macros were originally born of the frustration that happens when you “make all install” a tarball only to run it and find out you’re missing GIR dependencies. And especially in the case of using new API, like ShortcutsWindow that I mentioned above, your program might even start up correctly and crash halfway through when you try to open a ShortcutsWindow. It’s a courtesy to your users (not to mention downstream operating system packagers, for whom your runtime dependencies might not be obvious.)

The macros are now also available in your JHbuild setup, through the m4-common module. How that module works is not very discoverable, and Philip W was kind enough to explain it to me when I asked how to get the new macros in. So I’ll explain it again here for future reference.

M4-common contains the agreed-upon set of M4 macros from the Autoconf Archive that GNOME applications use for building. It’s a Git repository that pulls in a known version of the Autoconf Archive as a Git submodule, and installs the M4 macros from it that GNOME applications need to build. If your application uses any of these macros, then it should have a dependency on m4-common in JHbuild.

As written on the GNOME Wiki, if you get an error like

./configure: line 12053: syntax error near unexpected token `GOBJECT_INTROSPECTION_CHECK'
./configure: line 12053: `AX_REQUIRE_DEFINED(GOBJECT_INTROSPECTION_CHECK)'

then you probably need to build m4-common and make sure you have a dependency on it.

Note that you don’t need to care about any of that if you’re building from a released tarball; that’s because Autotools bundles all the macros you use in the tarball when you run “make dist”.

Thanks to Philip Withnall and Cosimo Cecchi for code review and good ideas; and thanks to Endless Computers for allowing me to contribute these macros developed on company time to the Autoconf Archive.

Geek tip: Running Python GUIs in Sublime Text 2

I’m trying out Sublime Text 2 as a code editor on Windows, since there really aren’t any other free ones I like; unfortunately, my favorite free editor, Gedit, is abysmal on Windows. (Although it looks like something may be done about that soon.) Sublime works really well and has a distraction-free mode, and the extensibility is amazing. I’m not convinced yet that it’s worth $60 for a license — I’d probably pay $25, come on, it’s a text editor — but perhaps if I got a job where I needed text editors more often, I’d try to convince my boss to spring for it. (Although in that case, I’d more likely try to convince my boss to let me work on a platform where I could run Gedit.)

One thing that confused me is that I couldn’t get any of my Python GUI code to work on it. For example, when I’m at work in my lab, I often make small changes to my laser beam profiling program (although really it’s more of a camera driver right now; I haven’t found an excuse to actually write any Gaussian profiling code.) As far as I could figure out, I ought to be able to press Ctrl+B in Sublime Text to run the program. However, nothing ever happened, and I thought the feature must be broken.

Then one time I tried running another non-GUI Python script from Sublime Text. To my surprise, it worked! The feature apparently wasn’t really broken, just selective.

The key to this mystery was the following sentence from the Sublime Text 2 reference wiki:

On Windows, GUIs are suppressed.

Wait, what the what? Why would anybody want that? I guess so the console window doesn’t show up. Fortunately, it’s easy to remedy. Go to Preferences→Browse Packages, open the Default directory and then open exec.py in the editor. Around line 26, you’ll see:

# Hide the console window on Windows
startupinfo = None
if os.name == "nt":
    startupinfo = subprocess.STARTUPINFO()
    startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW

Comment out the last line of this block. Now, whenever you start a Python program, the console will be displayed. If you want to get rid of it, make your build system invoke pythonw instead of python. (Tools→Build System→New Build System, then copy Packages/Python/Python.sublime-build and change python to pythonw.)

Geek tip: ImagingSource camera in Python

In my lab there are some ImagingSource cameras that we use for detection. I was trying to get a model DMK 41BU02 to work so that I could control it directly from my measurement program and not have to use the crappy imaging software that comes with it.

Most USB cameras work with the OpenCV computer vision library without any trouble, and this is how I control them in my instrumentation library, which I wrote in Python. If they don’t work, then there is always an ActiveX interface which usually works.

The ImagingSource cameras, however, include their own driver library, IC Imaging Control. I tried to use SWIG to write a quick Python wrapper for it. However, you can only compile your program with this library if you use Visual C++. Fail!

Thankfully, this article (in German) on the blog of one Edgar Klenske tipped me off: the IC Imaging Control library is itself just a wrapper around DirectShow. That also explains why the headers only compile with VC++. Edgar Klenske’s solution is to use the VideoCapture module, which is itself a wrapper around DirectShow, which comes precompiled so you don’t have to use VC++. Luckily for me, he posted that tip just last week!

And so I was able to subclass my Camera module to interface with DirectShow cameras. Perhaps next I’ll try to install the new OpenCV 2.3 to see if their support for DirectShow has improved any.

My verdict is, never buy any camera from ImagingSource. Their slogan is, ironically, “Technology based on standards.” Sorry, but having your drivers only work with one compiler is more like a lack of standards — standards of both technology and decency.