Comparing Apples and AppleOranges

Via Zach Holman’s blog post I found an interesting Twitter discussion that kicked off with these questions:

A couple of tough questions for all of you:
1. Is the date 2022-06-01 equal to the time 2022-06-01 12:00:00?
2. Is the date 2022-06-01 between the time 2022-06-01 12:00:00 and the time 2022-12-31 12:00:00?
3. Is the time 2022-06-01 12:00:00 after the date 2022-06-01?

I’ve been involved for two years and counting¹ in the design of Temporal, an enhancement for the JavaScript language that adds modern facilities for handling dates and times. One of the principles of Temporal that was established long before I got involved, is that we should use different objects to represent different concepts. For example, if you want to represent a calendar date that’s not associated with any specific time of day, you use a class that doesn’t require you to make up a bogus time of day.² Each class has a definition for equality, comparison, and other operations that are appropriate to the concept it represents, and you get to specify which one is appropriate for your use case by your choice of which one you use. In other, more jargony, words, Temporal offers different data types with different semantics.³

For me these questions all boil down to, when we consider a textual representation like 2022-06-01, what concept does it represent? I would say that each of these strings can represent more than one concept, and to get a good answer, you need to specify which concept you are talking about.

So, my answers to the three questions are “it depends”, “no but maybe yes”, and “it depends.” I’ll walk through why I think this, and how I would solve it with Temporal, for each question.

You can follow along or try out your own answers by going the Temporal documentation page, and opening your browser console. That will give you an environment where you can try these examples and experiment for yourself.

Question 1

Is the date 2022-06-01 equal to the time 2022-06-01 12:00:00?

As I mentioned above, Temporal has different data types with different semantics. In the case of this question, what the question refers to as a “time” we call a “date-time” in Temporal⁴, and the “date” is still a date. The specific types we’d use are PlainDateTime and PlainDate, respectively. PlainDate is a calendar date that doesn’t have a time associated with it: a single square on a wall calendar. PlainDateTime is a calendar date with a wall-clock time. In both cases, “plain” refers to not having a time zone attached, so we know we’re not dealing with any 23-hour or 25-hour or even more unusual day lengths.

The reason I say that the answer depends, is that you simply can’t say whether a date is equal to a date-time. They are two different concepts, so the answer is not well-defined. If you want to do that, you have to convert one to the other so that you either compare two dates, or two date-times, each with their accompanying definition of equality.

You do this in Temporal by choosing the type of object to create, PlainDate or PlainDateTime, and the resulting object’s equals() method will do the right thing:

> Temporal.PlainDate.from('2022-06-01').equals('2022-06-01 12:00:00')
true
> Temporal.PlainDateTime.from('2022-06-01').equals('2022-06-01 12:00:00')
false

I think either of PlainDate or PlainDateTime semantics could be valid based on your application, so it seems important that both are within reach of the programmer. I will say that I don’t expect PlainDateTime will get used very often in practice.⁵ But I can think of a use case for either one of these:

• If you have a list of PlainDateTime events to present to a user, and you want to filter them by date. Let’s say we have data from a pedometer, where we care about what local time it was in the user’s time zone when they got their exercise, and the user has asked to see all the exercise they got yesterday. In this case I’d use date semantics: convert the PlainDateTime data to PlainDate data.
• On the other hand, if the 2022-06-01 input comes from a date picker widget where the user could have input a time but didn’t, then we might decide that it makes sense to default the time of day to midnight, and therefore use date-time semantics.

Question 2

Is the date 2022-06-01 between the time 2022-06-01 12:00:00 and the time 2022-12-31 12:00:00?

I think the answer to this one is more unambiguously a no. If we use date-time semantics (in Temporal, PlainDateTime.compare()) the date implicitly converts to midnight on that day, so it comes before both of the date-times. If we use date semantics (PlainDate.compare()), 2022-06-01 and 2022-06-01 12:00:00 are equal as we determined in Question 1, so I wouldn’t say it’s “between” the two date-times.

> Temporal.PlainDateTime.compare('2022-06-01', '2022-06-01 12:00:00')
-1
> Temporal.PlainDateTime.compare('2022-06-01', '2022-12-31 12:00:00')
-1
> Temporal.PlainDate.compare('2022-06-01', '2022-06-01 12:00:00')
0
> Temporal.PlainDate.compare('2022-06-01', '2022-12-31 12:00:00')
-1

(Why these numbers?⁶ The compare methods return −1, 0, or 1, according to the convention used by Array.prototype.sort, so that you can do things like arr.sort(Temporal.PlainDate.compare). 0 means the arguments are equal and −1 means the first comes before the second.)

But maybe the answer still depends a little bit on what your definition of “between” is. If it means the date-times form a closed interval instead of an open interval, and we are using date semantics, then the answer is yes.⁷

Question 3

Is the time 2022-06-01 12:00:00 after the date 2022-06-01?

After thinking about the previous two questions, this should be clear. If we’re using date semantics, the two are equal, so no. If we’re using date-time semantics, and we choose to convert a date to a date-time by assuming midnight as the time, then yes.

Other people’s answers

I saw a lot of answers saying that you need more context to be able to compare the two, so I estimate that the way Temporal requires that you give that context, instead of assuming one or the other, does fit with the way that many people think. However, that wasn’t the only kind of reply I saw. (Otherwise the discussion wouldn’t have been that interesting!) I’ll discuss some of the other common replies that I saw in the Twitter thread.

“Yes, no, no: truncate to just the dates and compare those, since that’s the data you have in common.” People who said this seem like they might naturally gravitate towards date semantics. I’d estimate that date semantics are probably correct for more use cases. But maybe not your use case!

“No, no, yes: a date with no time means midnight is implicit.” People who said this seem like they might naturally gravitate towards date-time semantics. It makes sense to me that programmers think this way; if you’re missing a piece of data, just fill in 0 and keep going. I’d estimate that this isn’t how a lot of nontechnical users think of dates, though.

In this whole post I’ve assumed we assume the time is midnight when we convert a date to a date-time, but in the messy world of dates and times, it can make sense to assume other times than midnight, as well. This comes up especially if time zones are involved. For example, you might assume noon, or start-of-day, instead. Start-of-day is often, but not always midnight:

Temporal.PlainDateTime.from('2018-11-04T12:00')
  .toZonedDateTime('America/Sao_Paulo')
  .startOfDay()
  .toPlainTime()  // -> 01:00

“These need to have time zones attached for the question to make sense.” If this is your first reaction when you see a question like this, great! If you write JavaScript code, you probably make fewer bugs just by being aware that JavaScript’s Date object makes it really easy to confuse time zones.

I estimate that Temporal’s ZonedDateTime type is going to fit more use cases in practice than either PlainDate or PlainDateTime. In that sense, if you find yourself with this data and these questions in your code, it makes perfect sense to ask yourself whether you should be using a time-zone-aware type instead. But, I think I’ve given some evidence above that sometimes the answer to that is no: for example, the pedometer data that I mentioned above.

“Dates without times are 24-hour intervals.” Also mentioned as “all-day events”. I can sort of see where this comes from, but I’m not sure I agree with it. In the world where JavaScript Date is the only tool you have, it probably makes sense to think of a date as an interval. But I’d estimate that a lot of non-programmers don’t think of dates this way: instead, it’s a square on your calendar!

It’s also worth noting that in some calendar software, you can create an all-day event that lasts from 00:00 until 00:00 the following day, and you can also create an event for just the calendar date, and these are separate things.

A 24-hour interval and a calendar date. Although notably, Google Calendar collapses the 24-hour event into a calendar-date event if you do this.

“Doesn’t matter, just pick one convention and stick with it.” I hope after reading this post you’re convinced that it does matter, depending on your use case.

“Ugh!” That’s how I feel too and why I wrote a whole blog post about it!

How do I feel about the choices we made in Temporal?

I’m happy with how Temporal encourages the programmer to handle these cases. When I went to try out the comparisons that were suggested in the original tweet, I found it was natural to pick either PlainDate or PlainDateTime to represent the data.

One thing that Temporal could have done instead (and in fact, we went back and forth on this a few times before the proposal reached its currently frozen stage in the JS standardization process) would be to make the choice of data type, and therefore of comparison semantics, more explicit.

For example, one might make a case that it’s potentially confusing that the 12:00:00 part of the string in Temporal.PlainDate.from('2022-06-01').equals('2022-06-01 12:00:00') is ignored when the string is converted to a PlainDate. We could have chosen, for example, to throw if the argument to PlainDate.prototype.equals() was a string with a time in it, or if it was a PlainDateTime. That would make the code for answering question 1 look like this:

> Temporal.PlainDate.from('2022-06-01').equals(
... Temporal.PlainDateTime.from('2022-06-01 12:00:00')
... .toPlainDate())
true

This approach seems like it’s better at forcing the programmer to make a choice consciously by throwing exceptions when there is any doubt, but at the cost of writing such long-winded code that I find it difficult to follow. In the end, I prefer the more balanced approach we took.

Conclusion

This was a really interesting problem to dig into. I always find it good to be reminded that no matter what I think is correct about date-time handling, someone else is going to have a different opinion, and they won’t necessarily be wrong.

I said in the beginning of the post: “to get a good answer, you need to specify which concept you are talking about.” Something we’ve tried hard to achieve in Temporal is to make it easy and natural, but not too obtrusive, to specify this. When I went to answer the questions using Temporal code, I found it pretty straightforward, and I think that validates some of the design choices we made in Temporal.

I’d like to acknowledge my employer Igalia for letting me spend work time writing this post, as well as Bloomberg for sponsoring Igalia’s work on Temporal. Many thanks to my colleagues Tim Chevalier, Jesse Alama, and Sarah Groff Hennigh-Palermo for giving feedback on a draft of this post.

---

[1] 777 days at the time of writing, according to Temporal.PlainDate.from('2020-01-13').until(Temporal.Now.plainDateISO()) ↩

[2] A common source of bugs with JavaScript’s legacy Date when the made-up time of day doesn’t exist due to DST ↩

[3] “Semantics” is, unfortunately, a word I’m going to use a lot in this post ↩

[4] “Time” in Temporal refers to a time on a clock face, with no date associated with it ↩

[5] We even say this on the PlainDateTime documentation page ↩

[6] We don’t have methods like isBefore()/isAfter() in Temporal, but this is a place where they’d be useful. These methods seem like good contenders for a follow-up proposal in the future ↩

[7] Intervals bring all sorts of tricky questions too! Some other date-time libraries have interval objects. We also don’t have these in Temporal, but are likewise open to a follow-up proposal in the future ↩

Advertisement

The Surrealist Clock of JavaScript

It’s been a long time since I last blogged. In the interim I started a new job at Igalia as a JavaScript Engine Developer on the compilers team, and attended FOSDEM in Brussels several million years ago in early February back when “getting on a plane and traveling to a different country” was still a reasonable thing to do.

In this blog post I would like to present Temporal, a proposal to add modern and comprehensive handling of dates and times to the JavaScript language. This has been the project I’m working on at Igalia, as sponsored by Bloomberg. I’ve been working on it for the last 6 months, joining several of my coworkers in a cross-company group of talented people who have already been working on it for several years.

This is the kind of timekeeping you get with the old JavaScript Date… (Public domain photograph by Julo)

I already collaborated on a blog post about Temporal, “Dates and Times in JavaScript”, so I won’t repeat all that here, but all the explanation you really need is that Temporal is a modern replacement for the Date object in JavaScript, which is terrible. You may also want to read “Fixing JavaScript Date”, a two-part series providing further background, by Maggie Pint, one of the originators of Temporal.

How Temporal can be useful in GNOME

I’m aware that this blog is mostly read by the GNOME community. That’s why in this blog post I want to talk especially about how a large piece of desktop software like GNOME is affected by JavaScript Date being so terrible.

Of course most improvements to the JavaScript language are driven by the needs of the web.¹ But a few months ago this merge request caught my eye, fixing a bug that made the date displayed in GNOME wrong by a full 1,900 years! The difference between Date.getYear() not doing what you expect (and Date.getFullYear() doing it instead) is one of the really awful parts of JavaScript Date. In this case if there had been a better API without evil traps, the mistake might not have been made in the first place, and it wouldn’t have come down to a last-minute code freeze break.

In the group working on the Temporal proposal we are seeking feedback from people who are willing to try out the Temporal API, so that we can find out if there are any parts that don’t meet people’s needs and change them before we try to move the proposal to Stage 3 of the TC39 process. Since I think GNOME Shell and GNOME Weather, and possibly other apps, might benefit from using this API when it becomes part of JavaScript in the future, I’d be interested in finding out what we in the GNOME community need from the Temporal API.

It seems to me the best way to do this would be to make a port of GNOME Shell and/or GNOME Weather to the experimental Temporal API, and see what issues come up. Unfortunately, it would defeat the purpose for me to do this myself, since I am already overly familiar with Temporal and by now its shortcomings are squarely in my blind spot! So instead I’ll offer my help and guidance to anyone who wants to try this out. Please get in touch with me if you are interested.

How to try it out

Since Temporal is of course not yet a built-in object in JavaScript, to try it out we will need to import a polyfill. We have published a polyfill which is experimental only, for the purpose of trying out the API and integrating it with existing code. Here’s a link to the API documentation.

The polyfill is primarily published as an NPM library, but we can get it to work with GJS quite easily. Here’s how I did it.

First I cloned the tc39/proposal-temporal repo, and ran npm install and npm run build in it. This generates a file called polyfill/script.js which you can copy into your code, into a place in your imports path so that the importer can find it. Then you can import Temporal:

const {Temporal} = imports.temporal.temporal;

Note that the API is not stable, so only use this to try out the API and give feedback! Don’t actually include it in your code. We have every intention of changing the API, maybe even drastically, based on feedback that we receive.

Once you have tried it out, the easiest way to tell us about your findings is to complete the survey, but do also open an issue in the bug tracker if you have something specific.

Intl, or how to stop doing _("%B %-d %Y")

While I was browsing through GNOME Shell bug reports to find ones related to JavaScript Date, I found several such as gnome-shell#2293 where the translated format strings lag behind the release while translators figure out how to translate cryptic strings such as "%B %-d %Y" for their locales. By doing our own translations, we are actually creating the conditions to receive these kinds of bug reports in the first place. Translations for these kinds of formats that respect the formatting rules for each locale are already built into JavaScript engines nowadays, in the Intl API via libicu, and we could take advantage of these translations to take some pressure off of our translators.

In fact, we could do this right now already, no need to wait for the Temporal proposal to be adopted into JavaScript and subsequently make it into GJS. We already have everything we need in GNOME 3.36. With Intl, the function I linked above would become:

_updateTitle() {
    const locale = getCachedLocale();
    const timeSpanDay = GLib.TIME_SPAN_DAY / 1000;
    const now = new Date();
    const rtf = new Intl.RelativeTimeFormat(locale, {numeric: 'auto'});

    if (this._startDate <= now && now <= this._endDate)
        this._title.text = rtf.format(0, 'day');
    else if (this._endDate < now && now - this._endDate < timeSpanDay)
        this._title.text = rtf.format(-1, 'day');
    else if (this._startDate > now && this._startDate - now < timeSpanDay)
        this._title.text = rtf.format(1, 'day');
    else if (this._startDate.getFullYear() === now.getFullYear())
        this._title.text = this._startDate.toLocaleString(locale, {month: 'long', day: 'numeric'});
    else
        this._title.text = this._startDate.toLocaleString(locale, {year: 'numeric', month: 'long', day: 'numeric'});
}

(Note, this presumes a function getCachedLocale() which determines the correct locale for Intl by looking at the LC_TIME, LC_ALL, etc. evnvironment variables. If GNOME apps wanted to move to Intl generally, I think it might be worth adding such a function to GJS’s Gettext module.)

Whereas in the future with Temporal, it would be even simpler and clearer, and I couldn’t resist rewriting that method! We wouldn’t need to store a start Date at 00:00 and end Date at 23:59.999 which is really just a workaround for the fact that we are talking here about a date without a time component, that is purely a calendar day. Temporal covers this use case out of the box:

_updateTitle() {
    const locale = getCachedLocale();
    const today = Temporal.now.date();

    const {days} = today.difference(this._date);
    if (days <= 1) {
        const rtf = new Intl.RelativeTimeFormat(locale, {numeric: 'auto'});
        // Note: if this negation seems a bit unwieldy, be aware that we are
        // considering revising the API to allow negative-valued durations
        days = Temporal.Date.compare(today, this._date) < 0 ? days : -days;
        this._title.text = rtf.format(days, 'day');
    } else {
        const options = {month: 'long', day: 'numeric'};
        if (today.year !== this._date.year)
            options.year = 'numeric';

        this._title.text = this._date.toLocaleString(locale, options);
    }
}

Calendar systems

One exciting thing about Temporal is that it will support non-Gregorian calendars. If you are a GNOME user or developer who uses a non-Gregorian calendar, or develops code for users who do, then please get in touch with me! In the group of people developing Temporal everyone uses the Gregorian calendar, so we have a knowledge gap about what users of other calendars need. We’d like to try to close this gap by talking to people.

A Final Note

In the past months I’ve not been much in the mood to write blog posts. My mind has been occupied worrying about the health of my family, friends, and myself; feeling fury and shame at the inequalities of our society that, frankly, the pandemic has made harder to fool ourselves into forgetting if it doesn’t affect us directly; and fury at our governments that perpetuate these problems and resist meaningful attempts at reform.

With all that’s going on in the world, blogging about technical achievements feels a bit ridiculous and inconsequential, but, well, I’m writing this, and you’re reading this, and here we are. So keep in mind there are other important things too. Be safe, be kind, but don’t forget to stay furious after the dust settles.

---

[1] One motivation for why some are eagerly awaiting Temporal as part of the JavaScript language, as opposed to a library, is that it would be built-in to the browser. The most popular library for fixing the deficiencies of Date, moment.js, can mean an extra download of 20–100 kb, depending on whether you include all locales and support for time zones. This adds up to quite a lot of wasted data if you are downloading this on a large number of the websites you visit, but this specifically doesn’t affect GNOME. ↩

Taking Out the Garbage

From the title, you might think this post is about household chores. Instead, I’m happy to announce that we may have a path to solving GJS’s “Tardy Sweep Problem”.

For more information about the problem, read The Infamous GNOME Shell Memory Leak by Georges Stavracas. This is going to be a more technical post than my previous post on the topic, which was more about the social effects of writing blog posts about memory leaks. So first I’ll recap what the problem is.

Garbage, garbage, everywhere

At the root of the GNOME desktop is an object-oriented technology called GObject. GObjects are reference counted, but not garbage collected. As long as their reference count is nonzero, they are “alive”, and when their reference count drops to zero, they are deleted from memory.

GObject reference counting

Graphical user interfaces (such as a large part of GNOME Shell) typically involve lots of GObjects which all increase each other’s reference count. A diagram for a simple GUI window made with GTK might look like this:

A typical GUI would involve many more objects than this, but this is just for illustrating the problem.

Here, each box is an object in the C program.

Note that these references are all non-directional, meaning that they aren’t really implemented as arrows. In reality it looks more like a list of numbers; Window (1); Box (1); etc. Each object “knows” that it has one reference, but it knows nothing about which other objects own those references. This will become important later.

When the app closes, it drops its reference to the window. The window’s reference count becomes zero, so it is erased. As part of that, it drops all the references it owns, so the reference count of the upper box becomes zero as well, and so on down the tree. Everything is erased and all the memory is reclaimed. This all happens immediately. So far, so good.

Javascript objects

To write the same GUI in a Javascript program, we want each GObject in the underlying C code to have a corresponding Javascript object so that we can interact with the GUI from our Javascript code.

Javascript objects are garbage collected, and the garbage collector in the SpiderMonkey JS engine is a “tracing” garbage collector, meaning that on every garbage collection pass it starts out with objects in a “root set” that it knows are not garbage. It “traces” each of those objects, asking it which other objects it refers to, and keeps tracing each new object until it hits a dead end. Any objects that weren’t traced are considered garbage, and are deleted. (For more information, the Wikipedia article is informative: https://en.wikipedia.org/wiki/Tracing_garbage_collection)

We need to integrate the JS objects and their garbage collection scheme with the GObjects and their reference counting scheme. That looks like this:

The associations between the Javascript objects (“JS”) and the GObjects are bidirectional. That means, the JS object owns a reference to the GObject, meaning the reference count of every GObject in this diagram is 2. The GObject also “roots” the JS object (marks it as unable to be garbage collected) because the JS object may have some state set on it (for example, by writing button._alreadyClicked = false; in JS) that should not be lost while the object is still alive.

The JS objects can also refer to each other. For example, see the rightmost arrow from the window’s JS object to the button’s JS object. The JS code that created this GUI probably contained something like win._button = button;. These references are directional, because the JS engine needs to know which objects refer to which other objects, in order to implement the garbage collector.

Speaking of the garbage collector! The JS objects, unlike the GObjects, are cleaned up by garbage collection. So as long as a JS object is not “rooted” and no other JS object refers to it, the garbage collector will clean it up. None of the JS objects in the above graph can be garbage collected, because they are all rooted by the GObjects.

Toggle references and tardy sweeps

Two objects (G and JS) keeping each other alive equals a reference cycle, you might think. That’s right; as I described it above, neither object could ever get deleted, so that’s a memory leak right there. We prevent this with a feature called toggle references: when a GObject’s reference count drops to 1 we assume that the owner of the one remaining reference is the JS object, and so the GObject drops its reference to the JS object (“toggles down“). The JS object is then eligible for garbage collection if no other JS object refers to it.

(If this doesn’t make much sense, don’t worry. Toggle references are among the most difficult to comprehend code in the GJS codebase. It took me about two years after I became the maintainer of GJS to fully understand them. I hope that writing about them will demystify them for others a bit.)

When we close the window of this GUI, here is approximately what happens. The app drops its references to the GObjects and JS objects that comprise the window. The window’s reference count drops to 1, so it toggles down, dropping one direction of the association between GObject and JS object.

Unlike the GObject-only case where everything was destroyed immediately, that’s all that can happen for now! Everything remains in place until the next garbage collection, because at the top of the object tree is the window’s JS object. It is eligible to be collected because it’s not rooted and no other JS object refers to it.

Normally the JS garbage collector can collect a whole tree of objects at once. That’s why the JS engine needs to have all the information about the directionality of the references.

However, it won’t do that for this tree. The JS garbage collector doesn’t know about the GObjects. So unfortunately, it takes several passes of the garbage collector to get everything. After one garbage collection only the window is gone, and the situation looks like this:

Now, the outermost box’s JS object has nothing referring to it, so it will be collected on the next pass of the garbage collector:

And then it takes one more pass for the last objects to be collected:

The objects were not leaked, as such, but it took four garbage collection passes to get all of them. The problem we previously had, that Georges blogged about, was that the garbage collector didn’t realize that this was happening. In normal use of a Javascript engine, there are no GObjects that behave differently, so trees of objects don’t deconstruct layer by layer like this. So, there might be hours or days in between garbage collector passes, making it seem like that memory was leaked. (And often, other trees would build up in the intervening time between passes.)

Avoiding toggle references

To mitigate the problem Georges implemented two optimizations. First, the “avoid toggle references” patch, which was actually written by Giovanni Campagna several years ago but never finished, made it so that objects don’t start out using the toggle reference system. Instead, only the JS objects hold references to the GObjects. The JS object can get garbage collected whenever nothing else refers to it, and it will drop its reference to the GObject.

A problem then occurs when that wasn’t the last reference to the GObject, i.e. it’s being kept alive by some C code somewhere, and the GObject resurfaces again in JS, for example by being returned by a C function. In this case we recreate the JS object, assuming that it will be identical to the one that was already garbage collected. The only case where that assumption doesn’t hold, is when the JS code sets some state on one of the JS objects. For example, you execute something like myButton._tag = 'foo';. If myButton gets deleted and recreated, it won’t have a _tag property. So in the case where any custom state is set on a JS object, we switch it over to the toggle reference system once again.

In theory this should help, because toggle references cause the tardy sweep problem, so if fewer objects use toggle references, there should be fewer objects collected tardily. However, this didn’t solve the problem, because especially in GNOME Shell, most JS objects have some state on them. And, sadly, it made the toggle reference code even more complicated than it already was.

The Big Hammer

The second optimization Georges implemented was the affectionately nicknamed “Big Hammer”. It checks if any GObjects toggled down during a garbage collector pass, and if so, restart the garbage collector a few seconds after. This made CPU performance worse, but would at least make sure that all unused objects were deleted from memory within a reasonable time frame (under a minute, rather than a day.)

Combined with some other memory optimizations, this made GNOME 3.30 quite a lot less memory hungry than its predecessors.

An afternoon at Mozilla

Earlier this year, I had been talking on IRC to Ted Campbell and Steve Fink on the SpiderMonkey team at Mozilla for a while, about various ins and outs of being an external (i.e. not Firefox) user of SpiderMonkey’s JS engine API. Early September I found myself in Toronto, where Ted Campbell is based out of, and I paid a visit to the Mozilla office one afternoon.

I had lunch with Ted and Kannan Vijayan of the SpiderMonkey team where we discussed the current status of external SpiderMonkey API users. Afterwards, we made the plans which eventually became this GitHub repository of examples and best practices for using the SpiderMonkey JS engine outside of Firefox. We have both documentation and code examples there, and more on the way. This is still in progress, but it should be the beginning of a good resource for embedding the JS engine, and the end of all those out-of-date pages on MDN!

I also learned some good practices that I can put to use in GJS. For example, we should avoid using JS::PersistentRooted except as a last resort, because it roots objects by putting them in a giant linked list, which is then traced during garbage collection. It’s often possible to store the objects more efficiently than that, and trace them from some other object, or the context.

Ending the tardy sweeps

In the second half of the afternoon we talked about some of the problems that I had with SpiderMonkey that were specific to GJS. Of course, the tardy sweep problem was one of them.

For advice on that, Ted introduced me to Nika Layzell, an engineer on the Gecko team. We looked at the XPCOM cycle collector and I was surprised to learn that Gecko uses a scheme similar to toggle references for some things. However, rather than GJS sticking with toggle references, she suggested a solution that had the advantage of being much simpler.

In “Avoiding toggle references” above, I mentioned that the only thing standing in the way of removing toggle references, is custom state on the JS objects. If there is custom state, the objects can’t be destroyed and recreated as needed. In Gecko, custom state properties on DOM objects are called “expandos” or “expando properties” and are troublesome in a similar way that they are in GJS’s toggle references.

Nika’s solution is to separate the JS object from the expandos, putting the expandos on a separate JS object which has a different lifetime from the JS object that represents the GObject in the JS code. We can then make the outer JS objects into JS Proxies so that when you get or set an expando property on the JS object, it delegates transparently to the expando object.

Kind of like this:

In the “before” diagram, there is a reference cycle which we have to solve with toggle references, and in the “after” diagram, there is no reference cycle, so everything can simply be taken care of by the garbage collector.

In cases where an object doesn’t have any expando properties set on it, we don’t even need to have an expando object at all. It can be created on demand, just like the JS object. It’s also important to note that the expando objects can never be accessed directly from JS code; the GObject is the sole conduit by which they can be accessed.

Recasting our GUI from the beginning of the post with a tree of GUI elements where the top-level window has an expando property pointing to the bottom-level button, and where the window was just closed, gives us this:

Most of these GObjects don’t even need to have expando objects, or JS objects!

At first glance this might seem to be garbage-collectable all at once, but we have to remember that GObjects aren’t integrated with the garbage collector, because they can’t be traced, they can only have their reference counts decremented. And the JS engine doesn’t allow you to make new garbage in the middle of a garbage collector sweep. So a naive implementation would have to collect this in two passes, leaving the window’s expando object and the button for the second pass:

This would require an extra garbage collector pass for every expando property that referred to another GObject via its JS object. Still a lot better than the previous situation, but it would be nice if we could collect the whole thing at once.

We can’t walk the whole tree of GObjects in the garbage collector’s marking phase; remember, GObject references are nondirectional, so there’s no generic way to ask a GObject which other GObjects it references. What we can do is partially integrate with the marking phase so that when a GObject has only one reference left, we make it so that the JS object traces the expando object directly, instead of the GObject rooting the expando object. Think of it as a “toggle reference lite”. This would solve the above case, but there are still some more corner cases that would require more than one garbage collection pass. I’m still thinking about how best to solve this.

What’s next

All together, this should make the horrible toggle reference code in GJS a lot simpler, and improve performance as well.

I started writing the code for this last weekend. If you would like to help, please get in touch with me. You can help by writing code, figuring out the corner cases, or testing the code by running GNOME Shell with the branch of GJS where this is being implemented. Follow along at issue #217.

Additionally, since I am in Toronto again, I’ll be visiting the Mozilla office again this week, and hopefully more good things will come out of that!

Acknowledgements

Thanks to Ted Campbell, Nika Layzell, and Kannan Vijayan of Mozilla for making me feel welcome at Mozilla Toronto, and taking some time out of their workday to talk to me; and thanks to my employer Endless for letting me take some time out of my workday to go there.

Thank you to Ted Campbell and Georges Stavracas for reading and commenting on a draft version of this post.

The diagrams in this post were made with svgbob, a nifty tool; hat tip to Federico Mena Quintero.

JavaScript news from GNOME 3.30

Welcome back to the latest news on GJS, the Javascript engine that powers GNOME Shell, Endless OS, and many GNOME apps.

I haven’t done one of these posts for several versions now, but I think it’s a good tradition to continue. GNOME 3.30 has been released for several weeks now, and while writing this post I just released the first bugfix update, GJS 1.54.1. Here’s what’s new!

If you prefer to watch videos rather than read, see my GUADEC talk on the subject.

JavaScript upgrade!

GJS is based on SpiderMonkey, which is the name of the JavaScript engine from Mozilla Firefox. We now use the version of SpiderMonkey from Firefox 60. (The way it goes is that we upgrade whenever Firefox makes an extended support release (ESR), which happens about once a year.)

This brings a few language improvements: not as many as in 2017 when we zipped through a backlog of four ESRs in one year, but here’s a short list:

• Asynchronous iterators (for await (... in ...))
• Rest operator in object destructuring (var {a, b, ...cd} = ...)
• Spread operator in object literals (obj3 = {...obj1, ...obj2})
• Anonymous catch (catch {...} instead of catch (e) {...})
• Promise.prototype.finally()

There are also some removals from the language, of Mozilla-specific extensions that never made it into the web standards.

• Conditional catch (catch (e if ...))
• For-each-in loops (for each (... in ...))
• Legacy lambda syntax (function (x) x * x)
• Legacy iterator protocol
• Array and generator comprehensions ([for (x of iterable) expr(x)])

Hopefully you weren’t using any of these, because they will not even parse anymore! I wrote a tool called moz60tool that will scan your source files and hopefully flag any uses of the removed syntax. It’s also available as a shell extension by Andy Holmes.’

Time for your code to get a checkup… Photo by rawpixel.com on Pexels.com

ByteArray

A special note about ByteArray: the SpiderMonkey upgrade made it necessary to rewrite the ByteArray class, since support for intercepting property accesses in C++-native JS objects was removed, and that was what ByteArray used internally to implement expressions like bytearray[5].

The replacement API I think would have made performance worse, and ByteArray is pretty performance critical; so I took the opportunity to replace ByteArray with JavaScript’s built-in Uint8Array. (Uint8Array didn’t exist when GJS was invented.) For this, I implemented a feature in SpiderMonkey that allows you to store a GBytes inside a JavaScript ArrayBuffer object.

The result is not 100% backwards compatible. Some functions now return a Uint8Array object instead of a ByteArray and there’s not really a way around that. The two are not really unifiable; Uint8Array’s length is immutable, for one thing. If you want the old behaviour back, you can call new ByteArray.ByteArray() on the returned Uint8Array and all the rest of your code should work as before. However, the legacy ByteArray will have worse performance than the Uint8Array, so instead you should port your code.

Technical Preview: Async Operations

The subject of Avi Zajac’s summer internship was integrating Promises and async functions with GIO’s asynchronous operations. That is, instead of this,

file.load_contents_async(null, (obj, res) => {
    const [, contentsBytes, etag] = obj.load_contents_finish(res);
    print(ByteArray.toString(contentsBytes));
});

you should be able to do this:

file.load_contents_async(null)
.then((contentsBytes, etag) => print(ByteArray.toString(contentsBytes)));

or this:

const [contentsBytes, etag] = await file.load_contents_async(null);
print(ByteArray.toString(contentsBytes));

If you don’t pass in a callback to the operation, it assumes you want a Promise instead of a callback, and will return one so that you can call .then() on it, or use it in an await expression.

This feature is a technology preview in GNOME 3.30 meaning, you must opt in for each method that you want to use it with. Opt in by executing this code at the startup of your program:

Gio._promisify(classPrototype, asyncMethodName, finishMethodName);

This is made a bit extra complicated for file operations, because Gio.File is actually an interface, not a class, and because of a bug where JS methods on interface prototypes are ignored. We also provide a workaround API for this, which unfortunately only works on local (disk) files. So the call to enable the above load_contents_async() code would look like this:

Gio._promisify(Gio._LocalFilePrototype, 'load_contents_async', 'load_contents_finish');

And, of course, if you are using an older GNOME version than 3.30 but you still want to use this feature, you can just copy the Promisify code into your own program, if the license is suitable. I’ve already been writing some code for Endless Hack in this way and it is so convenient that I never want to go back.

Debugger

At long last, there is a debugger. Run it with gjs -d yourscript.js!

The debugger commands should be familiar if you’ve ever used GDB. It is a bit bare-bones right now; if you want to help improve it, I’ve opened issues #207 and #208 for some improvements that shouldn’t be too hard to do.

The debugger is based on Jorendb, a toy debugger by Jason Orendorff which is included in the SpiderMonkey source repository as an example of the Debugger API.

Performance improvements

We’ve made some good improvements in performance, which should be especially apparent in GNOME Shell. The biggest improvement is the Big Hammer patch by Georges Stavracas, which should stop your GNOME Shell session from holding on to hundreds of megabytes at a time. It’s a mitigation of the Tardy Sweep problem which is explained in detail by Georges here. Unfortunately, it makes a tradeoff of worse CPU usage in exchange for better memory usage. We are still trying to find a more permanent solution. Carlos Garnacho also made some further improvements to this patch during the 3.30 cycle.

The other prominent improvement is better memory usage for GObjects in general. A typical GNOME Shell run contains thousands or maybe ten-thousands of GObjects, so shaving even a few bytes off per object has a noticeable effect. Carlos Garnacho started some work in this direction and I continued it. In the end we went from 128 bytes per GObject to 88 bytes. In both cases there is an extra 32 byte penalty if the object has any state set on it from JavaScript code. With these changes, GNOME Shell uses several tens of megabytes less memory before you even do anything.

I have opened two issues for further investigation, #175 and #176. These are two likely avenues to reduce the memory usage even more, and it would be great if someone were interested to work on them. If they are successful, it’s likely we could get the memory usage down to 56 bytes per GObject, and eliminate the extra 32 byte penalty.

Eventually we will get to that “well-oiled machine” state… Photo by Celine Nadon on Unsplash

Developer Experience

I keep insisting it’s no coincidence, that as soon as we switched to GitLab we started seeing an uptick in contributors whom we hadn’t seen before. This trend has continued: we merged patches from 22 active contributors to GJS in this cycle, up from 13 last time.

Claudio André landed many improvements to the GitLab CI. For one thing, the program is now built and tested on more platforms and using more compile options. He also spent a lot of effort ensuring that the most common failures will fail quickly, so that developers get feedback quickly.

From my side, the maintainer tasks have gotten a lot simpler with GitLab. When I review a merge request, I can leave the questions of “does it build?” and “are all the semicolons there?” to the CI, and concentrate on the more important questions of “is this a feature we want?” and “is it implemented in the best way?” The thumbs-up votey things on issues and merge requests also provide a bit of an indication of what people would most like to see worked on, although I am not really using these systematically yet.

We have some improvements soon to be deployed to DevDocs, and GJS Guide, a site explaining some of the more basic GJS concepts. Both of these were the subject of Evan Welsh’s summer internship. Evan did a lot of work in upstream DevDocs, porting it from the current unsupported CoffeeScript version to a more modern web development stack, which will hopefully be merged upstream eventually.

It’s about time we had a signpost to point the way in GJS. Photo by Jens Johnsson on Pexels.com

We also have an auto formatter for C++ code, so if you contribute code, it’s easier to avoid your branches failing CI due to linter errors. You can set it up so that it will correct your code style every time you commit; there are instructions in the Hacking file. It uses Marco Barisione’s clang-format-hooks. The process isn’t infallible, though: the CI job uses cpplint and the auto formatter uses clang-format, and the two are not 100% compatible.

There are a few miscellaneous nice things that Claudio made. The test coverage report for the master branch is automatically published on every push. And if you want to try out the latest GJS interpreter in a Flatpak, you can manually trigger the “flatpak” CI job and download one.

What’s coming in 3.32

There are a number of efforts already underway in the 3.32 cycle.

ES6 modules should be able to land! This is an often requested feature and John Renner has a mostly-working implementation already. You can follow along on the merge request.

Avi Zajac is working on the full version of the async Promises feature, both the gobject-introspection and GJS parts, which will make it no longer opt-in; Promises will “just work” with all GIO-based async operations.

Also related to async and promises, Florian Müllner is working on a new API that will simplify calling DBus interfaces using some of the new ES6 features we have gained in recent releases.

I hope to land Giovanni Campagna’s old “argument cache” patch set, which looks like it will speed up calls from JS into C by quite a lot. Apparently there is a similar argument cache in PyGObject.

Finally, and this will be the subject of a separate blog post coming soon, I think we have a plausible solution to the Tardy Sweep problem! I’m really excited to write about this, as the solution is really ingenious (I can say that, because I did not think of it myself…)

Contributors

Thanks to everyone who participated to bring GJS to GNOME 3.30: Andy Holmes, Avi Zajac, Carlos Garnacho, Christopher Wheeldon, Claudio André, Cosimo Cecchi, Emmanuele Bassi, Evan Welsh, Florian Müllner, Georges Basile Stavracas Neto, James Cowgill, Jason Hicks, Karen Medina, Ole Jørgen Brønner, pixunil, Seth Woodworth, Simon McVittie, Tomasz Miąsko, and William Barath.

As well, this release incorporated some old patches that people contributed in the past, even up to 10 years ago, that were never merged because they needed some tweaks here or there. Thanks to those people for participating in the past, and I’m glad we were finally able to land your contributions: Giovanni Campagna, Jesus Bermudez Velazquez, Sam Spilsbury, and Tommi Komulainen.

GUADEC 2018 Reminiscences

This year’s GUADEC in Almería, Spain, was over two months ago, and so here is a long overdue post about it. It was so long ago that I might as well call it a reminiscence! This will be a different kind of post than the ones I’ve done in past years, as plenty of other bloggers have already posted summaries about the talks.

Board of Directors

I didn’t even get to see that many talks anyway, as this was my first GUADEC after being elected to the GNOME Foundation board of directors and I found myself doing a lot of running around to complete things. The board has to prepare for a number of meetings including the GNOME Foundation’s Annual General Meeting that’s always held at GUADEC, and so there was plenty of preparation to be done.

So, except for a few sessions, I mainly followed the “hallway track” this year.

It’s an exciting time to be on the board; it’s been in the news recently that the GNOME Foundation has received two substantial donations, and is hiring some new roles. If you want more information and background, Rosanna Yuen, director of operations, explains all about it in this GUADEC talk.

Interns

Somehow I found myself mentoring two interns this summer, Avi Zajac and Evan Welsh, and both of them were able to attend GUADEC. (I co-mentored Evan with my coworker Manuel Quiñones, who unfortunately could not be there.) I had not done a good job introducing them to each other, but they connected with each other and realized that they were both working with me! If you haven’t already, make sure to read Avi’s blog post and Evan’s blog post for their perspectives on how it went. I was glad to have both of them there and really enjoyed meeting up in person.

Both internships have finished up in the meantime. Evan’s website is viewable here, as well as some improvements to DevDocs which I hope to deploy soon. Avi’s project was released as a technical preview in GNOME 3.30 and will be covered in my next blog post.

JavaScript Talk

I gave one scheduled talk, on GJS and JavaScript, and one unscheduled talk, on Endless Code.

I will cover the material from the JavaScript talk in my next post about the new features in GJS, but for now I wanted to post the slides for everyone’s reference. The video of the talk is here.

Endless Hack Talk

I was voted into one of the conference’s open talk slots with a proposal to talk about Endless Code (since then, renamed to Endless Hack). This is a new (well, it was new at the time of the conference) project at Endless. It’s a continuation of this feature which (I didn’t work on, but) my coworkers demoed about two years ago:

Want to hack on an app in @endlessglobal? Simple, just flip over the window and start coding. All code, build tools, and deps are there. pic.twitter.com/T3maWdw2eI

— Jono Bacon (@jonobacon) January 5, 2017

The Endless Hack product generalizes this idea to the whole desktop. The idea is that you should be able to tinker with everything, and there’s a narrative that guides you along the way and teaches you programming concepts. It’s aimed at children and young teenagers. Although this product hasn’t been released yet, and although some of the source code is currently open it’s not in a finished or usable state yet, I did want to talk about it at GUADEC because I think the ability to learn by tinkering is an important part of the free software experience and a direct consequence of one of the Four Software Freedoms, and it’s something the GNOME community should be aware of.

We also made a survey asking people about their experience learning how to program, or not learning how to program, and it’s still open, because I did not do a very good job in the talk of publicizing the link! You can fill it out here.

I haven’t dared to watch the video of me talking completely unrehearsed, but you can watch it here if you want.

Unconference

I had high hopes for organizing a GJS unconference session like last year, but after a certain point I was just completely tired out. We did eventually have a GJS session that consisted of people hacking on their favourite thing. Happily, I was able to convince Georges Stavracas to fix a regression that was preventing GNOME Shell from starting. I got a chance to work with Meg Ford on testing with JavaScript, and I also got some work done on the GJS debugger, a new feature in GNOME 3.30. I will talk about all this and more in my next post!

We also used some of the unconference time for a kickoff session for the GNOME Developer Center. Bastian Ilsø is leading this initiative and has a lot of material for you to read on what’s happened in the meantime.

Acknowledgements

I’d like to thank the GNOME Foundation for making it possible for me to attend the conference and the board meetings.

Thank you to my coworker Lisette Silva for convincing me to submit the open talk and giving some last-minute feedback beforehand.

More Memory, More Problems

In GJS we recently committed a patch that has been making waves. Thanks to GJS contributor Georges Basile “Feaneron” Stavracas Neto, some infamous memory problems with GNOME Shell 3.28 have been mitigated. (What’s the link between GNOME Shell and GJS? GNOME Shell uses GJS as its internal Javascript engine, in which some of the UI and all of the extensions are implemented.)

There is a technical explanation, having to do with toggle-refs, a GObject concept which we use to interface the JS engine’s garbage collector with GObject’s reference counting system. Georges has already provided a fantastic introduction to the technical details so I will not do another one here. This post will be more about social issues, future plans, and answers to some myths I’ve seen in various comments recently. To read this post, you only need to know that the problem has to do with toggle-refs and that toggle-refs are difficult to reason about.

Not a Memory Leak

I really don’t want to call this a memory leak, much less “the GNOME memory leak” that’s become common in the press coverage lately. I find that that sets the wrong expectations for users suffering from these memory problems. You might say that for the end user it makes no difference, their computer’s memory is being occupied by GNOME Shell, so what’s the point in not calling it a memory leak? And you would be partially right. The effect is no different. The expectations are different though, especially for users who have some technical knowledge. A memory leak is a simple problem to fix. When you have one, you run your software under Valgrind or ASAN, you get a backtrace that shows where the memory was allocated that you didn’t free, and you free it. Problem solved. You can even run Valgrind in your automatic tests to prevent new leaks. That’s not the case here, and if we refer to it as a memory leak then it can only cause frustration on the part of users who are aware how simple it is to fix a memory leak.

This problem is different. It’s not a leak in the traditional sense. The memory does eventually get freed, but GNOME Shell holds onto it for too long; long enough to cause problems on some systems. As GJS contributor Andy Holmes put it, it’s a “tardy GC sweep.” I think that has a catchy ring to it, so I’ll call it the “tardy sweep problem” from now on.

Meme by Andy Holmes, used with kind permission

To be honest, I found that the OMG!Ubuntu article about “the memory leak” attracted a lot of comments that don’t sit well with me, and I think that the wrong expectations set by calling it a “memory leak” are partly to blame. With this post, I hope to give a better idea of what GNOME users can expect.

On the bright side, due to the recent publicity and especially the OMG!Ubuntu article, more GNOME developers are talking about the memory problems and suggesting things, which is causing an exciting confluence of ideas that I couldn’t have come up with on my own.

Edit: I want to be absolutely clear that with the above I’m not blaming bug reporters for not knowing whether something is a “proper” memory leak or not. This is intended to bring some attention to the wrong expectations that arise, especially among technically savvy users, when GNOME developers and the tech press use the term “memory leak,” and illustrate why we ourselves should not use the term here.

 

The Big Hammer

CC0 licensed image, by stevepb

I’ve been calling this patch the “Big Hammer” because it’s a drastic measure: starting a whole new garbage collection cycle in order to clean up some objects that we already know should be cleaned up.

The tardy sweep problem has now been mitigated with the Big Hammer, but reducing GNOME Shell’s memory usage has been a battle for years, and it has very little to do with memory leaks.

There are many other causes of high memory usage in GNOME Shell. Some are real memory leaks, that generally get fixed before too long. GNOME Shell developers have had their suspicions about NVidia drivers for years. Another cause is JS memory leaks in GNOME Shell. (Contrary to popular belief, it is possible to leak memory in pure Javascript code. Andy’s new heapgraph tool is useful when tracking these down, but throughout most of the life of GNOME Shell this tool didn’t exist.) There’s also memory fragmentation, which can look like a memory leak in a resource monitor.¹ In addition, when diagnosing reports from users, configurations vary wildly. Memory usage simply differs from system to system. Finally, people have different configurations of Shell extensions, some of which leak memory as well.

The Sad Lifecycle of a GNOME Shell Memory Leak Bug Report

1. User reports “I have a memory leak”
2. Developer runs Valgrind, sifts through Valgrind trace, finds a small leak and fixes it
3. Problem isn’t fixed
4. Repeat 2 and 3 until no leaks shown in the Valgrind trace
5. Problem still isn’t fixed
6. User eagerly awaits each point release hoping for relief, and is disappointed each time
7. Bug report gains popularity, accretes followers like a katamari, some of whom vent about unrelated bugs, hound the developer, or become abusive, until the original point of the bug report is lost
8. Developer can’t do anything productive with the bug report at this point. They know there’s still a memory problem and it’s not a traditional leak, but the bug report is not helping them find it
9. Users don’t accept that answer
10. Developer closes the bug report and makes users even angrier. (Or, developer ignores the bug report until it fizzles out, and makes users sad.)

Here are some examples of long-running bug reports where you can see this dynamic in action. It’s quite sad to observe, because everybody involved is doing what makes perfect sense from their perspective (except for a few people behaving badly), yet the result is a mess.

I hope this illustrates why it’s important to assume that people are acting in good faith.

I know some people will argue that the developer mustn’t close the bug report until the bug is “fixed”, meaning that there is no more unnecessary memory usage. But in my opinion that’s just not a useful way to think of bug reports. GNOME Shell developers know (and so do I, from the GJS side) that GNOME Shell uses a lot of memory. I agree it’s nice to keep a bug report open so that users know that we’re aware of it and it’s on our to-do lists somewhere, but very soon the time we spend dealing with the noise on the bug report eclipses whatever benefit it might bring to the community.

I hope GitLab will improve things a bit here, since if you feel strongly about an issue in the bugtracker, you can upvote it or add an emoji reaction to it. This is a good way for users to show that an issue is important to them, and if enough people use it then it’s a good indicator for me to see which issues are prioritized highest by users and contributors.

5 Myths About GNOME Shell’s Memory Problems (Paraphrased)

“GNOME developers never cared about the tardy sweep problem until OMG!Ubuntu reported on it. They don’t care about users until someone makes them.”

Carlos Garnacho, of GNOME Shell fame, pointed out to me that in a more global perspective there has been a very active hunt for actual memory leaks across many of GNOME Shell’s dependencies for quite some time, and he has personally patched leaks in IBus, AccountsService, libgweather, gnome-desktop, and more.

It seems the tardy sweep problem has gotten worse in recent versions of GNOME, although it’s hard to measure between different systems with different configurations. I don’t know why it’s gotten worse.²

It seems to have been known for a long time, though: for history buffs, it was alluded to in a comment in commit ae34ec49, back in 2011. That knowledge was apparently lost when GJS was without a maintainer for a couple of years. To be honest, it has taken me well over a year to get familiar enough with the toggle-ref code (which integrates the JS garbage collector with GObject’s refcounting system), that I feel even remotely comfortable making or reviewing changes to it. I only fully realized the implications of the tardy sweep problem after talking to Georges and seeing his memory graph.

It seems that a previous GJS maintainer, Giovanni Campagna, was trying to mitigate the tardy sweep problem already five years ago, with a patch that allowed objects to escape the tardy sweep by opting out of the whole toggle-ref system in some cases. Unfortunately, as far as I can tell from my bug tracker archaeology, his patch went through a few reviews and the answer was always “Wow, this is really complicated, I need to study it some more in order to understand it.” Then it fell by the wayside when he stepped down from GJS maintainership.

I picked the patch up again late last year and fixed up most of the bit-rot. It still had a few problems with it. I never found the time to fix it up completely, which Georges kindly took over for me. I initially preferred Giovanni’s patch above the Big Hammer, but unfortunately for me, Georges proved that it wasn’t as effective as we thought it would be, only clearing up about 5% of the tardy sweep memory.³

“GNOME fixed the problem with a shoddy solution. This will decrease performance but they won’t notice because they all have top-of-the-line machines.”

I don’t call it the Big Hammer for nothing. We were concerned about performance regressions too, so that’s reasonable. However, as you can read in the bug tracker, we did actually do some testing on lower-end hardware before merging the Big Hammer, and it was not as bad as I expected. Carlos has been doing some measurements and found that garbage collection accounts for about 2–3% of the time that GNOME Shell occupies the CPU.

However, it’s exactly because we want to be cautious that the Big Hammer has only been committed to master, which will first be released in the unstable GNOME 3.29.2 snapshot. I don’t plan to release it on a stable branch until we’ve run it some more.

Ubuntu has already put the Big Hammer in their LTS version. That’s more of a risk than I would have recommended, but it’s not my decision to make, and I am grateful that we will be getting some testing through that avenue. Endless is also considering putting the Big Hammer in their stable version.

(And alas, I don’t have a top-of-the-line machine. Feel free to donate me one if that’s what’s required to make me conform to some stereotype of GNOME developers. 😇)

“The problem should be fixed now. GNOME Shell will run smooth from now on.”

No, it’s not. GNOME Shell still isn’t that great with memory.

Carlos is working on some merge requests which are approaching being ready to merge, which should make things a bit more memory-efficient. He’s also had some success with experiments trying to reduce memory fragmentation, taking better advantage of SpiderMonkey’s compacting garbage collector.

We are also bouncing around some ideas for making the Big Hammer into a smaller hammer. In particular, we’re trying to see if the extra garbage collections can be restricted to only the JS objects that represent GObjects, since those are the only objects that are affected by the tardy sweep problem. We’re also trying to see if there’s a way to return black-marked (reachable) objects to their original white-marked (eligible for collection) state when a GObject is toggled down in the middle of a garbage collection.

Another approach to investigate is to make better use of incremental garbage collection. SpiderMonkey offers this facility but we don’t use it yet. The idea is, instead of pausing and doing a big garbage collection, we do a slice of a few milliseconds whenever we have time. I don’t know yet whether this will have a large or small effect, or even render the Big Hammer unnecessary.

We’re also going to update to SpiderMonkey 60 in GNOME 3.30 which will hopefully bring in another year’s worth of Mozilla’s garbage collector research and optimization.

Finally, I’m gradually working on another unfinished merge request left over from Giovanni’s tenure as GJS maintainer, that should drastically increase the performance of GNOME Shell’s animations (though not necessarily help with memory.)

“GNOME has no business releasing any new versions until this problem is fixed.”

GNOME has a fixed release schedule, so they release new versions on the release dates, with whatever is aboard the train at that time. That’s not going to change.

“This version of GNOME is going into Ubuntu LTS! GNOME needs to work harder to fix this.”

Of course, I want whatever version of GNOME ships with any Linux distribution to be as good as possible. But as the upstream GJS maintainer, I have no say over what a downstream Linux distribution chooses to ship. The best way for a Linux distro to make sure their release is shipshape, is to contribute resources towards fixing whatever they consider a blocker.

That sounds a bit callous, as if I refuse to fix any bugs that Ubuntu wants fixed; that’s not what I mean at all. But my free time is limited. I’m paid for a part of my GJS maintainer work, but only for specific features. I can’t work to anyone’s external deadlines in my free time, because otherwise I’ll burn out and that’s not good for anyone with any interest in GJS either. Sometimes I have other priorities besides sitting at the computer; sometimes I do have time but no ideas about a particular problem; sometimes my brain isn’t up to fixing a difficult memory problem and I choose to work on something easier.⁴ Bugfixing work isn’t fungible.

I picked Ubuntu to illustrate this example, because contributing is exactly what the Ubuntu team has done; Ubuntu contributors fixed stability bugs in GJS, as well as GNOME Shell and Mutter, for GNOME 3.28. To say nothing of contributors from other downstreams, as well. That’s great and I’m looking forward to more of it! Some commenters seem to see downstreams fixing bugs as something that GNOME developers should be ashamed of, but I believe everyone is better off for it when that happens!

Acknowledgements

Thanks to Carlos Garnacho and Andy Holmes, who commented on a draft version of this blog post. Thanks in addition to Andy who coined the term “tardy sweep” and provided Scruffy as the mascot; Heartbleed has branding, why shouldn’t we? And of course, thanks to Georges who kicked off the whole research in the first place!

---

[1] and has often made people angry in bug reports when told it’s not a memory leak ↩

[2] I have a hunch, though. When I updated SpiderMonkey to version 38 in GNOME 3.24, we went from a conservative collector to an exact-rooted, moving one — see this Wikipedia article for definitions of those terms. It may be that the old garbage collector, though generally considered inferior, did actually mitigate the tardy sweeps a little, because I think back then it would have been possible for more objects to make their way into an ongoing sweep. It’s also possible that it was made worse earlier than that, by some adjustments in GNOME Shell that adjusted how often the garbage collector was called. ↩

[3] Technical explanation: Tweener, which is the animation framework used by GNOME Shell, renders many objects ineligible to opt out of the toggle-ref system. I would like to see Tweener replaced with Clutter implicit animations in GNOME Shell, which would make Giovanni’s patch much more effective, but that’s a big project. ↩

[4] Like writing a blog post about a difficult memory problem. Joke’s on me, that’s actually really hard ↩

Weekend Website Experiment

As you may know if you read this blog via Planet GNOME, the GNOME project is busy switching to GitLab for its code hosting and bug tracking. I like GitLab! It’s a large step up from Bugzilla, which was what GNOME used for the last 20 years. Compared to GitHub, GitLab is about equal, with a few nicer things and a few less nice things.

The one thing that I miss from Bugzilla is a dashboard showing the overall status of the bugs for your project. I thought it would not be too hard to use the GitLab API to do some simple queries and plop them on a web page. So, last weekend I gave it a try. The final result is here. Click the button to log into GitLab, and you’ll be redirected back to the page where you’ll get the results of the queries.

I’d like to write up what I did because I learned a new thing, and I think more writeups illustrating the trial and error of learning a new thing are always good.

My goals were to write something without being too meticulous, and write something that was not intended to scale. (Both are things that I normally disapprove of. It’s good to try the other side once in a while.) So, I was just going to mix the HTML, CSS, and JS all in one file. I used the base CSS and overall page structure from my personal website.

I decided to use GitHub Pages for hosting. My personal website is already hosted there. GitLab also offers GitLab Pages, which is very similar, but GNOME hasn’t enabled it yet on their GitLab instance. So, I created a fork of GNOME/gjs on GitHub, and created a gh-pages branch. Whatever you commit to that branch shows up as your project’s GitHub Pages site on the web.

The first thing I figured out is that you have to be authenticated to query issues in the API, even if the same information is publicly available on the web. That’s too bad! My little project suddenly went from “easy” to “figuring out something I’ve never done before.” But that’s also exciting.

First, I got the queries running on a local webpage, using a temporary personal GitLab access token. Each item looked kind of like this:

Number of crashers: 16
Number of bugs: 25
etc.

Next, I decided to tackle the authentication problem. I did some searches on variations of “gitlab authentication plugin”, “add gitlab authentication to webpage”, etc., to see if there was something ready-made I could drop in. No such luck. I did find NodeJS modules that I could have used, had I been writing the site using NodeJS. I weighed the unknown cost of implementing the authentication in plain old browser JS against the unknown cost of setting up tools that I was unfamiliar with in order to use the ready-made module (I wasn’t even sure what tools I would have to use — Webpack, I think?) and decided to keep looking.

I next searched for things like “gitlab oauth2 in browser”, “gitlab oauth2 example”, since I knew that the login used the OAuth2 protocol. Eventually I landed on this page and figured out that the magic words I wanted were “implicit flow” or “implicit grant”:

Implicit Flow – This flow is designed for user-agent only apps (e.g. single page web application running on GitLab Pages).

That sounded like exactly my use case, so I read further. You have to send the user to a certain page on the GitLab site, and send a redirect URL which the user will be sent back to with the authorization token in the URL hash. I managed to keep everything on the same webpage. In pseudocode, the flow looks like this:

if we have the authorization token:
    fetch the numbers with the API calls
else if there is a hash in the URL:
    token = get the token from the hash
    store the token
    fetch the numbers with the API calls
else:
    show a button that links to the authorization URL on GitLab

For storing the token so that you don’t have to log in every time, I used localStorage. I have no idea if that’s good practice or not, but from what I could read online it seems that it’s at least not bad practice. It’s quite easy to retrieve the token, but only if you have access to the browser where it is stored. I don’t think localStorage can leak out over the web, but with the recently discovered vulnerabilities who knows…

Last, I made it look nicer. I had a pretty good idea of what I wanted it to look like: I wanted the numbers to be large, in colored boxes with rounded corners and thick borders. I tried a few things with floating <div>s before giving up and using a CSS flex layout. This makes the page probably unviewable on older browsers, but I was seriously done with CSS positioning.

The code is here, or just “View Source” while you’re on the page.

What I would do differently

Writing HTML, CSS, and JS directly for the web is tedious and repetitive. I wish my younger self had used some sort of framework like Bootstrap to make my personal site. Failing that, I wish I had decided not to reuse components from my personal site to do this, and instead started with a fresh site using a framework. Bootstrap or Semantic UI are two that I know of, and maybe should have tried out. The code ended up being 263 lines of HTML, CSS, and JS, much of it just repeated items in order to do the boxes in different colors.

Reuse of this code

You may notice I did not release this code under an open source license. That’s because it’s probably full of bad practices, so I don’t want people to copy it. If you can convince me that it’s done right or tell me what I did wrong, then I’ll (fix it and) open-source it, and other GitLab maintainers might find it useful.

Geek tip: g_object_new and constructors

tl;dr Don’t put any code in your foo_label_new() function other than g_object_new(), and watch out with Vala.

From this GJS bug report I realized there’s a trap that GObject library writers can fall into,

Avoid code at your construction site.

that I don’t think is documented anywhere. So I’m writing a blog post about it. I hope readers from Planet GNOME can help figure out where it needs to be documented.

For an object (let’s call it FooLabel) that’s part of the public API of a library (let’s call it libfoo), creating the object via its foo_label_new() constructor function should be equivalent to creating it via g_object_new().

If foo_label_new() takes no arguments then it should literally be only this:

FooLabel *
foo_label_new(void)
{
    return g_object_new(FOO_TYPE_LABEL, NULL);
}

If it does take arguments, then they should correspond to construct properties, and they should get set in the g_object_new() call. (It’s customary to at least put all construct-only properties as arguments to the constructor function.) For example:

FooLabel *
foo_label_new(const char *text)
{
    return g_object_new(FOO_TYPE_LABEL,
        "text", text,
        NULL);
}

Do not put any other code in foo_label_new(). That is, don’t do this:

FooLabel *
foo_label_new(void)
{
    FooLabel *retval = g_object_new(FOO_TYPE_LABEL, NULL);
    retval->priv->some_variable = 5;  /* Don't do this! */
    return retval;
}

The reason for that is because callers of your library will expect to be able to create FooLabels using g_object_new() in many situations. This is done when creating a FooLabel in JS and Python, but also when creating one from a Glade file, and also in plain old C when you need to set construct properties. In all those situations, the private field some_variable will not get initialized to 5!

Instead, put the code in foo_label_init(). That way, it will be executed regardless of how the object is constructed. And if you need to write code in the constructor that depends on construct properties that have been set, use the constructed virtual function. There’s a code example here.

If you want more details about what function is called when, Allison Lortie has a really useful blog post.

This trap can be easy to fall into in Vala. Using a construct block is the right way to do it:

namespace Foo {
public class Label : GLib.Object {
    private int some_variable;

    construct {
        some_variable = 5;
    }
}
}

This is the wrong way to do it:

namespace Foo {
public class Label : GLib.Object {
    private int some_variable;

    public Label() {
        some_variable = 5;  // Don't do this!
    }
}
}

This is tricky because the wrong way seems like the most obvious way to me!

This has been a public service announcement for the GNOME community, but here’s where you come in! Please help figure out where this should be documented, and whether it’s possible to enforce it through automated tools.

For example, the Writing Bindable APIs page seems like a good place to warn about it, and I’ve already added it there. But this should probably go into Vala documentation in the appropriate place. I have no idea if this is a problem with Rust’s gobject_gen! macro, but if it is then it should be documented as well.

Documented pitfalls are better than undocumented pitfalls, but removing the pitfall altogether is better. Is there a way we can check this automatically?

Modern Javascript in GNOME – GUADEC 2017 talk

I gave a presentation at GUADEC 2017 this morning on modern Javascript in GNOME, the topic of the last few posts on this blog. As I promised during the talk, here are the slides. There is a beefy appendix after the questions slide, with details about all the new language features, that you are welcome to use as a reference.

Thanks to the GNOME Foundation for the travel sponsorship, to my employer Endless for paying for some of this work, and especially to Rob McQueen for the last-minute loan of a USB-C video adapter!

Inventing GObject ES6 classes

Hello again! If you’re a GJS user, I’d like your opinion and ideas. After my last post where I talked about new features coming in GNOME 3.26 to GJS, GNOME’s Javascript engine, I’m happy to say that the patches are nearly ready to be landed. We just need to figure out how to build SpiderMonkey 52 consistently even though Mozilla hasn’t made an official standalone release of it yet.

A better literal depiction of JAVA SCRIPT I could not ask for… (Public domain image courtesy of Engin_Akyurt)

As I reported last time:

After that is done, I will refactor GJS’s class system (Lang.Class and GObject.Class). I believe this needs to be done before GNOME 3.26. That’s because [we will] gain ES6 classes, and I want to avoid the situation where we have two competing, and possibly incompatible, ways to write classes.

That’s what I’m busy doing now, in the run-up to GUADEC later this month, and I wanted to think out loud in this blog post, and give GJS users a chance to comment.

First of all, the legacy Lang.Class classes will continue to work. You will be able to write ES6 classes that inherit from legacy classes, so you can start using ES6 classes without refactoring all of your code at once.

That was the good news, now the bad

However, there is not an obvious way to carry over the ability to create GObject classes and interfaces from legacy classes to ES6 classes. The main problem is that Lang.Class and its subclasses formed a metaclass framework. This was used to carry out certain activities at the time the class object itself was constructed, such as registering with the GType system.

ES6 classes don’t have a syntax for that, so we’ll have to get a bit creative. My goals are to invent something (1) that’s concise and pleasant to use, and (2) that doesn’t get in the way when classes gain more features in future ES releases; that is, not too magical. (Lang.Class is pretty magical, but then again, there wasn’t really an alternative at the time.)

Here is how the legacy classes worked, with illustrations of all the possible bells and whistles:

	const MyClass = new Lang.Class({
	Name: 'MyClass',
	GTypeName: 'MyNamespaceMyClass',
	Extends: GObject.Object,
	Implements: [Gio.Initable, MyCustomInterface],
	Properties: {
	'prop': GObject.ParamSpec.int( /* etc., etc. */ ),
	},
	Signals: {
	'signal': { param_types: [ /* etc., etc. */ ] },
	},
	_init(props={}) {
	this.parent(props);
	// etc.
	},
	get prop() { /* … */ },
	method(arg) { /* … */ },
	});

view raw

legacy.js

hosted with ❤ by GitHub

The metaclass magic in Lang.Class notices that the class extends GObject.Object, and redirects the construction of the class object to GObject.Class. There, the other magic properties such as Properties and Signals are processed and removed from the prototype, and it calls a C function to register the type with the GObject type system.

Without metaclasses, it’s not possible to automatically carry out magic like that at the time a class object is constructed. However, that is exactly the time when we need to register the type with GObject. So, you pretty much need to remember to call a function after the class declaration to do the registering.

The most straightforwardly translated (fictional) implementation might look something like this:

	class MyClass extends GObject.Object {
	static get GTypeName { return 'MyNamespaceMyClass'; }
	static get Implements { return [Gio.Initable, MyCustomInterface]; }
	static get Properties {
	return {
	'prop': GObject.ParamSpec.int( /* etc., etc. */ ),
	};
	}
	static get Signals {
	return {
	'signal': { /* etc. */ },
	};
	}
	constructor(props={}) {
	super(props);
	// etc.
	}
	get prop() { /* … */ }
	method(arg) { /* … */ }
	}
	GObject.registerClass(MyClass);

view raw

verbose.js

hosted with ❤ by GitHub

The fictional GObject.registerClass() function would take the role of the metaclass’s constructor.

This is a step backwards in a few ways compared to the legacy classes, and very unsatisfying. ES6 classes don’t yet have syntax for fields, only properties with getters, and the resulting static get syntax is quite unwieldy. Having to call the fictional registerClass() function separately from the class is unpleasant, because you can easily forget it.

On the other hand, if we had decorators in the language we’d be able to make something much more satisfying. If you’re familiar with Python’s decorators, these are much the same thing: the decorator is a function which takes the object that it decorates as input, performs some action on the object, and returns it. There is a proposed decorator syntax for Javascript that allows you to decorate classes and class properties. This would be an example, with some more fictional API:

	@GObject.Class('MyNamespaceMyClass')
	@GObject.implements([Gio.Initable, MyCustomInterface])
	@GObject.signal('signal', { /* etc. */ })
	class MyClass extends GObject.Object {
	constructor(props={}) {
	super(props);
	// etc.
	}
	@GObject.property.int('Short name', 'Blurb', GObject.ParamFlags.READABLE, 42)
	get prop() { /* etc. */ }
	method(arg) { /* etc. */ }
	}

view raw

decorators.js

hosted with ❤ by GitHub

This is a lot more concise and natural, and the property decorators are similar to the equivalent in PyGObject, but unfortunately it doesn’t exist. Decorators are still only a proposal, and none of the major browser engines implement them yet. Nonetheless, we can take the above syntax as an inspiration, use a class expression, and move the registerClass() function around it and the GObject stuff outside of it:

	var MyClass = GObject.registerClass({
	GTypeName: 'MyNamespaceMyClass',
	Implements: [Gio.Initable, MyInterface],
	Properties: { 'prop': GObject.ParamSpec.int( /* etc. */ ) },
	Signals: { 'signal': { /* etc. */ } },
	}, class MyClass extends GObject.Object {
	constructor(props={}) {
	super(props);
	// etc.
	}
	get prop() { /* … */ }
	method(arg) { /* … */ }
	});

view raw

class.js

hosted with ❤ by GitHub

	var MyInterface = GObject.registerInterface({
	GTypeName: 'MyNamespaceMyInterface',
	Requires: [Gio.Initable],
	Properties: { 'prop': GObject.ParamSpec.int( /* etc. */ ) },
	Signals: { 'signal': { /* etc. */ } },
	}, class MyInterface {
	get prop() { /* … */ }
	method(arg) { /* … */ }
	});

view raw

interface.js

hosted with ❤ by GitHub

Here, the body of the class is almost identical to what it would be with the decorator syntax. All the extra stuff for GObject is contained at the top of the class like it would be with the decorators. We don’t have the elegance of the property decorator, but this is quite an improvement on the first iteration. It’s not overly magical, it even acts like a decorator: it takes a class expression, and gives back a GObject-ized class. And when decorators eventually make it into standard Javascript, the basic idea is the same, so converting your code will be easy enough. (Or those who use transpiling tools can already go ahead and implement the decorator-based API.)

This is the best API I’ve been able to come up with so far. What do you think? Would you want to use it? Reply to this post or come talk to me in #javascript on GNOME IRC.

Next steps

Note first of all that none of this code exists yet. Depending on what feedback I get here, I hope to have a draft version working before GUADEC, and around the same time I’ll post a more detailed proposal to the javascript-list mailing list.

In addition, I will be speaking about this and more at GUADEC in my talk, “Modern Javascript in GNOME“. If you are attending, come talk to me there! Thanks to the GNOME Foundation for sponsoring my travel and accommodations.