November Bug Squash Month: GJS

Here’s what happened during November Bug Squash Month in GJS.

First off, I didn’t really get on the ball to promote Bug Squash Month and I didn’t take pictures of any bug squashing activity… which I regret. I hope this post can make up for some of that.

During November I finally took the leap and offered to become a maintainer of GJS. My employer Endless has been sponsoring work on bugs 742249 and 751252, porting GJS’s Javascript engine from SpiderMonkey 24 to SpiderMonkey 31. But aside from that I had been getting interested in contributing more to it, and outside of work I did a bunch of maintenance work modernizing the Autotools scripts and getting it to compile without warnings. From there it was a small step to officially volunteering.

With not much of November remaining and a holiday and family visit coming up (life always is more important than bug squashing!) I decided to start out my bug-squashing campaign with what would get me the most results for the time spent: going through GJS’s bug tracker and closing obsolete or invalid bugs. This I managed to do, closing about 1/4 of all open bugs!

Then I made a list of all open bugs with attached patches and intended to review them to see if they still applied and why they hadn’t been committed yet. I got through a few, and had the dubious distinction of fixing up and committing patches from a 7 year old bug yesterday. But as you can see in the list, there are still 54 remaining. A good to-do list for the next Bug Squash Month, or whenever I feel like working on GJS but don’t know what to work on!

Did you know Bugzilla could generate graphs? I didn’t! Here’s a graph of the total bug count in GJS during November Bug Squash Month:

chart

The clunkiness of this chart kills me though…

My plans now that Bug Squash Month is over are to concentrate on fixing things that make it more pleasant to use and contribute to GJS:

  • Find an active co-maintainer so that we can review each other’s patches (could this be you?)
  • Make ES6 Promises available (this work is also being sponsored by Endless)
  • Rework the test suite to use an embedded copy of Jasmine so that writing automated tests becomes less of a pain
  • Find ways to bring in some of the conveniences that Node developers are used to

I’ve also decided to try an experiment: I’ve just made the Trello board public on which I keep track of what I’m working on and what I’d like to work on. Let me know if this is interesting to you and what features you might like to see on there! (It’s made possible by a Chrome extension, Bug 2 Trello.)

All in all November Bug Squash Month was a success, though next time I will get started earlier in the month. Come join me next time!

gnome_bugsquash_nov_2016

 

Would you write a 911 location app?

John Oliver talked in his show’s most recent episode about the US emergency services phone number, 911. It seems that now nobody uses land lines anymore, sometimes the emergency services have a hard time locating people from their cell phones.

John Oliver: “And if you’re thinking, ‘wait a minute, I can find my location on my cell phone,’ you’re not alone. Dispatchers wonder the same thing.”

Dispatcher: “I can check in on Facebook and it’ll tell you exactly what building I’m in. […] But when you call 911 we don’t get that accurate location information. The technology’s out there, it’s just not getting to us at this point.”

JO: “That’s a good point, because even the Domino’s app can tell where you are, and they’ve barely mastered the technology to make a palatable pizza! So we asked […] why it seems Ubers can find you better than ambulances can, and there doesn’t seem to be a simple satisfying answer.”

Here is my best guess at that answer, as a software engineer. Our industry has a pervasive culture of rush-jobs that get 90% of the way there and then save the rest for version 2; move fast and break things, yada yada. No emergency services provider would adopt it because it would not be reliable.

It’s reasonable to think that 90% would be better than what 911 apparently has now, which according to the video is sometimes only accurate to the nearest cell tower. However, the litigious nature of US society makes that impossible. The first time the software failed, the maker would get sued out of business.

Thus we are stuck, because we teach ourselves not to go the extra mile; and even if we went it, no-one could afford to take responsibility for making things better.

GJS and Autoconf

Here’s a leftover from the GNOME Developer Experience Hackfest that I participated in back in January: in my last post about it, I mentioned I had worked on some Autoconf macros for GJS that got added to the Autoconf Archive.

My plan was to port an existing GNOME application written in GJS to use the new macros, as an example for other projects to follow. I did so for GNOME Documents a while ago, but then forgot all about it as the patches sat spoiling on my hard drive. Recently I fixed that up and submitted a patch as I should have done long ago.

Here’s the guide to using the new macros in your project:

  • Replace any AC_PATH_PROG([GJS], [gjs]) (or, possibly, pkg-config --variable=gjs_console gjs-1.0; I might add this to the macro) with the shorter AX_PROG_GJS.
  • Use AX_CHECK_GIRS_GJS to check that you have the correct API version of each introspected dependency that you import in your code. Easy rule of thumb in a one-liner: git grep imports\.gi\. | cut -d: -f2 | awk '{$1=$1};1' | sort | uniq will pick out all the GIRs that you import. (Take special note for Cairo: its GIR name starts with a lowercase letter!)
  • Since GIRs don’t have a concept of versions other than the API version, use AX_CHECK_GIR_SYMBOLS_GJS to check for APIs that you use that aren’t available in all versions. For example, if you use the new GtkShortcutsWindow in your code, don’t try to check for GTK 3.20; instead check for the ShortcutsWindow symbol using this macro. (Use this sparingly, of course; there’s no need to check exhaustively for every new API that was added since 3.0, only the latest that you use!)

You might ask “why should I check these dependencies at build-time when they are only necessary at runtime?” You would be correct, it’s not necessary to check them at build time. However, these macros were originally born of the frustration that happens when you “make all install” a tarball only to run it and find out you’re missing GIR dependencies. And especially in the case of using new API, like ShortcutsWindow that I mentioned above, your program might even start up correctly and crash halfway through when you try to open a ShortcutsWindow. It’s a courtesy to your users (not to mention downstream operating system packagers, for whom your runtime dependencies might not be obvious.)

The macros are now also available in your JHbuild setup, through the m4-common module. How that module works is not very discoverable, and Philip W was kind enough to explain it to me when I asked how to get the new macros in. So I’ll explain it again here for future reference.

M4-common contains the agreed-upon set of M4 macros from the Autoconf Archive that GNOME applications use for building. It’s a Git repository that pulls in a known version of the Autoconf Archive as a Git submodule, and installs the M4 macros from it that GNOME applications need to build. If your application uses any of these macros, then it should have a dependency on m4-common in JHbuild.

As written on the GNOME Wiki, if you get an error like

./configure: line 12053: syntax error near unexpected token `GOBJECT_INTROSPECTION_CHECK'
./configure: line 12053: `AX_REQUIRE_DEFINED(GOBJECT_INTROSPECTION_CHECK)'

then you probably need to build m4-common and make sure you have a dependency on it.

Note that you don’t need to care about any of that if you’re building from a released tarball; that’s because Autotools bundles all the macros you use in the tarball when you run “make dist”.

Thanks to Philip Withnall and Cosimo Cecchi for code review and good ideas; and thanks to Endless Computers for allowing me to contribute these macros developed on company time to the Autoconf Archive.

Healthy Code

I work on a team of software developers that maintains several large codebases — too much code for any one person to easily know what’s going on in every part of it at any particular time. I found myself thinking a lot about how to keep the code healthy and a while ago I set my thoughts down as a list of good practices. Thanks to my coworkers at Endless for input, editing, and debate.

The good practices in this post differ slightly from the ones we adopted at work, which reflect the opinions of the whole team; these are worded to reflect my personal opinions.

Assumptions

I don’t like rules without a rationale. I believe these six assumptions underlie the rules that I set out below. That is, if you don’t agree with these assumptions then you probably won’t agree with the rules… ☺︎

  1. We can never know that our own code is correct.
  2. Left unchecked, we will believe our own code to be correct.
  3. Even small mistakes can lead to catastrophic data loss.
  4. Non-trivial programs have interconnections too complex to keep entirely in one person’s mind.
  5. Modifying non-trivial programs will break code unrelated to the modifications.
  6. The business value of maintainable code is only visible to developers.

Good practices for code health

Use your judgement

As always, rules apply only in the absence of any overriding reason to ignore them. Breaking them should be in mutual agreement between the writer of the code and the reviewer. (This system only works if everyone agrees about what the rules are in the first place, though.)

Example reason to break this rule: If no agreement can be reached, then the default is to follow the coding standards.

Review code

Code gets reviewed by a developer who didn’t write any part of it, because of assumptions #1, #2, and #3 — and to spread familiarity with different parts of the codebase throughout the team. Develop with ease of reading in mind, as if you are writing a letter to an unfamiliar code reviewer. Review code skeptically and with full attention, as if it came from a malicious agent out to erase your hard drive.

Example reason to break this rule: You are committing a trivial fix for a broken build and your continuous integration system acts as the code reviewer.

Observe the style

Code follows the coding style. Coding style is important because when code looks the same it’s quicker to read and errors jump out more easily. Apply automated tools when possible to save the code reviewer from becoming a parenthesis counter.

Example reason to break this rule: The code reviewer agrees with you that deviating from the style is more readable.

Test your code

Code needs automated tests. The rationale for this is assumptions #2 and #5, but could be the subject of an entire blog post itself. Lack of tests can be by itself a reason to fail code review, or at least start a dialogue between developer and reviewer about why tests are not necessary in this particular case.

Example reasons to break this rule: A one-off script. A component that proxies an external resource which can’t easily be mocked out.

Refactor on write

You will always have to deal with legacy code (code on which development has ceased but still must be maintained) and rushed code (code which you were forced by circumstances to check in that didn’t quite work well, works but is difficult to maintain, or is not tested.) By assumption #6, you will probably never set aside time to refactor code for its own sake. Therefore, refactor bit by bit to leave the code in a slightly better state each time it’s touched. In this way, code receives refactoring attention roughly proportional to the benefit you receive from refactoring it. If at all possible, add new code with a unit test even if the rest of the code is not written in a testable way.

Example reasons to break this rule: The code is already in good shape. The feature is critical and cannot be delayed. You are contributing your code to an open source project, in which case it is better to work with the upstream community to refactor.

Refactor only on write

Make your diffs per commit no larger than they have to be, in order to make code review easier. Since diffs go line-by-line, do not fix style errors in lines that that are not already being touched in the same commit. Use separate commits if there is an opportunity to make other style fixes.

Example reason to break this rule: If it makes more sense to fix lines other than the ones being edited in one shot (e.g. large sections with wrong indentation), do so throughout the whole file in a separate commit.

Pay down technical debt

Sometimes it’s not possible to build a feature without doing a large refactor first. Determine this as early as possible and include it in the time estimate for the feature. Do not shy away from paying down this debt; it will only compound if you borrow more on top of it. However, keep the changes incremental, and the functionality unimpeded while making these changes.

Example reason to break this rule: Extreme time constraints force you to take out a second mortgage on the code (even then, do this only with a healthy dose of disgust.)

Moving text messages between Android phones

I recently got a new Android phone secondhand, and after resetting it I wanted to move the text message archive over from my old phone. It turns out that you can do this easily if you have root access. Well, technically you can do anything easily if you have root access, but the trick is knowing how. I hope that by putting this out on the internet, other people will be able to know how too.

I had root access on both phones, as they were flashed with CyanogenMod. The new phone is a Nexus 4, and the old phone is an HTC G1 (Android 2.2 is the highest that could run on it.)

On both (and as far as I know, all) versions of Android, all the text messages are stored in this file, which you need root access to read:

/data/data/com.android.providers.telephony/databases/mmssms.db

Getting the file off the G1 was easy; I entered the Terminal Emulator app (I think it’s installed automatically when you flash CyanogenMod) and copied the file to the SD card:

su
cp /data/data/com.android.providers.telephony/databases/mmssms.db /sdcard/

(su requests superuser permissions, which you have to grant.) Then I connected the G1 to my computer with its USB cable and transferred the file off of it.

Getting the file onto the Nexus 4 was harder. What I did not know is that the Nexus 4 can’t mount its SD card as USB Mass Storage (see the explanation), so I ended up using my Apple laptop to do the transfer, and had to download a program called Android File Transfer. Still, I got it onto the phone’s SD card.

Since the newer version of Cyanogenmod comes with a file manager app, I decided to use that to put the file into the correct place, instead of Terminal Emulator (typing shell commands on a phone is no joke.) The file manager is set to “Safe mode” by default which means it won’t request root access. I changed it to “Prompt User mode” in the settings, then navigated to the above databases/ folder and made a backup copy of the old (empty? 100 KB? It’s a sqlite DB so maybe there are still deleted records in there, but I don’t care to check) database. Then I copied the G1’s mmssms.db file over top of it. Unlike on the G1, there was also a mmssms.db-journal file there, which I hoped wouldn’t mess with things…

I couldn’t see my text messages after going into the messaging app, but after rebooting the phone, they were there.

Documentation for GJS

In the last days of January I attended the GNOME Developer Experience hackfest, remotely. This was a first for me since I have not attended a hackfest remotely before. It was also a first for the hackfest, that three remote participants signed up, and we had a discussion on desktop-devel-list as to how that should work.

My main project for the three days was to show people the documentation browser I’ve been working on. This got started a while after last year’s edition of the DX hackfest, which I didn’t attend, but the project was born out of a discussion with some people who were there and talked about it last year.

Some background: You can write applications for the GNOME desktop in many different programming languages, including C, Python, Javascript, and Vala; this is made possible through a piece of software called GObject Introspection, which is a bridge between C libraries and many other programming languages. At (yet another) DX hackfest, the one in 2013, GNOME decided to promote Javascript (and the GJS interpreter, which is basically the Javascript engine from Firefox 24 with GObject Introspection on top) as its preferred platform for application development. Not the exclusive platform, you can still develop apps in Python if you want, but there would be a focus on getting new developers started in Javascript, with tutorials and documentation.

Sadly, since then, we haven’t had good API documentation for developing apps in GJS. The best story we had for a long time was to tell people to refer to the C, Python, or Vala API references and mentally make the transition to Javascript, a process I’ve compared to playing a song on the guitar using chord sheets from the internet, but mentally transposing them to a different key as you go along. At one point, Giovanni Campagna generated some static Javascript documentation and hosted it on his website. This eventually got out of date.

So, this was the problem I was hoping to solve. After a short-lived attempt to write my own web app using Semantic UI (a technology I’d still like to check out someday!) I figured, why bother when there was something that would fit the bill perfectly well: DevDocs, an open-source documentation browser that combines all kinds of documentation from elsewhere (mostly web development technologies) and presents them in a unified fashion with a useful but simple and elegant web app.

In the final quarter of 2015 I worked on this on and off, adding some code to the documentation generator g-ir-doc-tool to make its output suitable for DevDocs to input, and some code to a forked version of DevDocs to import the GNOME documentation and make nice metadata and sidebar entries.

Screenshot of Javascript documentation on DevDocs

Here it is in action

At the hackfest I gave a demo (at 6:30 AM, because of the timezone difference!) to the other people there who were working on developer documentation. I got some good feedback from the people who were listening, discussed some of the shortcomings, and we discussed Mathieu Duponchelle’s new documentation tool, Hotdoc, as well. One good outcome was that the GNOME Developer Center now links to my instance of DevDocs (noting that it’s “experimental”.)

To talk about one of the limitations that we discussed at the hackfest: The approach I chose generates all the documentation from one file, the “GIR file,” which is the same approach that g-ir-doc-tool has always taken. This is nice because it’s self-contained, but also incomplete: API references often include separate pages not connected to any API in particular, such as this one. I think using Hotdoc will help overcome this limitation, since Hotdoc is made for combining these sorts of things. I’m also happy not to work with g-ir-doc-tool since all the stuff I added to it was basically bolted on the side, not really useful for anything else, and therefore not likely to be accepted upstream.

I’m now hosting an instance of the GNOME Javascript API documentation on my modified version of DevDocs: try it out at http://docs.ptomato.name:9292/.

During the hackfest I also collaborated with Philip Withnall on some autoconf macros for GJS and GObject Introspection, which I’ll talk about in a following post…

Some of the other stuff that people worked on is summarized on Philip W.’s blog.

Geek tip: Malloc debugging on OSX

I’ve been trying to chase down an annoying bug that I suspected to be a case of using uninitialized memory. The problem is, it only shows up about 1 in 30 times (I was lucky to notice it in the first place), and never in a debugger.

Fortunately I found that there’s a library on OSX that tweaks malloc() to help you debug:

DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib MallocPreScribble=1 ./myprogram

Or, to do this in LLDB, since due to System Integrity Protection, your linker-affecting environment variables get wiped when you execute a system program:

lldb -- ./myprogram
env DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
env MallocPreScribble=1
run

This triggered the bug every time, both inside and outside the debugger.

For more information about what you can do with libgmalloc, see this documentation. It only tells how to use that facility in Xcode, though, so the above instructions should help if you’re on the command line.