launchpadblog

The new bug listings listings were the first time my squad, the Orange Squad, had a chance to work on some really nice in-depth client-side UI since our squad was formed. Not only were we implementing the feature, we wanted to lay groundwork for future features.  Here are some of the new things we’ve done.

Synchronized client-side and server-side rendering

Early on, we decided to try out the Mustache template language, because it has client and server implementations. Although we wanted to make a really responsive client-side UI, we also wanted to have server-side rendering, so that we’re not broken for web crawlers and those with JavaScript disabled. Being able to use the same template on the server and the client seemed ideal, since it would ensure identical rendering, regardless what environment the rendering was done in.

It’s been a mixed bag. We did accomplish the goal of using a single template across client and server, but there are significant bugs on both sides.

The JavaScript implementation, mustache.js, is slow on Firefox. Rendering 75 rows of data takes a noticeable length of time. If you’re a member of our beta team, you can see what I mean. Go to the bugs page for Launchpad itself in Firefox. Click Next. Now click Previous. This will load the data from cache, but it still takes a visible length of time before the listings are updated (and the Previous link goes grey).

mustache.js also has bugs that cause it to eat newlines and whitespace, but those can be worked around by using the appropriate entity references, e.g. replacing “\n” with “
”

The Python implementation, Pystache, does not implement scoping correctly. It is supposed to be possible to access outer variables from within a loop. On the client, we use this to control the visibility of fields while looping over rows. On the server, we have to inject these control variables into every row in order for them to be in scope.

We needed a way to load new batches of data. Mustache can use JSON data as its input. Launchpad’s web pages have long had the ability to provide JSON data to JavaScript, but Brad Crittenden and I recently added support for retrieving the same data without the page, via a special ++model++ URL. This seemed like the perfect fit to me, and it’s turned out pretty well. Using the ++model++ URL rather than a the Launchpad web service means the server-side rendering can tightly parallel the client-side rendering.  Each uses the same data to render the same template.  It also means we don’t have to develop a new API, which would probably be too page-specific.

Client-side Feature Flags

While in development, the feature was hidden behind a Feature Flag. But at one point, we found we wanted access to feature flags on the client side, so we’ve now implemented that.

History as Model

We wanted users to be able to use their browser’s Next and Back buttons in a sensible way, especially if they wanted to return to previous settings. We also wanted all our widgets to have a consistent understanding of the page state.

We were able to address both of these desires by using YUI’s History object as a common model object for all widgets.  History provides a key/value mapping, so it can be used as a model.  That mapping gets updated when the user clicks their browser next and back buttons.  And when we update History programmatically, we can update the URL to reflect the page state, so that the user can bookmark the page (or reload it) and get the same results.  Any update to History, whether from the user or from code, causes an event to be fired.

We’re able to boil the page state down to a batch identifier and a list of which fields are currently visible. The actual batches are stored elsewhere, because History isn’t a great place to store large amounts of data.  For one thing, there are limits on the amount of data that can be stored.  For another, the implementation that works with old browsers, HistoryHash, can’t store anything more complex than a string as a value.

All our widgets then listen for events indicating History has changed, and update themselves according to the new values in the model.

Summing up

It’s been an interesting feature to work on, both because of the new techniques we’ve been able to try out, and because we’ve been closely involved with the Product team, trying to bring their designs to life.  We haven’t quite finalized it yet, but I’m going on leave today, so I wanted to let you know what we’ve all been up to.  Happy holidays!

I recently posted about Initializing page JavaScript from the JSONCache. Now I’m pleased to announce that you can also get updated copies of the IJSONRequestCache, to make it easier to update your page.

Brad Crittenden and I started work on this at the Dublin Thunderdome, and it’s finally been deployed. What this means is that for basically any page on Launchpad, you can append /++model++ to the URL, to get a fresh copy of the IJSONRequestCache. With ++model++, a change will typically require only two roundtrips; one to make a change, and one to retrieve an updated model. Future work may reduce this to a single roundtrip.

Why ++model++, not ++cache++? Cache is a really poor name for what the IJSONRequestCache is. Rather than providing fast access to whatever data has been previously retrieved, it is a complete collection of all the relevant data.

In Launchpad, the IJSonRequestCache is associated with the view, so we’re trying to rebrand it as the “view model”. This may seem strange from an MVC (Model, View Controller) perspective, but MVC can be recursive. A view may use a model to render itself.

DRY is one of my watchwords. That’s “Don’t Repeat Yourself“, of course.

If you’re designing your page to work with-and-without JavaScript enabled, then you’ll likely be tempted to repeat some of the
user-interface text in the template and in the JavaScript file. Especially for text that appears only some of the time.

One alternative is to render that text all the time, and only show it some of the time. In Launchpad, you add the “unseen” class to text in order to hide it.

In YUI3, it’s easy to add or remove classes using Node.toggleClass(). For example:

foo.toggleClass('unseen', baz);

This adds the ‘unseen‘ class to foo when baz is true, and removes it when baz is false.

This approach means you don’t need to repeat the text. It means your HTML stays in an XHTML file, with all the syntax-highlighting and other tool support that implies. It makes it easier to live with the fact that Launchpad don’t have a hardcore templating system in JavaScript.

It doesn’t eliminate all issues. If the page needs to have more than one instance of the text, this approach doesn’t directly apply. However, it’s easy to imagine extending it by using a hidden copy as a prototype and copying it as needed.

One disadvantage is that testing becomes harder. With conditional rendering, you can just check whether the text is rendered. With conditional visibility, you need to check whether the text has the unseen class, which is harder. If plan to apply this technique, it’s best to start out using it, or else you’ll have to rewrite all your tests when you switch approaches.

Another issue is that you will probably need to leave placeholders. For example, one of our conditionally-rendered strings included a link to a branch. When there’s no assigned branch, that text must be hidden. When someone assigns the branch via AJAX, we need to have a placeholder for the branch link, so that we can set it to the correct value with Javascript. At present, there’s no way to generate placeholders for arbitrary types, so it must be done manually.

Launchpad has a nice feature for initialising on-page JavaScript, called the JSONCache. I recommend considering it as your first choice when writing JavaScript that needs to be initialised. The one downside is that due to bug #740208, it’s not available for users who aren’t logged in. But then, users who aren’t logged in don’t need a lot of JavaScript, because they can’t write to most of Launchpad.

I found out about the JSONCache when we were working on he sourcepackage +sharing-details pages recently. This JavaScript has model objects, but I didn’t know how we should initialise them. We considered initialising them from the page HTML. We considered using the web service. But it turned out there’s already a great facility for this: the JSON cache.

If you’ve hacked on Launchpad’s HTML, you probably already know that every you can get LP.cache.context and LP.cache.me from any Launchpad page. These come for free, but with a little more work, you can get any object you need:

        cache = IJSONRequestCache(self.request)
        cache.objects.update({
            'productseries': self.context.productseries,
            'upstream_branch': self.upstream_branch,
            'product': self.product,
        })

These entries will then appear in the LP.cache with the names specified in the dict.

However, they are provided as plain JavaScript mappings, unlike the values returned by lp.client.Launchpad. You can use convert_cache to create a copy of the cache containing lp.client.Entry values. This allows you to intermix values from the lp.cache and the web service freely.

convert_cache currently lives in translations/javascript/sourcepackage_sharing_details.js, but should probably be moved into lp.client.

One extension I’d love to see would be a way to retrieve an updated copy of just the JSONRequestCache for a page. That way, we could reduce our number of round-trips, and also be confident that everything was up-to-date.

I recently came back to JavaScript, having largely ignored it for a couple of years. Development in JavaScript is much nicer than I remember. I thought I should share what I’ve learned.

The first thing I’ll mention is YUI tests. If you’re a TDD developer (and if not, why not?), YUI unit tests are the way TDD should be. The change/test cycle is very fast (to run the tests, hit “reload” in your browser). And since they’re proper unit tests, they feel like a safety net when you start to make changes.

They are a good way to test your model code and HTML changes. Other members of the Launchpad team have used them to do more integration-style testing, as well. It’s somewhat difficult to set up I/O dependent tests, but that will encourage you to make most of your code I/O-independent.

For information on how to get started with them, see: Developing with YUI.Test

Launchpad’s test suite takes a long time to run. Far too long to wait for when you’ve made a change. And the likelihood that you’ve broken any given test is pretty small. So you probably want to start by running the tests that were likely to break.

You can usually guess which tests to run; if you’ve changed “archive.py“, you should probably run “test_archive“. But some connections are easier to miss, so I’ve hacked up a Fault Line, a bzr plugin that uses past changes to guess which test files correlate to the files you’ve changed. You can run it like so:

bin/test -m $(bzr fault-line --module-regex)

This will look at all the files you’ve changed, look at their recent history, and see which files tended to change in the last 100 revisions where you changed the specified files. The --module-regex option causes it to output a regular expression, assuming that the test files are Python modules. Otherwise, it would just output a list of the test files it found.

Thanks to Jelmer Vernoij, this is even easier to achieve for testing bzr. You just need to run “bzr selftest --auto“.

To install Fault Line, just run:
bzr branch lp:fault-line ~/.bazaar/plugins/faultline

It currently requires bzr 2.3 or later (e.g. stock Natty bzr).

Fault line is pretty limited right now. For example, the way it guesses what’s a test file is hard-coded. But a few days ago, it wasn’t even a proof of concept. Who knows what the future holds?