Friday, March 26, 2010

MSR Video: Research Perspectives on JavaScript

Just today, Channel 9 (the MSDN channel that covers Microsoft Research) posted a video about "Research Perspectives on JavaScript", featuring Eric Meijer as interviewer and the Bens (Ben Livshits and Ben Zorn) as interviewees. This video is particularly interesting to me, because their JSMeter project is highly related to our PLDI paper this June (we even get several mentions in the video). I'll summarize some of the main points of conversation, but will omit some details as the video is quite long (50 mins). My comments/opinions will be interspersed and emboldened.

Eric begins by asking about the names. How do they always come up with funny names like Gatekeeper, JSMeter, and so on? Ben Zorn explains that it is important to pick a good name, because they tend to stick in people's minds better than paper titles (unless authored by Phil Wadler). I agree. That said, I would much rather have a boring paper title than a ridiculous backronym project name, the likes of which are way too common on large projects in the sciences.

Next, the JSMeter project is discussed. The project goal is to find out what exactly JavaScript code in the wild is doing, and how it compares to C, Java, or other languages. They instrument Internet Explorer's interpreter, and aim to measure the behavior of *real* applications that end-users visit every day.

As far as their methodology (as in, what data is actually recorded), it is very similar to what we did in our research. There are some differences: they measure physical heap usage of the DOM vs JavaScript whereas we only measure the objects on JavaScript heap (without respect to physical size of objects), and they measure callbacks and events explicitly. As far as analysis of the data, our approaches diverge according to our goals, but cover many of the same statistics.

The first point Ben Livshits talks about (and their main conclusion in the paper) is their observation that SunSpider and other JavaScript benchmarks do not have much in common with real-world applications such as GMail, Facebook, Bing Maps, etc. The second observation is that function callbacks are typically very short on sites that need high responsiveness. SunSpider is called out for having a few event handlers with very long execution times, which biases against interpreters that handle small functions well (such interpreter behavior is desirable in the real world).

I'm not objectively sure what "short" is, but we did see in our work that function size was fairly consistent with respect to static code size and dynamic bytecode stream length. We could not distinguish function invocations which were callbacks though, unfortunately. I also wonder if opportunities for method-level JIT'ing are overrepresented in SunSpider and (especially) V8 benchmarks.

There was some discussion of whether JavaScript as a language will evolve to be better, and also the tricky questions of what would one add or remove to the language. Ben Zorn points out that JavaScript, unlike Java or C, is usually part of a complex ecosystem in the browser. This means that ultimately, it may evolve, but only slowly and in lockstep with other complementary languages. He also calls it a "glue" language as opposed to a general purpose language, one that mainly deals with making strings and gluing together DOM and other technologies.

I agree that it is bounded by other technologies, but it can also be a general-purpose scripting language (see for instance its use in writing large parts of Firefox, and as a scripting plugin for many environments and games). I think the issue of poorly designed semantics (the root of all trickiness in efficient implementation) is orthogonal to the issue of whether it's a generally expressive and useful language. PHP is another language in this vein (apparently useful, but horrible semantics).

Erik asks about the use of dynamic features of JavaScript by developers. Ben Livshits immediately concedes that many people use eval, and while some of its uses are easy to constrain/replace with safe behavior (JSON parsing, for example), some are "more difficult". But, he does not see this as a very big problem because a lot of contemporary JavaScript code is written by code generators. Ben Zorn explains that with results from JSMeter and our work, researchers and implementors can gauge the impact of certain restrictions (such as saying "no eval" or "no with").

We actually approached it from the other end in an effort to investigate the implications of assumptions already made in JavaScript research. Since our sources and data are freely available on the project webpage, it's possible to go in the other direction as well by tinkering with our framework and replaying executions on an interpreter simulator with different semantics.

Our conclusions are a bit different in this area, as well. You can read the paper for more details, but shortly, we think static typing and static analyses for JavaScript will either be too brittle, too expensive (due to code size), or too flexible to make any useful guarantees. That said, we see lots of room for heuristic-based optimizations, which have already made inroads into implementations of Chrome and Firefox.

We learn that there is a dichotomy between sites that use frameworks and libraries, and sites that use handwritten code. We saw about half of the top 100 sites used an identifiable libary. Script size is also seen to be very large. Erik asks about the functional nature of JavaScript- do scripts often use higher order functions? They defer to our study for quantitative numbers (thanks for the mention) and say that usually it is frameworks and translators that use HOF's (for example, JQuery's extensive use of map/reduce patterns and chaining). Of course, callbacks and event handlers are one ubiquitous use of closures. Ben Livshits talks a bit about library clashes (i.e., different libraries may change built-in objects in incompatible ways), which they did some work to detect statically in other research. I know that Benjamin Lerner at UW has done some work in this space, in the context of Firefox plugins and how to make them play nicely together. He makes an anonymous jab at some news sites that sidestep such incompatibilities by loading separate widgets in their iframes (at the expense of horrible page load times).

Erik returns to the issue of language design: what would you like to remove or add to the language? Ben Livshits talks about subsets of JavaScript, and their usefulness for writing safe and limited code (such as in ads). This approach has been used in several papers, but does not yet seem to have much traction with browser vendors. It would be nice, though. In general, it is agreed that JavaScript needs fewer features, not more. I would start with removing the 'with' operator. Ben Zorn says that nothing needs to be added, because things like classes or other language features can be built on top of prototypes. That said, he is not convinced either way as to the usefulness of a prototype-based vs a class-based object system. Yeah, me neither. He then explains prototype-based objects and ways to optimize programs in this paradigm, such as V8's "hidden classes".

Ben Livshits says that the big strength and weakness of the Web are the same thing: being able to load code and data from disparate servers, and combine them fairly arbitrarily via very late binding. Predictably, the two Bens are at odds over whether this is a good thing or bad thing for the web in the large. On the one hand, having no rules makes it easy for a 12 year old (or a developer with the same skill level) to hack something together because the platform is so tolerant to broken code. On the other hand, this flexibility invites a lot of security problems and ties the hands of those more proficient developers who want more invariants and guarantees about their code. This lack of discipline and control is probably what drives companies to translate large programs into JavaScript from some other language like C# or Java.

One lesson of JSMeter that Ben Livshits talks about is the possible benefits of more integration between the language and the platform. Many times, browsers load the same page over and over, but do not learn anything about how that page actually behaved. Ben's example is that if code only runs for a few seconds, then it is not useful to run the garbage collector (as opposed to other methods, such as mass freeing by killing a process or using arena/slab allocation). Right now, browsers are utterly amnesic about what happened the last time (or 10, or 1000 times) they loaded a page, and only cache the source text on the browser (as opposed to the parsed AST). This is something that jumped out at me as well. Sounds like an interesting thing to look at. They talk about this again near the end.

Erik asks whether the parallel but separate specifications and implementations of JavaScript and complementary technologies like the DOM are necessary. Why not just make one specification to rule them all? Both Bens say that the border is fairly arbitrary, and increasingly applications pay a large price when they cross that boundary frequently. Ben Livshits also says that going forward, it is a bad idea to ignore these other technologies when thinking about analyses and optimizations. They did not suggest any specific methods for such cross-border optimizations, though.

This is a huge problem with tracing JIT's like TraceMonkey, because they have to end a trace whenever it leaves JavaScript for native DOM methods (usually implemented in C++). V8 (Google Chrome's JavaScript engine) tries to minimize the number of such exits by implementing most of the standard library of JavaScript in JavaScript, and using only a small number of stubs. Another approach may be to compile the browser with the same compiler that does the JIT'ing (say, with LLVM) and then there is less penalty for crossing the DOM/JavaScript execution boundary in machine code.

Ben Zorn goes as far to claim that JIT's can only go so far to improve JavaScript performance, and DOM interactions are lower-hanging fruit right now. He bases this on the fact that most scripts are not computation-heavy, either because they are interactive and wait on the user to create events, or spend most of the CPU time inside of native DOM methods. Ben Livshits thinks that one of the biggest challenges is that JavaScript (and web applications in general) are network-bound, instead of CPU or memory-bound. Essentially, download time and network latency dominate any other sources of delay.

I agree on the 'being interactive' part, but disagree on the compute-heavy part. Especially with things like games, the canvas element, and animations in JavaScript, numerical computation is starting to become significant. Furthermore, as JavaScript becomes more and more the 'assembly of the web', I would guess that the CPU time will tilt towards general-purpose computation, and away from DOM calls (which most significantly are used for the V in MVC).


It's great to hear at length from some of the other folks doing research work around JavaScript. I'm looking forward to seeing the final version of the JSMeter paper at USENIX Webapps, and also am looking forward to our paper being presented at PLDI in June. Every time our work is presented, we get lots of new and diverse feedback, which raises ideas we have not yet considered and forces us to dig deeper into our understanding and data.

No comments:

Post a Comment