My good friend Tobias Metzke and I worked on extending the Cloud9 IDE with some tool support for CoffeeScript and Jasmine. This was done during a seminar about web based software development at our home university. Our results can be seen in this screen cast. For the long version you may check out our paper – I converted the LaTeX to HTML and you can read it here, but be careful, some stuff might not work as expected (links etc.).
That’s why I’d recommend you to read the PDF if you’re interested in what we did.
That’s it, have a nice weekend and keep on coding!
CoffeeScript & Jasmine Testing – Tool Support for Cloud9 IDE
Tobias Pfeiffer, Tobias Metzke
Proper application testing leads to higher quality code on the long term. Thus, development environments encouraging and supporting programmers in software testing clearly benefit the quality of the produced code. This paper investigates the capabilities of web-based IDEs in terms of software testing support and integration. An example IDE has been explored by an prototype application written in CoffeeScript using the TDD methodology. Furthermore, the shortcomings discovered have been countered by proper tool support, integrated in the IDE. The tools and their benefits described in this paper contribute to a more encouraging web-based IDE that could serve as a replacement for traditional development environments in the near future.
Applications that used to be on desktops only can now be found in browsers with all their former functionality at hand. What is more, the capabilities of the world wide web are used to enrich these systems even more.
Software development environments are likely to follow this trend as well. In this paper, we investigated the Cloud9 Online IDE  and its functionality. We chose this specific web-based IDE because it already had a good reputation among different development communities and was recommended by Trevor Burnham . To explore the capabilities of the IDE, we implemented an example application with the help of the IDE.
Furthermore, the investigation focused on the support for the Test-Driven Development (TDD)  work flow1 by the IDE. As described by Jeff Langr , writing tests in TDD methodology most often results in tests of higher quality and a cleaner code base. This is why we favor writing our applications in TDD methodology where applicable. We therefore regard the support of the TDD cycle as a quality measure of an IDE.
The insights gained by the research are discussed in this paper. As a our main contribution, we added tool support for the IDE based on the shortcomings found in this investigation. We described the new tools and displayed their benefits. This was done by comparing the work flows used before and after the tools were plugged into the Cloud9 IDE.
This paper first gives an overview of work related to ours in the following section. Afterwards, Section 3 describes the insights gained by investigating the Cloud9 IDE with the help of an example application. After presenting the new tool support in Section 4, this paper concludes with a brief summary in Section 5.
2 Related Work
Web-based software development environments are getting increasingly popular, which can be depicted from their sheer number , , . This trend most likely has its roots in the fact that more and more desktop applications are migrated to the web. Many of these IDEs work just like desktop applications, however there are other IDEs like Lively Kernel  leveraging the web-based concept more. These are more unlike regular desktop IDEs.
However the development environments tend to become in-browser application. Thus they need to compete with traditional desktop-based IDEs.
All of the online IDEs have in common that they are accessible from everywhere with nothing but a browser. They do not have to be set up and can be used from whatever computer while providing basically the same user experience. Depending on the concept of the IDE they even take load of the pc that is used for development, as the programs are executed on servers and not on the local machine. Collaboration features can also often be found in web-based IDEs.
There are also downsides compared to classic IDEs. Web-based IDEs need to be constantly connected to the Internet which is a well-known handicap. Another common drawback is that online IDEs are often not as feature rich as traditional desktop-based IDEs. This might be due to technological limitations and the fact that they are not as mature as their desktop siblings.
3 Working with the Cloud9 IDE
To investigate the potential of the Cloud9 IDE we developed an example application in CoffeeScript with the help of the IDE. The goal of this application was to recommend a set of articles that could interest the reader of a given article. The functionality of the application is further described in the following section.
3.1 Example Application
The example application CoffeeRecommender  is a recommendation engine based on the Plista contest. We used it to investigate the features of the Cloud9 IDE. Based on the information provided by a given web article3, this application finds related articles in a set of given articles4. The goal is to find the ones that appear most interesting to the reader of the given article. Figure 1 show the application, embedded in the work flow of its environment.
The application, written in CoffeeScript in the Cloud9 IDE, has been developed in TDD methodology. Therefore, we used the Jasmine testing framework. Consequently, every new feature was first manifested in a set of tests that described the way it is used and the expected results. The work flow we went through for implementing a new feature is illustrated and discussed in the next section.
3.2 Work Flow
Following a Test-Driven Development work flow when implementing the example application, we went through the development work flow shown in Figure 2 most of the time. It describes the steps we had to go through when a code change led to failing tests. All of the steps shown in this figure are an abstraction of all the actions the Cloud9 user has to go through.
The 5 phases consist of the following actions:
- write code
- The programmer writes tests and the feature code that should make these tests pass and saves the changes.
- type ‘npm test’ in console
- The programmer runs the tests by switching to the console, typing this command and running all the tests present due to missing possibilities of only passing on single tests for execution.
- find error in console log
- In the console output of the test results the programmer has to look for the error she produced with the code she wrote. Figure 7 in the Appendix shows an example of such an output. The time needed to find the corresponding line in that mass of lines can become a major barrier for testing the code at all.
- find line in CoffeeScript file
Ideally, the work flow to implement a new feature would be highly supported by an IDE to enable fast enrichment of the developed application. Unfortunately, when following TDD and using CoffeeScript in the Cloud9 IDE the programmer does not get the support needed for rapid development. How this has been improved by proper tool support by us is shown in Section4.
Besides these shortcomings in the usage of the IDE itself and in the support of the testing work flow, there are drawbacks to the plug-in development for the Cloud9 IDE as well. These are presented in Section 4.4, after the implementation of new tool support was displayed in the following section.
4 Enhancing the Cloud9 IDE
We wanted to mitigate the problems of our work flow with the Cloud9 IDE as described in section 3.2. Mainly we focused on the following problems:
- insufficient support for the TDD cycle
- lack of support for CoffeeScript
4.1 The Livecoffee Plug-In
4.1.1 The old livecoffee plug-in
4.1.2 Enhancing the functionality
We extended the existing functionality with our fork of the plug-in  in three different ways. We improved the approximate line matching, provided a visual aid for the line matching and enabled other plug-ins to use part of the functionality of this plug-in through loose coupling.
4.2 The Jasmine Plug-In
The goal for our Jasmine plug-in  is to make Test-Driven-Development with the Jasmine testing framework and CoffeeScript as comfortable as possible. Therefore we have integrated a panel into Cloud9, whose structure is similar to the one of the new test panel in Cloud9.
4.2.1 The panel
The panel is one of the common panels on the left hand side in the IDE. Here all recognized testing files are shown. In order to do this we rely on a convention: All test files are assumed to be in the spec folder of the project or in a sub-folder of the spec folder. In addition they should end in .spec.coffee, which is a convention introduced by Jasmine itself.
Whenever tests have been executed then this panel can be used to see every single test case, with its description, and whether it passed or failed. Figure 4 is a screen shot of the panel with some failing tests.
4.2.2 Executing Tests
We provide many different ways to execute tests. The distinct means are all present as they are more fitting for particular situations or developers. However the granularity with which you can execute tests is capped at the file they are defined in. So you can only execute test files separately not individual test cases. Following is a description of all the different ways you can execute tests with our plug-in at the moment:
- Automatically after file save
- Tests are executed automatically when a file is saved according to naming conventions. So whenever a file called implementation.coffee is modified the plug-in will look for a file called implementation.spec.coffee and execute it if present. Similarly if you modify a test file itself, then after you save it the very same file will be executed. This behavior heavily favors Test-Driven-Development which relies on frequent feedback.
- Manually execute all tests via short-cut
- You can press the Ctrl + J short-cut in order to execute all tests.
- Manually execute a test by double clicking
- As it can be seen in figure 4 the different test files are shown in the Jasmine panel. Double clicking on any of those files will execute it.
- Manually execute a selection of tests with the Run-Button
- You can select a sub-set of test files to run by clicking them while holding the Ctrl key. Afterwards you can click on the Run Tests button in order to execute this sub selection of tests.
4.2.3 Locating the error in the CoffeeScript file
Developers may access this feature by right clicking on the specific test case that failed and select Show Error in Coffee. The error message is displayed in our panel as well (see figure 4). If the programmer should need additional information, then he can still check out the complete error log including all stack traces on the Cloud9 console.
4.2.4 How the plug-in works
The plug-in itself is client-side, which means that it is executed in the environment of the browser. This is how the user interface and the communication with the livecoffee plug-in works.
However in order to execute tests we need to execute code on the server-side. This process is more complicated and a simplified version is described in figure 5. On the server side the tests are executed with the version of jasmine-node  that the given project has installed. It is assumed that npm  was used to install jasmine-node. This was a conscious choice as we do not want to introduce additional dependencies to the Cloud9 IDE. Moreover we do not want to force users to use a specific version of jasmine-node.
With the execution of tests being as it is right now, the only way to get a hold of the test results is parsing the console output generated by jasmine-node on the server-side. This results in some conventions and liabilities of the plug-in. It is a convention that the first describe block of a spec-file should begin with the name of the spec file. Otherwise there is no way for us to determine which tests belong to which file. Moreover the parsing can be considered rather brittle, if the jasmine-node plug-in ever significantly changes the format of their verbose output, then our plug-in will likely break.
It would be a better solution to accompany the client-side jasmine plug-in with a server-side plug-in. That way the server-side plug-in could run the tests and gather the results using a custom reporter5. The results gathered by the reporter could then be sent to the client-side plug-in as JSON objects. Due to the given limitations concerning time, man power and lack of documentation for Cloud9 server-side plug-ins and Jasmine reporters we chose to work with what we knew worked, although it might be a bit frail.
4.3 New Work Flow
The usual Test-Driven-Development cycle with failing tests in Cloud9, as described in section 3.2, has been significantly shortened and thereby improved by our two plug-ins. The new work flow is depicted in figure 6. There are basically just the following three steps in the development cycle:
- write code
- Write your code and then save it, this step remains unchanged.
- see failing test in panel
- As soon as you save a file ideally the corresponding test-file is executed automatically, alternatively it is possible to run tests as described in section 4.2. Now the left hand panel shows whether all tests pass or if some failed. For failing tests the error message is displayed.
- be guided to CoffeeScript line
- The user can right click on any failing test and select Show Error in Coffee. This results in the file of the project, where the error originated, being opened and the cursor to be placed in the line were the error occurred. Because of that the programmer is directly able to continue to write code to fix the bug.
When no error occurs during the testing cycle the circle is also shortened. The programmer just sees that all the tests pass and can continue to write code right away. This is opposed to switching to the console and typing npm test to run tests.
4.4 Extending the Cloud9 IDE with plug-ins
Our experience when extending the Cloud9 IDE has been mixed. We were able to provide our functionality solely through plug-ins, without modifying any code that we have not written ourselves. This is a sign of a very good plug-in system. However we found the documentation of the plug-in system and the IDE itself lacking. There is documentation for the Ajax.org Platform  and the Ace Editor , but documentation for plug-ins and other functionality is missing. Documentation for writing plug-ins is seemingly being worked on right now . However, plug-in authors can only look at the already existing plug-ins and try to figure out what they do and how they do it so far. There are many undocumented assumptions about properties of plug-ins that have to be fulfilled in order for the plug-in to work correctly. Moreover, the IDE is constantly being developed and evolving, which is good. Unfortunately even core APIs, like opening a file, seem to be changed. This results in big compatibility issues with newer and older Cloud9 versions. It is also discouraging, that third-party plug-ins can not be used in the hosted Cloud9 edition. However the developers seem to be eager to include plug-ins in the official Cloud9 repository, which is very pleasant. Especially the missing documentation hinders the development of plug-ins for the Cloud9 IDE, so we hope that the maintainers will provide appropriate documentation soon. We hope that this will result in more plug-ins being written for Cloud9 extending its functionality even further.
Testing your application, preferably with TDD, can increase the quality of the code basis tremendously. Thus, development environments that encourage TDD and support the programmer in the steps through the testing cycle can be of great benefit. In this paper we showed how insufficiently a programmer is supported in the testing cycle when using CoffeeScript and the Jasmine testing framework in the Cloud9 IDE. However, when providing proper tool support, the developer can intuitively work on his application and is steadily supported in the testing process. Since there are already good responses to the tools , , , , we believe the tools will soon be part of the default Cloud9 experience in the near future. With the tools presented in this paper and the steady development the Cloud9 IDE is going through, we suggest this IDE has a great chance to serve as a replacement for some of today’s desktop IDEs in the near future. Developers should begin to think about a shift from classic desktop development environments to web-based IDEs and exploit the benefits of real-time collaboration and global team based development.
Beck, Kent. Test-Driven Development By Example. Addison-Wesley, 2003.
Langr, Jeff. Test-Driven Development: A Guide for Non-Programmers. Pragmatic Bookshelf 2011. http://pragprog.com/magazines/2011-11/testdriven-development.
CoffeeScript Wiki. Text editor plugins. https://github.com/jashkenas/coffee-script/wiki/Text-editor-plugins
Cloud9 IDE Support. Create a CoffeeScript / Node.js Project. http://cloud9ide.zendesk.com/entries/20559696-create-a-coffeescript-node-js-project.
Piper, Tane. Live CoffeeScript: An extension for Cloud9 IDE for CoffeeScript functionality. https://github.com/tanepiper/cloud9-livecoffee-ext
Ficarra, Michael. CoffeeScript II. https://github.com/michaelficarra/CoffeeScriptRedux.
Howell, Steve. CS/JS Code Browser. https://github.com/showell/CoffeeScriptLineMatcher.
Cloud9 IDE Googlegroup. Cloud9 IDE Plugin Documentation. https://groups.google.com/forum/?fromgroups#!topic/cloud9-ide/-GKEtn2lM_8
Metzke, Tobias. Commit in Jasmine plug-in for the Cloud9 IDE. Commit comment by Ruben Daniels. https://github.com/PragTob/cloud9-jasmine-ext/commit/d74610498f2a89fb7930c93328fe2db496e525dd.
Jongboom, Jan. Tweet about Jasmine plug-in for the Cloud9 IDE. https://twitter.com/drbernhard/status/228856743835361282.
Pardee, Matt. Tweet about Jasmine plug-in for the Cloud9 IDE. https://twitter.com/matt_pardee/status/228869510491414529.
A. Console Output – Error Log
1TDD is based on the thought that the tests should be written first, e.g. before the implementation. This way every line of code written in a TDD project should be tested.
2Issue at the CoffeeScript repository discussing the need for line mappings https://github.com/jashkenas/coffee-script/issues/558
3this article contains information such as: title, subtitle, category, first paragraph of the content and further more
4These articles are provided by the Plista team in a training step.
5Jasmine offers the possibility to run with different reporters, to which the results are reported in detail.