1. About This Case Study
Target Audience
The target audience of this case study is those interested in applications that provide alternative or innovative user experiences using HTML5 Web applications. The focus is on assistive technology which is designed to enable wider access to media, apps and other online technology. This access may be for users who have varying access requirements, such as older users or those with physical or cognitive impairment. Alternatively it may be for use in environments that require alternative interaction styles, for example in bright light or with restricted access to a mobile device.
This case study also provides an insight into the development of applications that can run on a wide range of devices, without the need to cater for a variety of platforms. It also covers the benefits of applications that are composed of reusable user interface (UI) components in the form of W3C Widgets. Such widgets allow rapid development and innovation through recombination in different configurations.
What Is Covered
This case study considers how to create an HTML5 app version of an existing assistive technology program called Maavis – http://maavis.fullmeasure.co.uk [1] Maavis was designed by Peter Cudd at the University of Sheffield School of Health and Related Research (ScHARR) [FN1] which works on enabling people with dementia to engage with ICT. This has since been extended to support other users including older learners with mental health problems and children with multiple disabilities.
This original Maavis application can only be installed and run on machines with Windows, thus limiting its use. Creating a new version using HTML5 and Widgets allows Maavis to run on the many platforms that have a Web browser. This includes mobile and tablet devices that are becoming increasingly popular with users since they greatly increase the number of locations and situations in which Maavis can be employed. In addition, HTML5 skills are widely available which broadens the potential developer bass by lowering the technical entry requirements. Thus it will reach more users and attract more developers, both of whom are required if open innovation in software through open development methods is to succeed. Indeed, there is a great need for such open innovation in the accessibility field in order to provide useful solutions for users’ individual requirements.
Figure 1. Screens displayed to users of the original Maavis system.
HTML5 and other open Web standards such as widgets provide an eco-system that develops rapidly, is adopted widely and supports contributions to the specification. For example, the addition of open video playing and formats has removed several important barriers to media access, including proprietary lock-in to specific tools that only work on some platforms. One instance of such lock-in is Windows Media which cannot be played on non-Windows platforms without an extra (free) component installed (assuming a player is available for the platform). Building the application to comprise several widgets provides reusable UI components which can then be deployed individually or in combination with other applications. Widgets also provide methods of sharing and reusing components, and so encourage good practice through example. Moreover, accessibility is 'baked in' thereby ensuring it is more widely available.
The approach taken was to prototype key elements of the original desktop Maavis application which provides simple access to media and communications. These prototyped elements have been decomposed into potentially useful widgets and hard-coded into a simulated version of Maavis with specific screens. This shows that a HTML5 version of Maavis is feasible and clearly demonstrates how it may be used to create innovative applications.
Figure 2. HTML5 Maavis versions of the four screens depicted in Figure 1.
In particular, the HTML5 audio and video elements are used with custom HTML 'controls' designed to be part of a standard interaction scheme that provides user selection of various activities or resources. The local storage facilities of HTML5 are also investigated as a means of saving both state and settings.
Note that the original Maavis application was developed in Mozilla XUL (XML User Interface Language) [FN2] which is very similar to HTML, Javascript and CSS, but supports full platform access. XUL was chosen for initial development since, although a Web solution was favoured, requirements such as off-line use, and the state of HTML technology meant it could not be used. Since Maavis was developed, touch-screen and mobile-Web technologies have significantly matured, while many of the problems originally solved in Maavis now have other implementations, such as using HTML5 storage and responsive layouts.
What Is Not Covered
This case study does not explore general accessibility of HTML apps or how to use HTML5 features to improve it. Web accessibility is well understood (although not always well implemented) and HTML5 adds several new facilities that aid and enhance general accessibility. It is naturally planned that a HTML Maavis will include accessibility best practice as described by the Rave in Context widget project's WidgetAccessibilityBestPractices [2] in order to ensure maximum utility for as many people as possible. Note that Rave in Context has also defined a widget template system that has now moved into the Apache Wookie (Incubating) Project.
Also excluded is coverage of any issues that arise in creating HTML applications that are usable and accessible on a wide range of devices and platforms. For example some platforms have gaps in the full accessibility stack support. This is a key topic for further phases of work and OpenDirective are actively supporting the open development of widely accessible cross platform HTML apps and widgets, e.g Apache Wookie (Incubating), Apache Rave (Incubating) and Rave in Context.
No attempt was made to reproduce several original Maavis features. For example, running local applications, including Skype integration for video calls, is not possible as browser security forbids running programs (unless you want to risk using ActiveX in IE on Windows only). The integrated scanning for use with accessibility switch control is also excluded. These features require facilities not currently available in the sandboxed browser environment in which HTML apps run. It is interesting to note that Mozilla is exploring joystick support which will allow alternative device input such as switches.
2. Use Case
Providing simplified access to media and information offers far greater ease of access to people with cognitive or physical impairments. Such support is also greatly valued when users need to access technology under difficult conditions such as a poorly lit environment, or when operating a device with limited methods of interaction.
The academic use case is to facilitate access to online course and/or research materials by more people, in a wider variety of situations and in novel ways. If, for example, virtual learning environments (VLEs) adopted this approach, then students with various impairments will have an enriched learning experience through being able to access content in ways appropriate to their specific preferences and requirements.
An HTML application (or Web app) that provides simplified access will be available to users on more platforms, including tablets and smartphones, as long as a suitable browser exists. Development and maintenance will be simplified as a single platform is targeted (at least in theory). However HTML runs in a restricted browser environment, which for security reasons has limited access to the host platform facilities. Thus there are questions about whether such a tool can be created with all the functionality that requires local access. However, on the other hand it is evident that more content, media and interaction now happens online or in the cloud, which means that using an HTML5 tool to mediate access looks increasingly feasible, even beneficial.
Maavis provides access features through a simple touch, pointer or accessibility switch access interface using groups of buttons and media 'players' arranged into screens. Viewers are provided to access music, video, photos and Web information (Web sites). Maavis is designed for use by people with cognitive or physical impairments, which includes older users. In fact, Maavis was originally designed for people with dementia in a managed care environment. Thus simplicity of interaction is of paramount importance. Subsequent development phases added features for adults with learning difficulties and children with physical impairments, both in educational settings.
A key feature of Maavis which makes it interesting here is that a facilitator can easily collect related media and provide them for access by the end-users. This makes it a suitable tool for providing access to a cohesive collection of content. For example, Maavis has been used to provide interactive Tai Chi lessons.
By re-implementing Maavis in HTML5 we will make it available to a wider audience. For example, an elderly woman with limited typing ability due to arthritis of the hands wishes to stay in communication with her family. She is very resistant to using a generic desktop or laptop computer, but is happy with touch-enabled tablet devices.
3. Solution
We have developed a prototype [FN3] that implements key features of the original XUL Maavis in HTML5 and also uses W3C widgets. The current Maavis product was analysed for decomposition into suitable widgets and a simple, functional, but hard-wired HTML5 prototype was created to demonstrate how this might be implemented. This was not intended to be a full replacement, but rather a code 'spike' to explore the issues. The protoype may be found at http://maavis.opendirective.net [3].
This prototype then generated discussion and ideas on further development within the Maavis project team. Such discussions can be found on the mailing list archives which are accessible via the Maavis Web site [FN4].
Key functional areas of Maavis performance were targeted and provided with an HTML5 implementation, which proved they could be made to work. Those functional areas were:
- Screens of buttons containing text and images and arranged in grids
- Screens with only buttons that provide navigation and selection
- 'Player' screens containing a viewer (or audio player) plus control button sets, specifically for audio, videos, still image slideshows and information (browser).
- A simple speech screen that speaks words when buttons are pressed
- A text-to-speech processor to read aloud screen titles and other information when requested
- Keyboard access (tab and enter keys to move around and activate screen options)
Widgets
According to the W3C Widget specification [FN5]:
Widgets are full-fledged client-side applications that are authored using Web standards such as [HTML] and packaged for distribution. They are typically downloaded and installed on a client machine or device where they run as stand-alone applications, but they can also be embedded into Web pages and run in a Web browser. Examples range from simple clocks, stock tickers, news casters, games and weather forecasters, to complex applications that pull data from multiple sources to be 'mashed-up' and presented to a user in some interesting and useful way.
For distribution, the Web contents of a widget are bundled up into an archive (actually a ZIP archive). When a widget is required, a request is made of a widget server such as Apache Wookie (incubating), The server extracts the files and makes them available to the browser which then displays them to the user.
In addition to running in a browser or as a stand-alone application, Widgets can also be used to provide reusable UI 'components' to be shared between applications. This idea is explored in the prototype by choosing a wide range of widget decompositions.
At the time of writing, the widgets used in the prototype are not packaged, but exist as discrete files included in iframes that represent the widgets. This approach allows rapid development by avoiding widget packaging and serving. However, as a result, it is also deprived of some of the advantages of widgets and fails to demonstrate their utility to the full. It is expected that further development based on this protoype will soon create useful widgets which will be made available as open source components, perhaps as part of the Wookie Project.
The following images demonstrate some of the widgets selected in the design.
Figure 3. Collection browser and control widgets.
Figure 4. Image viewer and player controls widgets.
Figure 5. Playlist and player controls widget.
Audio and Video Players
In the original Maavis application, audio and video players were created using a browser media player plug-in (VLC) while the XBL (XML Binding Language) standard was employed to make components out of HTML, CSS and script. Unfortunately XBL is not widely supported and the plug-in, while open source, is not available for all platforms. Until HTML5 provided theandtags the only other solution was Flash, which is not available on some platforms[6]. The prototype audio and video players build on the HTML5 elements by providing Maavis style controls that script their behaviour.
Text to Speech
An important feature of Maavis is the ability to read out screen titles and button labels using synthetic speech. Such text-to-speech support is usually provided by external assistive technology such as screen readers, but in this case it needs to be part of Maavis. As will be indicated in the section on Challenges below, there is no standard support for text-to-speech processing, so in the protoype we simply play pre-recorded audio using HTML5 audio. Maavis users have also requested a feature to record and play back audio in addition to automatic text-to-speech processing, so part of this has been proven to work.
Local and Session Storage
Maavis must store a range of settings and values during its operation. For example, facilitators select the available media and configure options for the end-users such as colours used or if text is read aloud. Previously this data would have to be stored using cookies, or on a server, as the HTML model does not support storage between pages. HTML5 adds a number of new options for storage of name-value pairs that persist for as long as the browser is open, or longer. For the prototype we used local storage to keep values required by the running code. For example, a simple button colour theme selection feature has been implemented and the audio player uses session storage to keep track of the play/pause state and track.
Future work will be to explore greater use of local and session storage, now that the developer has discovered how to work around for a bug in Firefox 7.0 which stopped local storage working when accessing local files as opposed to those served via a Web server.
If the Wookie Widget server is used, then Widget preferences provide another storage solution. The advantage here is that state would be saved across different devices (at least if served by Wookie). A mixed model could work which falls back on local storage when not hosted by Wookie. This is an area for further development.
4. Impact
This prototype demonstrates that a usable HTML5 version of Maavis can indeed be created. While there are problems still to be resolved, the rapid advancement of HTML, largely fuelled by the growing interest in HTML apps on both desktop and mobile platforms, means it is likely such problems will soon be resolved. Indeed Windows 8 will use HTML for applications [FN7] and GNOME has been looking at Web and desktop integration for some time [FN8].
The HTML5 version of Maavis has the potential to provide far wider and easier deployment options than the previous Windows-only version. Complexity of installation has been reduced, for example the need to install the VLC media player has been replaced by the HTML5 video and audio tags. As standards adoption progresses further alongside new technologies, we hope also to remove dependency on items such as Skype for video conferencing.
When complete, the new simple-access Widgets will provide useful and accessible components that, in the previous version, were locked into the Maavis application. Of particular interest is the potential to embed the widgets, using the Wookie Project, in other applications such as VLEs (Moodle currently supported), CMSs (Drupal and WordPress currently supported), social tools (Elgg [FN9] currently supported) and mobile campus information systems (such as Molly [FN10] a nd MyMobileBristol [FN11]). The team are also considering enhancements to the In-Folio e-Portfolio system from JISC TechDis [FN12] which focuses on supporting users with physical or learning disabilities.
The reuse of code in this way has an impact on both the sustainability of Maavis and of those projects reusing our code. Project sustainability can be increased through having components that are of value to a wider range of projects and users. By building these widgets in a project with a focus on accessibility, we not only create reusable components but we ensure that practitioners reusing our code are, in turn, producing material more accessible to their users. This is particularly important as accessibility is often regarded as a niche market and is routinely assigned a low priority in application design.
Furthermore, by adopting a standards-based implementation we ensure that Maavis will be able to reuse other HTML5 applications as they become available. For example, one of our team recently built an HTML5 widget that will display Open Document Format documents. This removes the need to install an office suite to read such documents. This widget can be incorporated into the HTML5 version of Maavis very quickly and easily. Similarly, using features provided by the Wookie Project it will also be easier to add community-sharing features that are planned for Maavis, such as providing shared family photo albums.
As a result of this work, not only can the existing audiences be reached in more flexible ways, but new business models based on Maavis as an online service become possible. HTML5 will allow Maavis to reach more users and developers, both of whom are necessary in order to make a success of open innovation in software through open development. This is an area that is being actively explored by the Maavis team with OpenDirective [FN13].
5. Challenges
Browser Support for HTML5
Perhaps the biggest challenge at present is the variation in browser support for HTML5 and related technologies. In fact, as HTML5 will never be a fully completed specification, but is, rather, a continuous evolution, this may never be completely resolved. Providing good fall-back behaviour for browsers with less than optimal support is complex and expensive work. Consequently, best-of-breed HTML5 "boilerplate" solutions, with fall-back operations, are being evaluated. Frameworks such has jQuery [FN14] and jQuery Mobile [FN15] which provide rich and fully tested UI components. Javascript and CSS libraries will also be used.
Supporting variations in target browser and platform support is being targeted through progressive enhancement and responsive design techniques which aim to ensure a good user experience on widely varying equipment. This is an area that is rapidly evolving and requires testing on a wide range of devices, so wide that it will be difficult to test everything. The Rave in Context Project [FN16], which began its development phase in September 2011 will further test and enhance the widgets developed in Maavis. Whilst this case study will not be able to report on this aspect of the project, we will be able to report on recommendations to the Rave in Context Project.
There are many ways to decompose an application into reusable widgets and we have explored several in the prototype. There may be other, more appropriate decompositions, depending on reuse cases that can be determined while others are likely to be tried as a result of feedback and experience gained from using them.
Security Issue with Cross-domain Access
In the prototype, widgets are represented by iframe elements which raised a number of issues.
The browser security concept of same origin policy restricts access to DOM from script on pages in a different domain. When using several widgets, the impact is that code in one widget cannot manipulate another. This means widgets loaded from different sources cannot easily communicate with each other on the client side. However, this is not a problem in the case study as all widgets load from the same origin. The traditional "hack" has been to use window.document.location.hash which can be accessed freely to pass data. Fortunately, modern browsers now support postMessage which provides the loosely coupled message-based connectivity best suited to this task. Libraries such as jQuery PostMessage Plugin [4] also exist to provide fall back for older browsers. The Wookie server is also likely to provide a way to share values between widgets it hosts.
User Control of Web Page Display
The information viewer raises a difficulty that does not occur in the original Maavis application which shows the Web page in an embedded browser element that is only available in XUL. The control buttons are scripted to control this browser element for pan and zoom. In the HTML pro type the same origin policy means that the control script cannot manipulate the displayed Web page. So as a work-around, the Web page is shown in an additional iframe inside the widget iframe in an attempt to provide scrolling by moving it. This does not work very well, neither does it provide zooming; so an alternative solution will be needed, such as a Web general-purpose browser widget.
Support for Accessibility Switch Users
The original XUL Maavis provides built-in scanning support to allow control with simple switch devices (briefly a highlight moves between user interface elements until a switch is operated by the user to select or activate that item). Currently browser standards support very limited input devices, namely pointer + click and keyboard, and events (pointer and keyboard), so switches cannot easily be used without extra assistive technology software. A new W3C events working group [FN17] has started to look at new browser inputs such as swipe gestures. Furthermore, Mozilla is exploring joystick support in Firefox, and if this becomes a standard, it will allow switch devices to be used, as they usually appear as joystick devices (actually USB HID (Human Interface Device)). For the prototype, a work-around was employed using tab (move) and enter key (select/action). This allows direct keyboard control and also means assistive technology programs for switch users can also interface successfully.
Keyboard Access Issues
Another observation is when using tab key access to move between elements iframes are tab stops themselves, and so receive focus before subsequent navigation into the contained widget elements. If the iframe elements are removed from the tab order using the tabindex='-1' property, none of the child elements are accessible. This is no doubt by design, and it is possible that the widget container should be a tab stop; for example it may have user-alterable properties. The Rave in Context Project group is looking at in-lining widgets without the iframe container, and that may provide a solution. In fact, the whole area of accessible manipulation of widget properties and size/location may need research before this becomes clear.
Embedded Text-to-speech Support
Synthentic text-to-speech processing is problematic as there is no standard platform-independent way of accessing speech from a browser in order to create self-voicing HTML applications. In fact, there is no platform standard - at all. A common approach is to use a Web service, pass it text and receive audio which is then played. This has some advantage by being server-side, but raises questions of scalability. One such service is the unofficial Google translate feature. The prototype avoids the issue by simply scripting an audio object to play a pre-existing audio file. Mozilla has been working on some very powerful audio APIs, and if this becomes standard, then it may be possible to have HTML Text to Speech (although it will be processor-intensive).
6. Lessons Learnt
HTML5 is a rapidly evolving set of standards. This project has sought to push the boundaries of those standards, for example, the original Maavis project embedded Skype video-calling features, something that cannot be done using pure HTML5. Fortunately the need for video conference features is not critical to the success of the early phases of the project and work by Google and Mozilla on the WebRTC standard looks like it will address this. However, it would have been a good idea to spend additional time understanding where these limitations lie, so that they could be planned for and circumvented in a controlled manner during development.
It is possible to bypass such limitations by using another standard, W3C Widgets, which allows additional features to be specified as required or optional for operation. At first this may seem like a good solution, however, this re-introduces the problem of a client needing installation expertise because each non-standard feature will require configuration. Therefore this approach limits the environment in which these widgets can be used. The HTML5 version of Maavis will continue to operate using only the widgets that are fully supported by the platform. Therefore one needs to be careful during the design of widgets to minimise the use of additional features wherever possible.
Many of these difficulties could have been avoided by seeking to implement on a single platform with the stated goal of supporting all of the standards in use. This is an important lesson we learned in the original 'browser wars'. That is, by designing for the platform that most closely tracks the standards we care about, we are reducing the demands on our immediate development effort whilst ensuring that we maximise the chances of our code being usable in future versions of competing browsers.
In this prototype, events handled on the buttons in one widget directly manipulate the DOM in others. This is fine in a prototype, but real widgets will not be able to do this due to potentially different origins. In addition, this tightly coupled design is not flexible, and some form of API or message exchange will be required in the final widgets. This could be done client-side, and most usefully would require new standards. There is some early development in this area, in particular that of looking at WebActions and WebIntents, which allow handlers to be registered to be invoked on specific user actions. Alternatively, interactions could be managed via a server-based service that widgets share. For example, Wookie provides a number of useful services such as OpenSocial and Wave. For HTML Maavis work on this will go hand in hand with the selection of a suitable widget decomposition to produce a suitably course-grained structure with widgets that are useful in their own right, but which can interact with others.
Access of local files (file://) in the browser rather than via a Web server (http://) can lead to issues due to differing behaviour. File access was initially used to ease development (i.e., no need to upload files to a server). File access also allows stand-alone operation without Internet access, something required in several scenarios of Maavis. In this case, we encountered a bug in Mozilla Firefox, the browser being used for development. The specific bug [FN18] has now been fixed. A simple solution in cases like this is to use a local Web server like XAMPP [FN19], both during development and deployment with only local access. Another problem we found is case-sensitivity of elements on differing file systems. Access using XAMPP or file:// on Windows ignores the case of folders and files, whereas access through an Apache Web server running on a Linux server does not. Therefore we found when files are accessed on a Web server, they may not be found due to case mismatch.
7. Conclusions
It is clear that HTML5 has reached a state of maturity that provides an environment where much of the Maavis key functionality can be implemented as an HTML app. Further, the growing availability of standards-conformant browsers on many platforms means the HTML5 Maavis can be deployed on many devices, and so provides flexibility and reach to more users.
The closeness of XUL to HTML meant that much of existing functionality was reproducible in HTML, at least in theory. Exceptions have already been noted. While HTML running in the browser has limited access to the platform for security reasons, new APIs with platform access are becoming available to Javascript in the browser, especially on mobile platforms - for example, geo-location. In addition, PhoneGap [FN20], a tool that allows HTML-developed apps to be compiled into native apps, provides various APIs and has a stated goal of becoming redundant when those APIs are provided by browsers.
This all means HTML Maavis will soon have access to new HTML and platform features, thereby providing the opportunity for innovative alternative access to media and communications.
Moreover, designing the application to comprise a number of reusable W3C widgets means those widgets can be used in other scenarios and applications. Furthermore, Maavis can take advantage of third-party widgets. Using the Wookie Widget server provides a number of useful extra features, especially the storage and sharing of values between widgets.
In fact, this ability to share widgets opens doors to collaboration between projects and companies, neither of which need necessarily be working in the same market or research space. This then fosters innovation through open development of software and will benefit the wider development community. This is a key focus of OpenDirective's work on important projects such as Apache Wookie (Incubating), Apache Rave (Incubating) and Rave in Context. These projects clearly demonstrate the advantages of factoring Web-based applications into functional UI units.