Research in the WAM team aims at making it easier to use and develop rich multimedia contents and applications on the web.
There are already a number of specialized web sites for sharing pictures; there are other sites for video clips, still others for music, and so on. But all these modalities stand for themselves, independently from each other. As opposed to this approach, our vision of the multimedia web emphasizes the tight integration of multiple modalities in a single, consistent set of resources, that takes advantage of the many possible interactions between different types of content. Available on the web, these multimedia resources are distributed, linked together, and use platform-neutral formats, that make them usable by anyone through any kind of terminal or network. In this vision, the web is not restricted to a publishing medium for single-media content, but it becomes an open platform for producing, sharing, transforming, using and reusing multimedia contents. In other words, the web is not only a giant repository, but it is first and foremost an environment for processing multimedia documents and data through many different kinds of applications.
To realize this vision, a number of challenges have to be faced. One of the key issues is the formats used for processing and sharing multimedia contents on the web. Formats must represent the many facets of multimedia contents. Depending on their richness and versatility, different options are made available for processing multimedia content and to use it in various contexts. A research theme in WAM is dedicated to models and languages for representing and describing rich multimedia web documents.
XML is the ground on which these formats are built. Designed for the web, XML offers many features for taking advantage of the specificities of the web. The language is flexible and open enough for representing a wide variety of contents, and it comes with many accompanying languages that address issues such as defining schemas, mixing XML languages (document formats), linking resources, querying and transforming data and documents, formatting and presenting documents, etc. As data and document formats for the web are based on XML, it is crucial to better understand how XML structures can be processed, and what are the theoretical tools that may help to develop an effective framework for processing XML structures. This is another research theme in WAM.
Once rich formats are available, documents and data encoded in these formats have to be created. Given the richness of the formats, this is a challenge. Authors need help from specialized tools to benefit from the many, complex possibilities offered by web multimedia formats. Methods used for static, textual documents, such as WYSIWYG or direct manipulation, do not work for dynamic multimedia web documents. New approaches have to be developed and experimented. This is the third research theme in WAM.
Given the prominent role of XML for representing all kinds of data on the web and elsewhere, XML structures processing becomes a key issue. There are already dedicated languages for processing XML structures, such as XSLT or XQuery, which abstract over data through a tree-based data model and provide a powerful execution model. Our research follows this approach.
Some properties are expected from these specialized languages in order to help solving the most common problems: expressiveness, verifiability, efficiency, reusability, evolvability, scalability, correctness, etc. These properties are studied using the fundamental connection between language theory, mathematical logic, structured languages and query languages.
The goal of the research published so far in the literature is often limited to establishing new theoretical properties and complexity bounds. Our research differs in that, in addition to these goals, we seek resolution algorithms, efficient implementation techniques, and concrete design that may be directly applied to XML systems. We also consider that some properties are of particular importance for XML structure processing, namely:
Type checking: The types we consider are structural constraints over documents and data expressed in formalisms such as DTD, XML Schema, or Relax-NG. Few techniques are able to exploit typing information of the input or output documents to provide type-safe processing. In this domain, algorithmic advances have led to the creation of research languages, such as XDuce, based on efficient containment of regular tree types. However, many challenges remain. While type-checking full XSLT or XQuery is theoretically impossible (these are Turing-complete languages), one challenge is to push the “decidability envelope” further for type-checking standard XML transformations. In particular, one of the most difficult issue is to find techniques for analyzing XPath queries with regular tree types. Another challenge is to provide effective algorithms usable in practice for realistic scenarios.
Efficiency: XML processing languages may benefit from static analysis whenever performance is a concern. Static analysis techniques usually take advantage of robust formal semantics to help development of optimized compilers and runtimes.
Most of our work so far focuses on the XPath query language, for which we try to check properties statically, in the presence of types (schemas) or not.
We have participated in the international endeavor for defining a standard multimedia document format for the web that accommodates the constraints of different types of terminals. SMIL is the main outcome of this work. It focuses on a modular and scalable format that combines efficiently the different dimensions of a multimedia web document: synchronization, layout and linking. Our current work on multimedia formats follows the same trends.
Specific formats for audio are an important work item in the team. More specifically, we are participating in IASIG (Interactive Audio Special Interest Group), an international initiative for creating a new format for interactive audio called iXMF (Interactive eXtensible Music Format). We have defined an XML version of iXMF (without scripting, but with integrated 3D audio rendering) and we have implemented it in an audio engine adapted to embedded systems, based on OpenSL/ES (Open Sound Library for Embedded Systems).
Regarding discrete media in multimedia documents, popular document languages such as XHTML can represent a very broad range of documents, because they offer very general elements that can be used in many different situations. This advantage comes at the price of a very low level of representation, often considered as presentational. The concepts of microformats and semantic XHTML were developed to tackle this weakness. More recently, RDFa was introduced with the same goal. These formats add semantics to web pages while taking advantage of the existing XHTML infrastructure. This approach enables new applications that can be deployed smoothly on the web, but authors of web pages have very little help for creating and encoding this kind of semantic markup. A language that addresses these issues is developed and implemented in WAM. Called XTiger, its role is to specify semantically rich XML languages in terms of other, less expressive XML languages, such as XHTML.
Another multimedia authoring model, called the LimSee3 model, was developed in WAM, putting the emphasis on continuous media, such as video and sound, and on synchronization and layout issues. It is a component-oriented document model integrating homogeneously logical, spatial, and time structures in a language-neutral way. It is a generic document model from which more specialized models (called templates) can be derived for different classes of applications. This also makes it possible to create dedicated authoring tools from a generic platform implementing the generic model.
Whereas document formats represent a multimedia document with all its internal structures, description languages describe a document from outside and provide metadata. In the area of description languages for multimedia documents, significant standardization efforts have been spent, such as MPEG-7 for instance, but the problem is not solved yet. Many application domains cannot cope with the description languages available today. We have worked on this issue in cooperation with INA, the French national archive of broadcast radio and television. We have defined a structure description language for audio-visual documents, focusing on formal consistency to make descriptions usable in very large bases, such as those of INA. Typical applications of this work are the producion of a thematic audio-visual offer from archives, or the product ion of the same interactive application on various media (CD-ROM, DVD, web).
Multimedia documents are considered through several kinds of structures: layout, time, linking, logical organization. We are working on techniques that allow users to manipulate all these structures in homogeneous environments. The main objective is to support new advances in document formats without making the authoring task more complex. The key idea is to present simultaneously several views of the document, each view showing a particular structure, and to allow users to manipulate each view directly and efficiently. As the various structures of a document are not independent from each other, views are “synchronized” to reflect in all views the consequences of every change. The XML markup, although it can be accessed at any time, is handled by the tools, and authors do not have to worry about syntactical issues.
Even with tools providing views for manipulation of various structures, the authoring task is often considered too complex for most users because it requires a deep understanding of the semantics of the document format (e.g. the SMIL timing model, or the most advanced features of XHTML). We are therefore working on a new authoring model for multimedia documents that would provide a strong basis for creating generic or dedicated authoring tools with appropriate user-friendly GUI.
Our approach is first to focus on the logical structure of the document while keeping some semantics of proven technologies such as SMIL. The second core idea is to tightly integrate template definitions in this document model: the template is itself a document constrained by a schema-like syntax. The continuum between templates and document instances permits to edit templates generically, like any other document and within the same environment. It also enables a more natural authoring process where documents can be created step by step from existing templates up to a final state where all place-holders are filled and all options are decided: during this process, the document status evolves from a pure template to a completed instance.
Broadly speaking, the main application domain of our research is the web and its numerous applications. This includes the recent evolutions of the web, with a special attention paid to the mobile web, the multimedia web, and the web of applications. The goal of our research is to enable new multimedia applications that can be deployed easily on the web, taking advantage of the existing infrastructure and the basic web technology.
The web is used for a very wide variety of applications, but the applications we have developed recently are more specifically targeted towards educational needs. The software we make available to the community helps teachers and students to benefit from the most recent advances in web technology. In the european project Palette (see section ), for instance, we have created such applications based on both the LimSee3 platform and the Amaya web editor.
For our work on interactive audio, the application domains we address are video games and audio guides. Regarding video games, we focus on ``serious games'' which may have an educational and social impact when they are used in the area of culture or health. The goal of audio guides is to help people to move around in a city, with the goal of bringing autonomy to people with disabilities, for instance to people who are blind.
Work on XML processing is related to one of the foundations of web architecture, resource representation. As such, it applies to a large part of web technology, be it used on the web or in other settings. At the moment, it has strong connections with researches in other area of computer science: data bases and programming languages, where XML structures play an increasingly important role.
Amaya is an open source web editor, i.e. a tool used to create and update documents directly on the web. Browsing features are seamlessly integrated with editing features in a uniform environment that allows users to save files locally and on remote servers as well. This follows the original vision of the web as a space for collaboration and not just a one-way publishing medium.
Amaya started to showcase web technologies in a fully-featured web client. The main motivation for developing Amaya was originally to provide a framework that can integrate many web technologies during their development, with the goal of demonstrating these technologies in action while taking advantage of their combination in a single, consistent environment.
Amaya implements a number of recent web technologies developed by W3C, such as HTML and the XHTML family, CSS style sheets, generic XML, MathML (for mathematical expressions), and SVG (for vector graphics). It allows all those document formats to be edited simultaneously in compound documents. It also includes a collaborative annotation application based on RDF, XLink, and XPointer.
It is a unique tool for manipulating simultaneously different kinds of contents through a formatted representation of documents, while closely following standard formats. Supporting several languages from the XML family allows Amaya to integrate seamlessly such complementary functionalities as browsing, editing, publishing, and annotating. This enables a more creative type of work on the web, allowing users not only to consume existing information but also to produce new information and to interact with other users. Developed jointly with W3C, the software is distributed world-wide through the W3C servers and their many mirrors. It is also part of several Linux distributions.
Amaya is also used as a platform for experimenting and distributing new editing techniques and document formats developed in WAM. It provides a full implementation of the XTiger language and its constraint-driven editing feature (see section ). It also helps users to create their own document types defined as XTiger templates.
Now that a number of document languages are implemented in the editor, developments focus on usability. The latest extensions are oriented towards robustness, completeness and ease of use. Many contributions are received from external developers and have to be coordinated with the project-team. They concern localization in various languages, including eastern languages, tests and adaptations to various platforms. Other contributions provide improvements and new features.
Three public releases were made in 2009, in January, July, and December.
LimSee3 is a new generation open source multimedia authoring tool developed in the context of the european project Palette (see section ). It aims at flexibility and easiness of use through extensive use of document templates. LimSee3 developments benefit from the project-team experience acquired with LimSee2, an already well-established SMIL editor.
Existing multimedia authoring tools can be classified in two broad categories:
General purpose tools make it possible to edit various kinds of multimedia documents and allow authors to precisely manipulate the underlying structures. Such tools are indeed very powerful, yet they remain exceedingly complex and require special skills and serious training to be used efficiently. LimSee2 fits in this category.
Dedicated tools are tailored for some particular domain or document type. They are easily accessible to casual users, thanks to their simplified approach to document authoring. Their main advantage lies in the simplicity and automation of some treatments, but users often find themselves trapped in a rigid framework that imposes strong limitations. In addition, authors have to use multiple tools to manipulate different types of documents.
Taking a different perspective, LimSee3 brings multimedia authoring at the reach of non-expert users while allowing very different types of documents to be produced. The main idea is to provide template-based authoring tools with rich composition capabilities and smooth adaptability. Based on the semantics of multimedia objects, LimSee3 allows authors to work in their own terms. With the integration of templates, users are guided in the production of sensible multimedia documents. Different templates allow authors to produce different types of documents.
As opposed to LimSee2, which sticks to the SMIL language, LimSee3 is language-independent. It is based on a component-oriented document model integrating homogeneously logical, spatial, and time structures in a language-neutral way. Templates are defined as constraints on these structures. Based on this logical structuring of multimedia documents, LimSee3 can generate different representations of the same document, in different languages or formats.
LimSee3 was developed as an entirely new open source project in the european project Palette (see section ).
The LibA2ML library, downloadable from http://gforge.inria.fr/projects/iaudio, is an API for importing, editing and exporting documents in the A2ML audio format (see section ). LibA2ML is intended to serve as initial support for a future authoring tool, but also to enable other applications (auditory guidance for instance) or plugins, to easily support A2ML documents. Initially developed for the tiny version of the A2ML format (J2ME devices), an update is planned in 2010 for the full version of A2ML (iPhones and game consoles).
The XML Reasoning Solver is a tool for the static analysis of XPath queries and XML schemas, based on the latest theoretical advances. It performs automated verification of properties that are expressed as logical formulas over trees. A logical formula may for instance express structural constraints or navigation properties (like e.g. path existence and node selection) in finite trees.
The tool can solve many fundamental XML problems such as satisfiability of XPath expressions in the presence of XML schemas, containment and equivalence of XPath expressions, and many other problems that can be formulated with XPath expressions and schemas like DTDs, XML Schemas, and Relax-NGs.
The solver is avaliable on-line.
Work on the static analysis of XML programs was continued along two main directions in 2009.
The first direction aims at exploring how our previous results on the static analysis of XPath can be extended to queries containing counting constraints. More precisely, we studied how a logic for finite trees capable of expressing upward and downward recursive navigation, can be equipped with a counting operator along regular path expressions. Queries with counting constraints are commonly used in document transformations or programs in which they select portions of documents subject to transformations. Static analysis of such a combined XPath fragment makes it possible to detect bugs in transformations and to perform many kinds of optimizations of document transformations. Preliminary results in this direction were published in and .
The second direction is related to the problem of XML Schemas evolution. In the web and in modern information systems, XML Schemas constantly evolve to cope with the natural evolution of the entities they describe. Schema evolutions may affect query results and potentially the validity of produced data. A challenge is therefore to assess and accommodate the impact of these changes in evolving XML applications. Such questions arise naturally in XML static analyzers. These analyzers often rely on decision procedures such as inclusion between XML schemas, query containment and satisfiability. However, existing decision procedures cannot be used directly in this context. The reason is that they are unable to distinguish information related to the evolution from information corresponding to bugs. We proposed a predicate language within a logical framework that can be used to make this distinction. Specifically, we proposed a system for monitoring the effect of schema evolutions on the set of admissible documents and on the results of queries. The system is very powerful in analyzing various scenarios where the result of a query may not be anymore what was expected. The system relies on a predicate language which allow a fine-grained analysis for a wide range of forward and backward compatibility issues. Moreover, the system can produce counterexamples and witness documents which are useful for debugging purposes. The current implementation has been tested with realistic use cases, where it allows identifying queries that must be reformulated in order to produce the expected results across successive schema versions. The corresponding results were published in and .
The diversity of presentation contexts for multimedia documents requires document specifications to be adapted. In an earlier work jointly with project-team EXMO, we have proposed a semantic adaptation framework for multimedia documents. This framework captures the semantics of document composition and transforms the relations between multimedia objects according to adaptation constraints. This year, we have shown that relying on document composition alone for adaptation restricts the set of relevant candidate solutions and may even divert the adaptation from authors intent. Hence, we have introduced functional annotations to guide the adaptation process. Theses annotations refine the role of multimedia objects in the document. We have applied this approach to SMIL documents, by embedding functional annotations encoded in RDF. These multimedia documents are then adapted thanks to an interactive adaptation tool .
Work on Amaya has focused this year on three main topics:
Formatting. CSS is the most popular language for defining the graphical aspect of web pages. Its syntax is very simple and it offers a number of properties for controlling every detail of document appearance. But despite this simplicity, its formatting model is very sophisticated. Web designers have learned how to take advantage of it to create complex graphical structures that allow documents to be automatically (re)formatted according to the available screen space. In that regard, key CSS features are absolute positionning and floating boxes. The formatting engine of Amaya was extended this year to support these features, taking into account the frequent changes that occur in an editor when the content and the structure of a document are manipulated simultaneously with its style.
Template editing. The implementation in Amaya of the feature that allows document designers to create and update XTiger templates (see section ) was completed this year. Now, every element and attribute from the XTiger language can be created and modified in a template with Amaya. This is not only useful for editing XHTML documents according to a template, a feature that Amaya already supports, but also for creating the XTiger templates that will be necessary for entering XML data as explained in section .
Protocols. The only way for Amaya to download and upload documents from/to a remote server was up to now through the HTTP protocol. But many web servers use SSL for secure communication and very few servers accept the HTTP PUT operation for publishing. To overcome these difficulties, support for SSL over HTTP was introduced in Amaya (URIs of type https:// can now be accessed) and a WebDAV option was added for publishing (WebDAV is now more popular for publishing on the web than HTTP Put).
All these new developments are included in version 11.3 of Amaya, released in December 2009.
In collaboration with partners of the Palette project (see section ), a study was conducted about integrating multimedia contents in a reification process for sharing teaching practice. The issue is that sharing based on raw recordings of courses or meetings brings a very limited benefit, due to the very nature of video and audio. It is indeed too long to listen to a full recording when only some specific pieces are of interest for a given purpose. To solve this issue, textual annotations are associated to recordings and provide an easy way to navigate contents, thanks to synchronization between audio/video and annotations. With LimSee3, annotations are entered at recording time and/or afterwards, by several users. This makes it easy to prepare the recording of a course before discussing it with colleagues. This also allows participants to add more annotations during discussion, to record their agreements or dissents. This study was published in .
LibA2ML (see section ) provides a strong basis for building editors for virtual interactive audio scenes (games for instance) or ARA scenes (guidance applications for instance). Our main interest is in the authoring of ARA scenes in the perspective of the Minalogic Autonomie project (2010-12) upon the indoor-ourdoor guidance of ill-seeing people. The concept of augmented reality audio (ARA) characterizes techniques where a real sound and voice environment is extended with virtual, geolocalized sound sources. An ARA scene is best experimented through the use of ARA bone conduction headsets.
ARA authoring is a non-static task (mobile mixing), for at least two reasons: (1) the author has to move in the rendering zone to apprehend the audio spatialization and the chronology of the audio events which depend upon the position of the listener, (2) the determination of trajectories which will be applied to the virtual sound sources is best done through a tracking system allowing the author to record his mouves and use them as trajectories.
For this non-static authoring task, we are considering an implementation of the see-through touch-screen interface concept to control the localization of the sound sources. The XML language used for the map on which these positions are recorded is OpenStreetMap (OSM) for outdoor authoring and an extension of OSM for indoor authoring.
The ARA scene will be described through the mixing of two XML languages, i.e. A2ML and OSM. This mixed format will allow a textual authoring of the sequencing of the sound sources and DSP acoustics parameters.
The XTiger language was designed initially to specify document templates that could guide an authoring tool, and then an author, in producing well structured XHTML documents. Several experiments conducted with our partners in the Palette project have shown that this basic feature may be extended. The XTiger language is actually an excellent means to help users to enter very structured documents and data through the familiar metaphor of text editing.
New work has started to use XTiger, with minor extensions, for entering well structured XML data through an XHTML document manipulated in a web browser. The goal is to allow average web users to feed XML data bases and XML applications simply by interacting with a very familiar tool, their usual browser. This work is done in cooperation with the Media group at EPFL.
We have designed a new version of A2ML, our XML format for interactive audio, to follow the trend in powerful mobile audio devices lead by Apple with its iPhone. A2ML, in its 2009 full version, offers, concerning the sequencing of sounds, a level of capabilities similar to that of iXMF, the interactive audio file format defined by the Interactive Audio Special Interest Group (IASIG). But, as opposed to iXMF, A2ML supports not only sequencing but also 3D sound rendering and parameter animation (DSP and positional parameters). SMIL internal events are supported and external events can be freely defined by the sound designer. Scripting, which is difficult to implementand use, has been replaced by a declarative way of handling dynamic soundtrack adaptation to the context. Consequently, A2ML is now in its full version an audio format suitable not only for mobile applications like auditory guidance, soundwalks or soundscape, but also for 3D games on mobile phones and consoles.
A sound manager/engine for the iPhone platform, which should be easily ported to all games consoles by translating objC code into C++ code, has been written and is under testing. It is worth noting that no sound manager has yet been written for the iXMF format, probably because implementing a sound manager for a binary format with scripting inside, is a very difficult task.
The construction of an A2ML sound model for an interactive jungle, usable in a game or in a sound installation, is explained in and downloadable (in MP3) from http://gforge.inria.fr/projects/iaudio.
None currently.
Codex is a project funded by ANR as part of the Emerging Domains program (DEFIS). It started in March 2009 for a duration of 36 months. WAM is working with five partners: INRIA Saclay-île-de-France (project-team GEMO), INRIA Lille-Nord-Europe (project-team MOSTRARE), University Paris-Sud, Centre universitaire de Blois, Innovimax SARL.
Codex seeks to push the frontier of XML technology innovation in three interconnected directions:
Languages and algorithms: prototypes are developed for efficient and expressive XML processing, in particular advancing towards massively distributed XML repositories.
Codex considers models for describing, controlling, and reacting to the dynamic behavior of XML corpora and XML schemas with time.
The project proposes theories, models and prototypes for composing XML programs for richer interactions, and XML schemas into rich, expressive, yet formally grounded type descriptions.
Multimedia Cooperative Publishing Chain (C2M) is a project funded by ANR as part of the Contents and Interaction program. It started in October 2009 for a duration of 24 months. WAM is working with five partners: Université de Technologie de Compiègne, Kelis Conseil et Développement, Amexio, Heudiasyc laboratory (CNRS), Institut National de l'Audiovisuel (INA).
The project aims at integrating XML publishing chains, Enterprise Content Management (ECM), and multimedia creation tools, in order to design a complete digital system for multimedia creation, management and publishing.
The main challenge lies in the convergence of several approaches:
storage and management of document fragments,
structured editing,
maintaining and repurposing content,
planning, cooperation and production.
Convergence is made possible by the maturity of XML technologies and by the collaborative practices popularized by the web.
Palette (Pedagogically sustained Adaptive Learning through the Exploitation of Tacit and Explicit knowledge) is a European IST FP-6 Integrated Project. It aims at developing an extensible set of innovative, interoperable and standard-based services that enhance the learning process in communities of practice. These services are validated through various pedagogical scenarios fostering the emergence of new learning practices that remove barriers for the exploitation of mental models, knowledge resources and competences of individuals inside and outside communities.
The project was terminated end of January 2009. The main contributions of the WAM project-team concern document models and authoring tools. More specifically, templating mechanisms are designed, developed and experimented in the context of communities of practice. These developments and experiments are based both on Amaya (see section ) and LimSee3 (see section ).
The Amaya web editor is developed jointly with W3C. The software is distributed by W3C.
Vincent Quint is a member of the W3C Advisory Committee. Nabil Layaïda is a member of the W3C Synchronized Multimedia working group.
Jacques Lemordant is a member of IASIG (Interactive Audio Special Interest Group) and of the AES (Audio Engineering Society).
Cécile Roisin is a member of the steering committee of the ACM Symposium on Document Engineering.
Vincent Quint is on the steering committee of the H2PTM conference series.
Pierre Genevès gives Master lectures on Core XML technologies and their theoretical foundations at Grenoble Universities (Master of Science in Informatics at Grenoble, final year) and at EPFL (Lausanne).
Nabil Layaïda gives Master lectures on SMIL and Multimedia Principles at Grenoble Universities (Master of Science in Informatics at Grenoble, M2R SIGAL: UE IST, final year).
Jacques Lemordant teaches XML Technologies at the L3 level of MIAGE (UJF, Grenoble), on Multimedia Technologies at the M1 level of RICM (Polytech, Grenoble) and Web Technology at the M2 level of IICAO (UJF, Grenoble).
Members of the WAM project-team were on the following program committees and editorial boards: Document numérique, DocEng 2009 (ACM Symposium on Document Engineering), H2PTM'09 (Hypertextes-Hypermédia), Information - Interaction - Intelligence.
Members of the WAM project-team have served as reviewers for the following venues in 2009: 12th International Symposium on Database Programming Languages (DBPL 2009), Technique et Science Informatiques (TSI).
Members of the WAM project-team served as project reviewer for the Qatar National Research Fund (QNRF), ANR (Contint program). Nabil Layaïda serves as Scientific Advisor for RaisePartner S.A, an INRIA spin-off.