The Open Source DataTurbine Initiative: empowering the scientific community with streaming data middleware

DataTurbine is a robust real-time streaming data engine that lets you quickly stream live data from experiments, labs, web cams and even Java-enabled cell phones. It acts as a “black box” to which applications and devices send and receive data. Think of it as express delivery for your data, be it numbers, video, sound or text. For ecological applications, DataTurbine is useful for moving data in near real time from sensors to field stations to data centers. DataTurbine handles time series data over networks with intermittent connectivity. It is vendor-neutral, so it works with sensors and dataloggers from a variety of manufacturers and research labs.

DataTurbine is a buffered middleware product, not simply a publish/subscribe system. In distributed systems, middleware is the software layer that lies between the operating system and the applications running at individual sites (Middleware). DataTurbine can receive data from various sources (sensors, web cams, etc.) and send data to various sinks (visualization interfaces, analysis tools, databases, etc). It has “TiVO” like functionality that lets applications pause and rewind live streaming data.

DataTurbine is open source and free. There is also an active developer and user community that continues to evolve the software and assist in applications. The development and publication of DataTurbine is currently supported by the Office of Cyberinfrastructure at the National Science Foundation and by the Gordon and Betty Moore Foundation.

Example applications

In this section we describe some of the key applications in which we are involved with the OSDT Initiative. In each application we provide an overview of the research issue and the role of OSDT in the system.

F.1 Ocean Acidification at Coral Reefs. While coral reefs have undergone unprecedented changes in community structure in the past 50 years, they now may be exposed to their gravest threat since the Triassic. This threat is increasing atmospheric CO2, which equilibrates with seawater and causes ocean acidification (OA). Key to understanding this process is measurements by OA-related sensors including pH, PCO2, temperature, and pressure. The deployment of OA-related sensors at MCR and Palmyra Atoll and the collection of measurements from these sensors will provide valuable insights into OA and lay the foundation for larger, more rigorous OA studies in the future. The ability to measure these phenomena in situ and in real time represents a significant advance in our ability to monitor key environmental processes. The OSDT Android Sensor Pod will provide a critical building block for enabling new science experiments on OA, and the results will be applicable to other coral reef researchers.

F.3 Free Ocean Carbon Enrichment Experiment. Monterey Bay Aquarium Research Institute (MBARI: scientists and engineers have designed FOCE (Free Ocean Carbon Enrichment) technology to enable the study of the impact of this pH change on ocean biogeochemistry and ecology. The FOCE technology concept enables small-scale, in situ CO2 enrichment experiments to be carried out, in a manner analogous to the terrestrial Free Air CO2 Enrichment (FACE) experiments (Long et al. 2006). FOCE is used to control the pH within a small volume of seawater that exchanges freely with the surrounding region in the ocean. The technology uses feedback from pH sensors and other instruments to inject CO2, creating ocean acidification conditions (reduced pH) in a small area, corresponding to future levels of greenhouse gases, while maintaining other environmental parameters. In this way, FOCE is like an ocean acidification time machine, allowing researchers to peer into the ocean’s future to see the effect on natural ecosystems such as coral reefs, cold-water corals, and other sensitive benthic habitats. DataTurbine provides asynchronous communication links between distributed components in FOCE setup, and is particularly well suited to streaming instrument data.

Fountain T., Tilak S., Shin P., Nekrasov M., 2012. The Open Source DataTurbine Initiative: empowering the scientific community with streaming data middleware. Bulletin of the Ecological Society of America 93:242–252. Article.

  • Reset


OA-ICC Highlights

%d bloggers like this: