• Home
• News & Events
• Workshops and Conferences
• Research & Education
• Documents
• Research Data & Codes
• Suggested Links

SDM-2010: Abstracts for Scientific Data Management and Analysis at Extreme Scale 2010 FSIO projects

• Damasc: Adding Data Management Services to Parallel File Systems
  Scott Brandt, University of California, Santa Cruz
  Maya Gokhale, Lawrence Livermore National Lab
• HECFSIO Topic: Next Generation I/O Architectures
  Scientific applications need high-level data management services. Scientists use file systems to store and manipulate large volumes of data, yet unstructured file systems provide poor support for highly structured scientific data and scientific codes and data often outlast the systems upon which they are processed and stored. To facilitate software development and minimize platform-specific impact on scientific codes, the high-end computing community has adopted multiple layers of abstractions and specialized file formats such as parallel NetCDF, NetCDF-4, HDF5, and MPI-IO. These APIs are limited in terms of the performance and functionality they can provide. Associated libraries providing access to the highly structured contents of scientific data files stored in the (unstructured) file systems can only optimize to the extent that the file system interfaces permit. As intermediate layers such as MPI-IO evolve, high-level layers such as HDF5 do not always keep up. Motivated by the need to manage and analyze large-scale scientific data effectively, we are developing Damasc, an enhanced file system with rich data management services for scientific computing provided as a native part of the file system. Damasc will make it easy to pose queries and updates over files stored in their native byte-stream format, without losing the inherent performance benefits of file system data storage by allowing scientists to write declarative queries and updates over views of underlying files. Views provide a flexible mechanism on how the underlying files should be interpreted and may remain virtual, with little or no overhead. To achieve this goal, we are adding a configurable thin layer to the file system that exposes the contents of files in a logical data model through which views can be defined and used for queries and updates. Queries and updates so posed can be optimized to efficient execution plans evaluated inside the file system. This enables a declarative querying interface that facilitates data analysis and the development of applications over efficiently stored files. The logic of data management is offloaded to the distributed file system and the application can focus on manipulating useful information extracted via appropriate queries and updates. We are also developing two important services on top of our enhanced file system: self-organizable storage and provenance capture. The Damasc design includes six components: 1) A thin layer of software that can process les of different formats to extract structured views in our logical data model and translate accesses to the underlying native byte-stream le system format; 2) A declarative query and update language for accessing les via the structured interface; 3) An optimization module that does cost-based rewriting to efciently evaluate declarative queries; 4) A self-organizing storage service that automatically indexes les based on query patterns to improve performance; 5) A provenance capture service that captures critical provenance information for subsequent understanding and analysis; and 6) An integration of the extended data management services into a petabyte-scale parallel le system without sacricing scalability and performance. We will prototype these components inside the Ceph file system. Damasc has four key benefits for the development of data-intensive scientific code: 1) applications can employ high-level services, such as declarative queries, views, and provenance tracking, that are currently available only within a database system; 2) the use of the new services becomes easier, as they are provided within a familiar file-based ecosystem; 3) common optimizations, such as indexing and caching, are readily supported across several file formats, thus avoiding effort duplication; and, 4) the potential for significant performance benefits as data processing is integrated more tightly with data storage.
• 
  
• An Information-Theoretic Framework for Enabling Extreme-Scale Science Discovery
  Han-Wei Shen, Ohio State University
  Rob Ross, Argonne National Lab
  Yi-Jen Chiang, Polytechnic Institute of New York University;
• HECFSIO Topic: Measurement and Understanding
  Keywords: Visualization, Information Entropy, Data Saliency
  As scientists eagerly anticipate the benefits of extremescale computing, roadblocks to science discovery at scale threaten to impede their progress. The disparity between computing and storing information, and the gap between stored information and the understanding derived from it, are two of the main barriers to success. This project addresses two difficulties faced by computational scientists. The first is deciding what data are the most essential for analysis, given that only a small fraction can be retained. The second is transforming these data into visual representations that rapidly convey the most insight to the viewer. We will quantify the amount of information in data using information-theoretic approaches. Computing the information entropy of data allows decisions to be made as to how data should be stored and subsequently analyzed. Data saliency will further be used to inform and steer visualization algorithms automatically, including temporal analyses, and it can enable new types of analyses to be performed. We will construct a data analysis and visualization framework based on information theory that allows us to evaluate the information content of simulation output, and we will test our approaches in applications that represent the next generation of extreme-scale science. We will work together with scientists to evaluate the results of our information-theoretic algorithms. With these tools, scientists will be able to preserve important and discard irrelevant data, enabling them to see results sooner. Informed visualization algorithms will generate more meaningful displays. This will result in more knowledge, faster, and will impact decisions critical to the mission of the Department of Energy.
• 
  
• Dynamic Non-Hierarchical File Systems for Exascale Storage
  Darrell Long, University of California, Santa Cruz
• HECFSIO Topic: Next Generation I/O Architectures
  Keywords: Provenance, Dynamic Non-Hierarchical File System, Scalable Indexing, Metadata Clustering
  Modern high-end computing (HEC) systems must manage petabytes of data stored in billions of files, yet current techniques for naming and managing files were developed 40 years ago for collections of thousands of files. HEC users are therefore forced to adapt their usage to fit an outdated file system model and interface, unsuitable for exascale systems. Attempts to enrich the interface, such as augmentation or replacement with databases, or the layering of additional interfaces and semantic extensions atop existing file systems result in performance-limited systems that do not adequately scale. Parallels exist between HEC systems and the web, where locating and browsing data sets has rapidly become dominated by search. The strengths and weaknesses of the web provide several useful lessons from which we have learned: 1) Although the web implements a hierarchical namespace, search has become the dominant navigation tool in the face of the massive volume of data that is accessible; 2) While finding some information is easy, finding the right or good information is not; 3) The easier it is for people to contribute information to a repository, the more critical it becomes to determine the veracity of that data; 4) The links that relate documents provide valuable insight into the importance of documents. From these observations we can see that simply modifying existing high performance filesystems to support search, and the requisite storage of additional semantic metadata, would be woefully inadequate. We propose to develop a radically different filesystem structure that addresses these problems directly, and which will leverage provenance (a record of the data and processes that contributed to its creation), file content, and rich semantic metadata to provide a scalable and searchable file namespace. Such a namespace would allow the tracking of data as it moves through the scientific workflow. This allows scientists to better find and utilize the data they need, using both content and data history to identify and manage stored information. We take advantage of the familiar search-based metaphor to provide an initial easy to-use interface that enables users to find the files they need and evaluate the authenticity and quality of those files. Realizing this vision requires research success in dynamic, nonhierarchical file systems design and implementation, large-scale metadata management, efficient scalable indexing, and automatic provenance capture.
• 
  
• ExaHDF5: An I/O Platform for Exascale Data Models, Analysis and Performance
  Prabhat, Lawrence Berkeley National Lab
  Koziol, The HDF Group
  Palmer, Pacific Northwest National Lab;
• HECFSIO Topic: XXXX
  It is reasonably well accepted that one of the primary bottlenecks in modern computational and experimental sciences is coping with the sheer volume and complexity of data. Storing, reading, finding, analyzing, and sharing data are tasks common across virtually all areas of science, yet advances in data management infrastructure, particularly I/O, have not kept pace with our ability to collect and produce scientific data. This “impedance mismatch” between our ability to produce and store/analyze data continues to grow and could, if not addressed, lead to situations where science experiments are simply not conducted or scientific data not analyzed for want of the ability to overcome data-related challenges. Our project consists of three thrust areas that address the challenges of data size and complexity on current and future computational platforms: We are extending the scalability of I/O middleware to make effective use of current and future computational platforms. We are incorporating advanced index/query technology to accelerate operations common to scientific data analysis. We are building upon our existing work on data model APIs that simplify simulation and analysis code development by encapsulating the complexity of parallel I/O. We are conducting these activities in close collaboration with specific DOE science code teams to ensure the new capabilities are responsive to scientists’ needs and are usable in production environments. Our approach includes a clear path for maintainability and production release.
• 
  
• Runtime System for I/O Staging in Support of In-Situ Processing of Extreme Scale Data
  Scott Klasky, Oak Ridge National Lab
  Arie Shoshani, Lawrence Berkeley National Lab
  Karsten Schwan, Georgia Tech
• HECFSIO Topic: XXXX
  Keywords: ADIOS, In-situ pipeline
  As we approach the extreme scale in computing, we must realize new strategies to deal with the daunting challenge of managing and exploiting the massive quantities of complex data produced by scientific simulations. The challenge is exacerbated by the fact that I/O and memory systems have not seen increases in performance at the same rate as those observed for computational elements. This not only leads to unfavorable tradeoffs concerning machine power consumption for I/O and memory vs. computation, but it also means that the time scientists will spend on analyzing and visualizing the results produced by their simulations will greatly slow down the knowledge discovery process. Our research will create and evaluate an I/O infrastructure and tools for extreme-scale applications and facilities so that they can reduce the time to discovery at small cost in machine resources and consequent power consumption. New tools must be highly scalable, portable, and easy-to-use, so that scientists can gain control of their science and concentrate on producing important scientific discovery in their own domain. Accelerating the rate of insight and scientific productivity, therefore, demands new solutions to managing the avalanche of data expected at extreme scale. Partnering with many application teams and working on petascale machines, our team has developed an approach and delivered proven technology that accelerates I/O and the knowledge discovery process by reducing, analyzing, and indexing the data produced by a simulation while it is still in memory (referred to as “in-situ” processing of data). These technologies include the Adaptable I/O system (ADIOS), FastBit indexing, and Parallel R. For the proposed project, we will leverage those technologies and integrate them to create a runtime system that will allow scientists to create easy-to-use scientific workflows that will run in situ on select nodes of the extreme scale machine. This will not only accelerate simulation and I/O, but it will also provide scientists with immediate and valuable insights into data with online methods that pre-analyze, index, visualize, and reduce the overall amount of data produced by their simulations.