All seminars will take place on Fridays at 11 a.m., either via Zoom or in-person. Check seminar details.
January 12, 2018
11:00am - 12:00pm
Ingestion, Indexing and Retrieval of High-Velocity Multidimensional Sensor Data on a Single Node
Sources of multidimensional data are becoming more prevalent, partly due to the rise of the Internet of Things (IoT), and so is the need to ingest and analyze data streams at rates higher than before. Some industrial IoT applications require ingesting millions of records per second, while processing queries on recently ingested and historical data. Unfortunately, existing database systems targeting multidimensional data exhibit low per-node ingestion performance, and even if they can scale horizontally in distributed settings, they require large number of nodes to meet such ingest demands. For this reason, in this paper we present a single-node datastore able to ingest multidimensional sensor data at very high rates. Its design centers around a two-level indexing structure, wherein the global index is an in-memory R*-tree and the local indices are serialized kd-trees. This study is confined to records with numerical indexing fields and range queries, and covers ingest throughput, query response time, and storage footprint. We show that the adopted design streamlines data ingestion and offers ingress rates two orders of magnitude higher than those of a selection of open-source database systems, namely Percona Server, SQLite, and Druid. Our prototype also reports query response times comparable to or better than those of Percona Server and Druid, and compares favorably in terms of storage footprint. We believe the experience reported here is valuable to researchers and practitioners interested in building database systems for high-velocity multidimensional sensor data.
Juan A. Colmenares is a researcher at Samsung Research America (SRA) at Mountain View, CA and the technical lead of the Smart Systems team. In 2013, he was an industrial researcher for Samsung in the Ubiquitous Swarm Lab at UC Berkeley. Previously, he was a postdoctoral scholar in the UC Berkeley's Parallel Computing Laboratory (Par Lab). He obtained my Ph.D. degree in Computer Engineering from UC Irvine in 2009. He is co-author of 40+ research papers and has served as a reviewer for conferences and journals.