Welcome to the DataEngineer Hub glossary — a comprehensive reference of 50 essential data engineering terms organized across 11 categories. Whether you are preparing for interviews, studying for certifications, or looking up unfamiliar concepts in documentation, this glossary provides clear, practical definitions written by experienced data engineers.
Each term includes a concise definition, key points, frequently asked questions, and links to related concepts. Topics span the full data engineering landscape: ETL/ELT pipelines, data warehousing, real-time streaming, cloud platforms, data orchestration, data quality, governance, observability, and analytics engines.
A unified analytics engine for large-scale data processing, providing high-level APIs for batch processing, streaming, machine learning, and graph ...
An in-process analytical database (OLAP) that runs inside your application — think SQLite but for analytics. It processes data locally with zero in...
A centralized platform for storing, managing, and serving ML features — ensuring consistency between model training and real-time inference.
A category of software tools that provide analysis of data stored in a database, typically for multi-dimensional business reporting.
A database optimized for storing and searching high-dimensional vectors (embeddings), enabling semantic search, RAG pipelines, and AI-powered appli...
A unified data analytics platform that combines data engineering, data science, and machine learning on a lakehouse architecture, built on Apache S...
The practice of managing and provisioning infrastructure through machine-readable definition files, rather than physical hardware configuration or ...
Snowflake's developer framework that enables data engineers and data scientists to write data pipelines, ML models, and UDFs in Python, Java, or Sc...
A centralized inventory of data assets in an organization, providing metadata, documentation, search capabilities, and lineage to enable data disco...
An architecture that uses AI and metadata to automatically discover, integrate, and govern data across distributed environments — making data acces...
A framework of policies, processes, and standards for managing data assets across an organization, ensuring data is secure, compliant, and properly...
The documentation and visualization of data as it flows from source to destination, showing transformations, dependencies, and ownership at each step.
A decentralized sociotechnical approach to sharing, accessing, and managing analytical data in complex and large-scale environments.
An open-source data integration platform with 300+ connectors for syncing data from APIs, databases, and files to data warehouses and lakes.
An open-source data integration platform that provides a visual drag-and-drop interface for automating data flow between systems, with built-in dat...
A technique for identifying and capturing changes made to data in a database, enabling real-time or near-real-time data replication to other systems.
An automated series of steps that extracts data from sources, transforms it, and loads it into a destination — the backbone of every data-driven or...
A data integration approach that provides a unified, real-time view of data from multiple sources without physically moving or replicating the data.
A fully managed data integration platform that automatically syncs data from hundreds of sources to data warehouses and lakes with minimal configur...
The process of moving data from a data warehouse back into operational systems (SaaS tools) used for business.
A set of properties of database transactions intended to guarantee data validity despite errors, power failures, and other mishaps.
A theorem stating that a distributed data store can only guarantee two of the three: Consistency, Availability, and Partition Tolerance.
The process of creating a visual representation of data structures and relationships, defining how data is stored, organized, and accessed in datab...
A data modeling technique optimized for data warehousing and BI, organizing data into fact tables (measurements) and dimension tables (context) for...
A data warehouse modeling technique that organizes data into a central fact table surrounded by dimension tables, optimized for fast analytical que...
Formal agreements between data producers and consumers that define the structure, semantics, and quality expectations of data, enabling reliable da...
The measure of how well data meets the requirements for its intended use, encompassing accuracy, completeness, consistency, timeliness, and validity.
An open-source Python framework for defining, documenting, and validating data quality expectations against datasets in data pipelines.
An open-source data quality testing framework that uses a simple YAML-based language (SodaCL) to define data quality checks and integrates with any...
An open-source data lakehouse platform that provides ACID transactions, record-level insert/update/delete, and incremental data processing on data ...
An open table format for huge analytic datasets that brings warehouse-like features (ACID transactions, time travel, schema evolution) to data lakes.
A database management system that stores data in columns rather than rows, optimized for analytics.
A centralized storage repository that holds vast amounts of raw data in its native format until needed for analysis, supporting structured, semi-st...
A modern data architecture that combines the low-cost, flexible storage of data lakes with the performance, ACID transactions, and governance of da...
A centralized repository designed to store, integrate, and analyze large volumes of structured data from multiple sources for business intelligence...
The use of tools and methodologies to automate the design, development, deployment, and management of data warehouses, reducing manual coding and a...
An open-source storage layer that brings ACID transactions, scalable metadata handling, and unified batch/streaming processing to data lakes.
A modern data architecture that combines the performance and governance of data warehouses with the low-cost and flexibility of data lakes.
A data design pattern that organizes data into three layers — Bronze (raw), Silver (cleaned), and Gold (business-ready) — to progressively improve ...
A concept in data warehousing to manage how data that changes slowly over time is stored and tracked.
A cloud-native data warehouse platform that separates storage and compute, enabling elastic scaling and pay-per-use pricing.
A layer that organizes files in a data lake into a structured table, enabling SQL-like features like ACID and time-travel.
An open-source transformation tool that enables data analysts and engineers to transform data in their warehouse using SQL and software engineering...
A data integration process that extracts data from source systems, transforms it into a suitable format, and loads it into a target data warehouse ...
A distributed stream processing framework that processes data event-by-event with millisecond latency, built for stateful computations over unbound...
A distributed event streaming platform used for building real-time data pipelines and streaming applications, handling trillions of events per day.
A stream processing technique where the application remembers information across multiple events in time.
A data processing paradigm that continuously ingests, processes, and analyzes data in real-time as it arrives, rather than waiting for batch collec...
Looking for side-by-side tool comparisons? Check out our Data Engineering Comparisons hub to see how popular tools like Snowflake, Databricks, Spark, Kafka, and more stack up against each other.
For hands-on tutorials and in-depth guides, browse our full article library.