Data is the lifeblood of data science. However, before a data scientist can clean, analyze, or build a model, they must ask a fundamental question: What is the structural nature of the data at hand?
How data is organized dictates the storage systems, query engines, mathematical models, and engineering pipelines required to process it. In this chapter, we explore the classic taxonomy of data and examine the modern hybrid architectures designed to bridge them.
By the end of this chapter, you will be able to:
The following diagram maps the entire landscape of data formats, illustrating how raw unstructured data, traditional relational schemas, and modern hybrid vector/metadata engines connect.
Loading diagram...
To compare these formats side-by-side, consider their core characteristics:
| Aspect | Structured Data | Semi-Structured Data | Unstructured Data |
|---|---|---|---|
| Format | Strict predefined schema | Flexible, self-describing schema | No predefined schema or model |
| Storage (Classic) | PostgreSQL, MySQL, SQLite | MongoDB, Cassandra, S3 | Object storage, local file systems |
| Storage (Modern) | Snowflake, Google BigQuery | Cassandra, Amazon DynamoDB | Vector Databases, S3/Data Lakes |
| Query Mechanism | SQL (Structured Query Language) | MongoDB queries, JSONPath, Spark SQL | Vector Search, NLP, Computer Vision |
| ETL / Parsing Tools | SQL, Pandas, dbt | PySpark, json_normalize, Jackson | LangChain, Unstructured.io, OpenCV |
| ML Use Cases | Tabular modeling, XGBoost, Regression | Feature engineering, recommendations | LLMs, embeddings, CNNs, clustering |
| Examples | SQL tables, CSVs, Excel | JSON document, XML, Server logs | Video files, audio files, raw PDF text |
| Global Data Vol. | ~20% of world's data | ~10% of world's data | ~80% of world's data |
| Parsing Difficulty | Extremely low | Medium (requires normalization) | High (requires feature extraction) |
Structured data represents the most traditional and clean format. It conforms to a strict, rigid schema—meaning every record possesses the exact same fields, and each field is bound to a specific data type (such as integer, string, or float).
Schema: A blueprint or formal description of the database structure, outlining the tables, fields, data types, and relationships governing the entries.
Because of this predictable layout, structured data is highly searchable using relational query engines. It is typically stored in standard relational databases (RDBMS) or tabular flat files.
Consider a simple representation of a classic data science classification benchmark, the Iris flower dataset:
| Sepal_length | Sepal_width | Petal_length | Petal_width | Species |
|---|---|---|---|---|
| 5.1 | 3.5 | 1.4 | 0.2 | setosa |
| 4.9 | 3.0 | 1.4 | 0.2 | versicolor |
| 4.7 | 3.2 | 1.3 | 0.2 | setosa |
| 4.6 | 3.1 | 1.5 | 0.2 | virginica |
Because every cell contains a predictable data type, reading and computing aggregates across millions of rows takes minimal compute power and can be accomplished with high speed.
What this code does: Uses Python's pandas library to load standard structured CSV data, checking its column schema and verifying data consistency immediately.
Loading code block...
Semi-structured data sits between clean schemas and chaotic raw assets. It does not conform to a classic relational system, but it is not completely unstructured either. Instead, it contains internal markers, tags, or keys that organize the data elements.
This "self-describing" property gives organizations immense schema flexibility—different rows can contain completely different variables.
Traditional relational databases struggle with self-describing, changing datasets. This led to the development of several distinct NoSQL database architectures:
The premier standard for REST APIs and web services. It presents information in lightweight, nested key-value pairs.
What this data represents: A JSON string denoting a customer profile with an embedded, dynamic purchase list.
Loading code block...
What this code does: Parses nesting in a JSON dataset and flattens it into a tabular pandas DataFrame.
Loading code block...
What this data represents: An XML document containing semi-structured customer profile details marked up with descriptive tags.
Loading code block...
Server and application logs are a practical example of semi-structured data. They have a rigid format for metadata (timestamp, severity, source) but free-text content:
What this log shows: Sample raw log entries depicting server messages of varying severity levels with timestamp metadata.
Loading code block...
Data engineers often parse these using regex or structured logging libraries (like JSON logging) to extract the timestamp, level, and source into structured fields, leaving the message as unstructured text.
Graph data represents entities and their relationships. Unlike traditional tables (which focus on attributes) or document stores (which focus on isolated documents), graphs excel at capturing connections.
Examples:
Graph databases (like Neo4j) store these relationships efficiently and allow lightning-fast queries like "find all friends of friends" or "find shortest path between two nodes" without complex relational joins.
Unstructured data has no predefined format. It is the most common type of data in the world (emails, social media posts, images, videos, audio recordings) and also the most challenging to work with. It typically requires Natural Language Processing (NLP) or Computer Vision techniques.
What this code does: Cleans and tokenizes unstructured text to compute and output word frequency counts.
Loading code block...
Relying on manual word-matching rules has limitations. If one document contains "cat" and another contains "kitten," a word counter sees zero overlap. Modern pipelines solve this by converting unstructured data into embeddings.
Core Intuition: An Embedding is an array of numerical coordinates (a vector) generated by a deep neural network, representing the contextual meaning of an unstructured asset.
For example, the sentence "Quantum computing uses quantum-mechanical phenomena" is transformed into a 384-dimensional vector coordinate: $$\vec{v} = [-0.02, 0.15, -0.08, \dots, 0.04]$$
Because similar concepts are mapped to nearby vector coordinates, a computer can immediately calculate mathematical distances (like Cosine Similarity) to understand relationships:
Loading code block...
As organizations became saturated with diverse data types, classical database limits were reached. The necessity to run deep analytics on vector coordinates and extract structured values from raw files gave birth to hybrid system patterns.
Loading diagram...
Historically, enterprise data was split between two architectures:
A Data Lakehouse bridges this gap, installing an open metadata interface layer directly over cheap data lake file objects to support ACID database transactions, safety constraints, and fast SQL queries.
Loading diagram...
These lake house layers (led by Apache Iceberg, Delta Lake, and Apache Hudi) allow companies to write SQL queries on unstructured or semi-structured folders as if they were classic relational columns.
In real-world data science, clean taxonomic boundaries are a luxury. Most data flows are complex hybrids:
<header>, <div class="content">), but contain completely free-form text paragraphs inside. When scraped, the tags are handled with structured web parsing, while the inside content is evaluated using NLP techniques.If you struggle to remember the difference between these types, use this analogy:
Test your understanding of the chapter concepts:
Question: A dataset contains raw video recordings of traffic patterns and a separate metadata file detailing timestamps and sensor locations. Analyze how you would categorize this collection. Answer: This is a hybrid collection. The raw video recordings are unstructured data (requiring computer vision to extract meaning), while the metadata file containing timestamps and locations is structured data (if in CSV/SQL) or semi-structured (if in JSON).
Question: Why are traditional database indexes (like B-Trees) insufficient for querying unstructured vectors, and what technology addresses this? Answer: B-Trees are optimized for one-dimensional exact matches or ranges (e.g., "is X > 10?"). Unstructured vectors are high-dimensional mathematical coordinates. Finding contextually similar items requires calculating spatial proximity (Nearest Neighbors), which is handled by specialized Vector Databases using spatial indexing.
Question: Explain how table formats like Apache Iceberg help transform a "Data Lake" into a "Data Lakehouse." Answer: Apache Iceberg acts as a metadata management layer. It tracks which files belong to a table, enabling ACID transactions (Atomicity, Consistency, Isolation, Durability) and fast SQL queries directly on top of cheap object storage like S3, effectively giving a Data Lake the reliability of a Data Warehouse.
Question: If a data team is working with volatile Customer Profiles where keys change weekly, would you recommend a PostgreSQL database or a document-based NoSQL database? Justify your choice. Answer: A document-based NoSQL database (like MongoDB) is the better choice. Because it is schema-less, it can handle "volatile" profiles where keys change frequently without requiring expensive table migrations or downtime, unlike a relational PostgreSQL database which requires a rigid, pre-defined schema.