Mission 1 · Spec 4.11
What is Big Data?
Volume, velocity and variety; why Big Data doesn't fit in relational databases; and processing it across many machines.
- Starter 5 min
- Learn 15 min
- Lab 10 min
- Quiz 10 min
- Exam 15 min
Every second
Every second, the world sends millions of messages, posts, sensor readings and card payments. Could a single database server on one machine keep up?
Reveal
No. Data like this is too big, arrives too fast and is too varied in form. That's Big Data, and it needs to be stored and processed across many machines.
Key ideas
Volume
Too much data to fit on a single server.
Velocity
Data arrives (and must be processed) very quickly, often as a continuous stream in real time.
Variety
Data comes in many forms: structured, unstructured text, images, video, sensor readings.
Why not relational?
Relational databases need data in rows and columns with a fixed structure, and are hard to scale across machines. Big Data is often unstructured.
Distributed processing
Big Data is spread across many servers and processed in parallel. Machine learning is used to find patterns in it.
Volume, velocity or variety?
Exam-style questions
1. Explain why Big Data is difficult to store in a relational database.
[3 marks]Mark scheme
- Relational databases require data to fit a fixed structure of rows and columns (1)
- Big Data is often unstructured / varied (1)
- The volume is too great for one server, and relational databases are hard to distribute across many machines (1)
TUTOR NOTES
- Exam habit: give an example for each "V" in context.
Mission 2 · Spec 4.11
Functional programming and modelling Big Data
Why functional programming suits distributed processing, and the fact-based model and graph schema for representing Big Data.
- Starter 5 min
- Learn 15 min
- Lab 15 min
- Quiz 10 min
- Exam 15 min
Split the work
1,000 machines each count words in part of a huge text. Why is it important that each machine's counting can't change any shared variable?
Reveal
If machines changed shared state, they'd interfere and results would depend on timing. Functional programming avoids side effects and mutable state, so work can be split and run in parallel safely.
Key ideas
Why functional?
Immutable data structures, statelessness and higher-order functions (map, filter, reduce) make it easy to write correct, efficient distributed code, because functions have no side effects.
Fact-based model
Each fact captures a single piece of information and is timestamped; facts are immutable and never deleted, so the full history is kept.
Graph schema
Nodes represent entities, edges represent relationships between them, and properties store information about entities, e.g. a user node with a "follows" edge to another user.
Map, filter, fold
Graph schema vocabulary
Exam-style questions
1. Explain why functional programming is suited to processing Big Data across multiple machines.
[3 marks]Mark scheme
- Functions have no side effects / data is immutable (1)
- so parts of the computation can run in parallel on different machines without interfering (1)
- Higher-order functions such as map and reduce naturally split work and combine results (1)
TUTOR NOTES
- Link: section 4.12 covers the functional programming itself in more depth.