Syllabus

Introduction

• Introduction to Scala and Spark

Scala Programming: Level One

• Arithmetic and Numbers
• Values and Variables
• Booleans and Comparison Operators
• Strings and Basic Regex
• Tuples
• Scala Basics - Assessment Test Exercises

Collections

• Intro to Collections
• Lists
• Arrays
• Sets
• Maps
• Collections - Assessment Test Exercise

Scala Programming: Level Two

• Flow Control
• For Loops
• While Loops
• Functions
• Scala Programming Exercises

Spark programming basics

• The Resilient Distributed Dataset
• Ratings Histogram Walkthrough
• Spark Internals
• Key / Value RDD's, and the Average Friends by Age example
• [POC] Running the Average Friends by Age Example
• Filtering RDD's, and the Minimum Temperature by Location Example
• [POC] Running the Minimum Temperature Example, and Modifying it for Maximum
• [POC] Counting Word Occurrences using Flatmap()
• [POC] Improving the Word Count Script with Regular Expressions
• [POC] Sorting the Word Count Results
• [Exercise] Find the Total Amount Spent by Customer and Sort Them by Total Amount Spent

Advanced Examples of Spark Programs

• [POC] Find the Most Popular Movie
• [POC] Use Broadcast Variables to Display Movie Names
• [POC] Find the Most Popular Superhero in a Social Graph
• Superhero Degrees of Separation: Introducing Breadth-First Search
• Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark
• Superhero Degrees of Separation: Review the code, and run it!
• Item-Based Collaborative Filtering in Spark, cache(), and persist()
• [POC] Running the Similar Movies Script using Spark's Cluster Manager
• [Exercise] Improve the Quality of Similar Movies

Running Spark on a Cluster

• [POC] Using spark-submit to run Spark driver scripts
• [POC] Packaging driver scripts with SBT
• Introducing Amazon Elastic MapReduce
• Creating Similar Movies from One Million Ratings on EMR
• Partitioning
• Best Practices for Running on a Cluster
• Troubleshooting, and Managing Dependencies

Spark DataFrames and Datasets

• Introduction to Spark DataFrames
• DataFrames Overview
• Spark DataFrame Operations
• GroupBy and Aggregate Functions
• Missing data
• Date and Timestamps
• DataFrame Project Exercises
• [POC] Using DataSets instead of RDD's

Regression with Spark

• Introduction to Linear Regression
• Introduction to Regression Section
• Linear Regression Documentation Example
• Alternate Linear Regression Data CSV File
• Linear Regression Walkthrough Part 1
• Linear Regression Walkthrough Part 2
• Linear Regression Exercise Project

Classification with Spark

• Introduction to Classification
• Classification Documentation Example
• Spark Classification - Logistic Regression Example - Part 1
• Spark Classification - Logistic Regression Example - Part 2
• Logistic Regression Project Exercise

Model Evaluation

• Model Evaluation Overview
• Spark Model Evaluation - Documentation Example
• Spark - Model Evaluation - Regression Example

Clustering with Spark

• Introduction to Clustering with Spark
• KMeans Theory Lecture
• Example of KMeans with Spark
• Clustering Project Exercise Overview

PCA with Spark

• PCA Theory Overview
• PCA with Spark - Project Exercise
• PCA with Spark - Documentation Example

DataBricks and Spark

• Databricks Overview
• Introduction to Spark Recommendation Systems
• Spark Recommender System Implementation
• Zeppelin Notebooks on AWS Elastic MapReduce.

Intro to Spark Streaming

• Spark Streaming Overview
• [POC] Set up a Twitter Developer Account, and Stream Tweets
• Structured Streaming

Projects

• Live Twitter Sentiments Analysis using Spark
• Yelp Data Processing Using Spark and Hive Part 1