Data Analysis with Python and PySpark (Paperback)

Think big about your data! PySpark brings the powerful Spark big data processing engine to the Python ecosystem, letting you seamlessly scale up your data tasks and create lightning-fast pipelines.

In Data Analysis with Python and PySpark you will learn how to:

Manage your data as it scales across multiple machines
Scale up your data programs with full confidence
Read and write data to and from a variety of sources and formats
Deal with messy data with PySpark’s data manipulation functionality
Discover new data sets and perform exploratory data analysis
Build automated data pipelines that transform, summarize, and get insights from data
Troubleshoot common PySpark errors
Creating reliable long-running jobs

Data Analysis with Python and PySpark is your guide to delivering successful Python-driven data projects. Packed with relevant examples and essential techniques, this practical book teaches you to build pipelines for reporting, machine learning, and other data-centric tasks. Quick exercises in every chapter help you practice what you’ve learned, and rapidly start implementing PySpark into your data systems. No previous knowledge of Spark is required.

About the technology
The Spark data processing engine is an amazing analytics factory: raw data comes in, insight comes out. PySpark wraps Spark’s core engine with a Python-based API. It helps simplify Spark’s steep learning curve and makes this powerful tool available to anyone working in the Python data ecosystem.

About the book
Data Analysis with Python and PySpark helps you solve the daily challenges of data science with PySpark. You’ll learn how to scale your processing capabilities across multiple machines while ingesting data from any source—whether that’s Hadoop clusters, cloud data storage, or local data files. Once you’ve covered the fundamentals, you’ll explore the full versatility of PySpark by building machine learning pipelines, and blending Python, pandas, and PySpark code.

What's inside

Organizing your PySpark code
Managing your data, no matter the size
Scale up your data programs with full confidence
Troubleshooting common data pipeline problems
Creating reliable long-running jobs

About the reader
Written for data scientists and data engineers comfortable with Python.

About the Author
Jonathan Rioux As a ML director for a data-driven software company, Jonathan Rioux uses PySpark daily. He teaches the software to data scientists, engineers, and data-savvy business analysts.

Table of Contents

1 Introduction
PART 1 GET ACQUAINTED: FIRST STEPS IN PYSPARK
2 Your first data program in PySpark
3 Submitting and scaling your first PySpark program
4 Analyzing tabular data with pyspark.sql
5 Data frame gymnastics: Joining and grouping

PART 2 GET PROFICIENT: TRANSLATE YOUR IDEAS INTO CODE
6 Multidimensional data frames: Using PySpark with JSON data
7 Bilingual PySpark: Blending Python and SQL code
8 Extending PySpark with Python: RDD and UDFs
9 Big data is just a lot of small data: Using pandas UDFs
10 Your data under a different lens: Window functions
11 Faster PySpark: Understanding Spark’s query planning

PART 3 GET CONFIDENT: USING MACHINE LEARNING WITH PYSPARK
12 Setting the stage: Preparing features for machine learning
13 Robust machine learning with ML Pipelines
14 Building custom ML transformers and estimators

Ebook License

End-User Warranty And License Agreement

1. Grant Of License

Manning Has Authorized The Download By You Of An Unrestricted Number Of Copies Of The Electronic Book (Ebook) In Any Of The Available Formats. Manning Grants You A Nonexclusive, Nontransferable License To Use The Ebook According To The Terms And Conditions Herein. This License Agreement Permits You To Install The Ebook On Any And All Your Devices For Your Personal Use Only.

2. Restrictions

You Shall Not: (1) Share, Resell, Rent, Assign, Timeshare, Distribute, Or Transfer All Or Part Of The Ebook Or Any Rights Granted Hereunder To Any Other Person; (2) Duplicate The Ebook, Except For A Single Backup Or Archival Copy; (3) Remove Any Proprietary Notices, Labels, Or Marks From The Ebook; (4) Transfer Or Sublicense Title To The Ebook To Any Other Party.

3. Intellectual Property Protection

The Ebook Is Owned By Manning And Is Protected By United States And International Copyright And Other Intellectual Property Laws. Manning Reserves All Rights In The Ebook Not Expressly Granted Herein. This License And Your Right To Use The Ebook Terminate Automatically If You Violate Any Part Of This Agreement. In The Event Of Termination, You Must Remove The Original And Any Copies Of The Ebook From All Your Devices.

4. Source Code Supplementary Material

Any Source Code Files Provided As A Supplement To The Book Are Freely Available To The Public For Download. Reuse Of The Code Is Permitted, In Whole Or In Part, Including The Creation Of Derivative Works, Provided That You Acknowledge That You Are Using It And Identify The Source: Title, Publisher And Year.

5. Limited Warranty

Manning Warrants That The Ebook Files, A Copy Of Which You Are Authorized To Download, Are Free From Defects In The Operational Sense That They Can Be Read By A Pdf Reader Or Epub Reader, Or Other. Except For This Express Limited Warranty, Manning Makes And You Receive No Warranties, Express, Implied, Statutory Or In Any Communication With You, And Manning Specifically Disclaims Any Other Warranty Including The Implied Warranty Of Merchantability Or Fitness Or A Particular Purpose. Manning Does Not Warrant That The Operation Of The Ebook Will Be Uninterrupted Or Error Free. If The Ebook Was Purchased In The United States, The Above Exclusions May Not Apply To You As Some States Do Not Allow The Exclusion Of Implied Warranties. In Addition To The Above Warranty Rights, You May Also Have Other Rights That Vary From State To State.

6. Limitation Of Liability

In No Event Will Manning Be Liable For Any Damages, Whether Arising For Tort Or Contract, Including Loss Of Data, Lost Profits, Or Other Special, Incidental, Consequential, Or Indirect Damages Arising Out Of The Use Or Inability To Use The Ebook.

7. General

This Agreement Constitutes The Entire Agreement Between You And Manning And Supersedes Any Prior Agreement Concerning The Ebook. This Agreement Is Governed By The Laws Of The State Of New York.