Hacks, Leaks, and Revelations

Data-science investigations have brought journalism into the 21st century, and—guided by The Intercept’s infosec expert Micah Lee— this book is your blueprint for uncovering hidden secrets in hacked datasets.

In the current age of hacking and whistleblowing, the internet contains massive troves of leaked information. These complex datasets can be goldmines of revelations in the public interest— if you know how to access and analyze them. For investigative journalists, hacktivists, and amateur researchers alike, this book provides the technical expertise needed to find and transform unintelligible files into groundbreaking reports.

Guided by renowned investigative journalist and infosec expert Micah Lee, who helped secure Edward Snowden’s communications with the press, youʼll learn the tools, technologies, and programming basics needed to crack open and interrogate datasets freely available on the internet or your own private datasets obtained directly from sources. Each chapter features hands-on exercises using real hacked data from governments, companies, and political groups, as well as interesting nuggets from datasets that never made it into published stories. You’ll dig into hacked files from the BlueLeaks law enforcement records, analyze social-media traffic related to the 2021 attack on the U.S. Capitol, and get the exclusive story of privately leaked data from anti-vaccine group America’s Frontline Doctors. Along the way, you’ll learn:

• How to secure and authenticate datasets and safely communicate with sources
• Python programming basics needed for data science investigations
• Security concepts, like disk encryption
• How to work with data in EML, MBOX, JSON, CSV, and SQL formats
• Tricks for using the command-line interface to explore datasets packed with secrets

About the Author

Micah Lee is the Director of Information Security at The Intercept and is known for helping secure Edward Snowden's communications while he leaked secret NSA documents. He used to work for the Electronic Frontier Foundation, and is currently an advisor to the transparency collective Distributed Denial of Secrets. He is also co-founder of the Freedom of the Press Foundation, a Tor Project core contributor, and he develops open source security and privacy tools like OnionShare and Dangerzone.

Table of contents

Introduction
Part I: Sources and Datasets
Chapter 1: Protecting Sources and Yourself
Chapter 2: Acquiring Datasets

Part II: Tools of the Trade
Chapter 3: The Command Line Interface
Chapter 4: Exploring Datasets in the Terminal
Chapter 5: Docker, Aleph, and Making Datasets Searchable
Chapter 6: Reading Other People’s Email

Part III: Python Programming

Chapter 7: An Introduction to Python
Chapter 8: Working with Data in Python

Part IV: Structured Data
Chapter 9: BlueLeaks, Black Lives Matter, and the CSV File Format
Chapter 10: BlueLeaks Explorer
Chapter 11: Parler, the January 6 Insurrection, and the JSON File Format
Chapter 12: Epik Fail, Extremism Research, and SQL Databases

Part V: Case Studies
Chapter 13: Pandemic Profiteers and COVID-19 Disinformation
Chapter 14: Neo-Nazis and Their Chat Rooms
Afterword
Appendix A: Solutions to Common WSL Problems
Appendix B: Scraping the Web
Index