Hands-On Web Scraping with Python: Extract quality data from the web using effective Python techniques

✍ Scribed by Anish Chapagain

• Publisher: Packt Publishing Pvt Ltd
• Year: 2023
• Tongue: English
• Leaves: 395
• Edition: 2
• Category: Library

✦ Synopsis

Work through practical examples to unlock the full potential of web scraping with Python and gain valuable insights from high-quality data

Key Features
Build an initial portfolio of web scraping projects with detailed explanations
Grasp Python programming fundamentals related to web scraping and data extraction
Acquire skills to code web scrapers, store data in desired formats, and employ the data pro

Book Description
Web scraping is a powerful tool for extracting data from the web, but it can be daunting for those without a technical background. Designed for novices, this book will help you grasp the fundamentals of web scraping and Python programming, even if you have no prior experience.

Adopting a practical, hands-on approach, this updated edition of Hands-On Web Scraping with Python uses real-world examples and exercises to explain key concepts. Starting with an introduction to web scraping fundamentals and Python programming, you’ll cover a range of scraping techniques, including requests, lxml, pyquery, Scrapy, and Beautiful Soup. You’ll also get to grips with advanced topics such as secure web handling, web APIs, Selenium for web scraping, PDF extraction, regex, data analysis, EDA reports, visualization, and machine learning.

This book emphasizes the importance of learning by doing. Each chapter integrates examples that demonstrate practical techniques and related skills. By the end of this book, you’ll be equipped with the skills to extract data from websites, a solid understanding of web scraping and Python programming, and the confidence to use these skills in your projects for analysis, visualization, and information discovery.

What you will learn
Master web scraping techniques to extract data from real-world websites
Implement popular web scraping libraries such as requests, lxml, Scrapy, and pyquery
Develop advanced skills in web scraping, APIs, PDF extraction, regex, and machine learning
Analyze and visualize data with Pandas and Plotly
Develop a practical portfolio to demonstrate your web scraping skills
Understand best practices and ethical concerns in web scraping and data extraction

Who this book is for
This book is for beginners who want to learn web scraping and data extraction using Python. No prior programming knowledge is required, but a basic understanding of web-related concepts such as websites, browsers, and HTML is assumed. If you enjoy learning by doing and want to build a portfolio of web scraping projects and delve into data-related studies and application, then this book is tailored for your needs.

✦ Table of Contents

Hands-On Web Scraping with Python
Contributors
About the author
About the reviewers
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Conventions used
Get in touch
Share Your Thoughts
Download a free PDF copy of this book
Part 1:Python and Web Scraping
1
Web Scraping Fundamentals
Technical requirements
What is web scraping?
Understanding the latest web technologies
HTTP
HTML
XML
JavaScript
CSS
Data-finding techniques used in web pages
HTML source page
Developer tools
Summary
Further reading
2
Python Programming for Data and Web
Technical requirements
Why Python (for web scraping)?
Accessing the WWW with Python
Setting things up
Creating a virtual environment
Installing libraries
Loading URLs
URL handling and operations
requests – Python library
Implementing HTTP methods
GET
POST
Summary
Further reading
Part 2:Beginning Web Scraping
3
Searching and Processing Web Documents
Technical requirements
Introducing XPath and CSS selectors to process markup documents
The Document Object Model (DOM)
XPath
CSS selectors
Using web browser DevTools to access web content
HTML elements and DOM navigation
XPath and CSS selectors using DevTools
Scraping using lxml – a Python library
lxml by example
Web scraping using lxml
Parsing robots.txt and sitemap.xml
The robots.txt file
Sitemaps
Summary
Further reading
4
Scraping Using PyQuery, a jQuery-Like Library for Python
Technical requirements
PyQuery overview
Introducing jQuery
Exploring PyQuery
Installing PyQuery
Loading a web URL
Element traversing, attributes, and pseudo-classes
Iterating using PyQuery
Web scraping using PyQuery
Example 1 – scraping book details
Example 2 – sitemap to CSV
Example 3 – scraping quotes with author details
Summary
Further reading
5
Scraping the Web with Scrapy and Beautiful Soup
Technical requirements
Web parsing using Python
Introducing Beautiful Soup
Installing Beautiful Soup
Exploring Beautiful Soup
Web scraping using Beautiful Soup
Web scraping using Scrapy
Setting up a project
Creating an item
Implementing the spider
Exporting data
Deploying a web crawler
Summary
Further reading
Part 3:Advanced Scraping Concepts
6
Working with the Secure Web
Technical requirements
Exploring secure web content
Form processing
Cookies and sessions
User authentication
HTML

processing using Python
User authentication and cookies
Using proxies
Summary
Further reading
7
Data Extraction Using Web APIs
Technical requirements
Introduction to web APIs
Types of API
Benefits of web APIs
Data formats and patterns in APIs
Example 1 – sunrise and sunset
Example 2 – GitHub emojis
Example 3 – Open Library
Web scraping using APIs
Example 1 – holidays from the US calendar
Example 2 – Open Library book details
Example 3 – US cities and time zones
Summary
Further reading
8
Using Selenium to Scrape the Web
Technical requirements
Introduction to Selenium
Advantages and disadvantages of Selenium
Use cases of Selenium
Components of Selenium
Using Selenium WebDriver
Setting things up
Exploring Selenium
Scraping using Selenium
Example 1 – book information
Example 2 – forms and searching
Summary
Further reading
9
Using Regular Expressions and PDFs
Technical requirements
Overview of regex
Regex with Python
re (search, match, and findall)
re.split
re.sub
re.compile
Regex flags
Using regex to extract data
Example 1 – Yamaha dealer information
Example 2 – data from sitemap
Example 3 – Godfrey’s dealer
Data extraction from a PDF
The PyPDF2 library
Extraction using PyPDF2
Summary
Further reading
Part 4:Advanced Data-Related Concepts
10
Data Mining, Analysis, and Visualization
Technical requirements
Introduction to data mining
Predictive data mining
Descriptive data mining
Handling collected data
Basic file handling
JSON
CSV
SQLite
Data analysis and visualization
Exploratory Data Analysis using ydata_profiling
pandas and plotly
Summary
Further reading
11
Machine Learning and Web Scraping
Technical requirements
Introduction to ML
ML and Python programming
Types of ML
ML using scikit-learn
Simple linear regression
Multiple linear regression
Sentiment analysis
Summary
Further reading
Part 5:Conclusion
12
After Scraping – Next Steps and Data Analysis
Technical requirements
What happens after scraping?
Web requests
pycurl
Proxies
Data processing
PySpark
polars
Jobs and careers
Summary
Further reading
Index
Why subscribe?
Other Books You May Enjoy
Packt is searching for authors like you
Download a free PDF copy of this book