Op werkdagen voor 23:00 besteld, morgen in huis Gratis verzending vanaf €20
U heeft gezocht op 9781491985571. Het product dat u zocht is niet meer in die editie leverbaar en is vervangen door de onderstaande editie.

Web Scraping with Python

Data Extraction from the Modern Web

Paperback Engels 2024 9781098145354
Verwachte levertijd ongeveer 15 werkdagen

Samenvatting

If programming is magic, then web scraping is surely a form of wizardry. By writing a simple automated program, you can query web servers, request data, and parse it to extract the information you need. This thoroughly updated third edition not only introduces you to web scraping but also serves as a comprehensive guide to scraping almost every type of data from the modern web.

Part I focuses on web scraping mechanics: using Python to request information from a web server, performing basic handling of the server's response, and interacting with sites in an automated fashion. Part II explores a variety of more specific tools and applications to fit any web scraping scenario you're likely to encounter.

- Parse complicated HTML pages
- Develop crawlers with the Scrapy framework
- Learn methods to store the data you scrape
- Read and extract data from documents
- Clean and normalize badly formatted data
- Read and write natural languages
- Crawl through forms and logins
- Scrape JavaScript and crawl through APIs
- Use and write image-to-text software
- Avoid scraping traps and bot blockers
- Use scrapers to test your website

Specificaties

ISBN13:9781098145354
Taal:Engels
Bindwijze:paperback
Aantal pagina's:300
Uitgever:O'Reilly
Druk:3
Verschijningsdatum:23-2-2024
Hoofdrubriek:IT-management / ICT
ISSN:

Lezersrecensies

Wees de eerste die een lezersrecensie schrijft!

Over Ryan Mitchell

Ryan Mitchell is a Software Engineer at LinkeDrive in Boston, where she develops their API and data analysis tools. She is a graduate of Olin College of Engineering, and is a Masters degree student at Harvard University School of Extension Studies. Prior to joining LinkeDrive, she was a Software Engineer working on web scraping and data analysis at Abine.

Andere boeken door Ryan Mitchell

Inhoudsopgave

Preface
What Is Web Scraping?
Why Web Scraping?
About This Book
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. Building Scrapers
1. How the Internet Works
Networking
Physical Layer
Data Link Layer
Network Layer
Transport Layer
Session Layer
Presentation Layer
Application Layer
HTML
CSS
JavaScript
Watching Websites with Developer Tools

2. The Legalities and Ethics of Web Scraping
Trademarks, Copyrights, Patents, Oh My!
Copyright Law
Trespass to Chattels
The Computer Fraud and Abuse Act
robots.txt and Terms of Service
Three Web Scrapers
eBay v. Bidder’s Edge and Trespass to Chattels
United States v. Auernheimer and the Computer Fraud and Abuse Act
Field v. Google: Copyright and robots.txt

3. Applications of Web Scraping
Classifying Projects
E-commerce
Marketing
Academic Research
Product Building
Travel
Sales
SERP Scraping

4. Writing Your First Web Scraper
Installing and Using Jupyter
Connecting
An Introduction to BeautifulSoup
Installing BeautifulSoup
Running BeautifulSoup
Connecting Reliably and Handling Exceptions

5. Advanced HTML Parsing
Another Serving of BeautifulSoup
find() and find_all() with BeautifulSoup
Other BeautifulSoup Objects
Navigating Trees
Regular Expressions
Regular Expressions and BeautifulSoup
Accessing Attributes
Lambda Expressions
You Don’t Always Need a Hammer

6. Writing Web Crawlers
Traversing a Single Domain
Crawling an Entire Site
Collecting Data Across an Entire Site
Crawling Across the Internet

7. Web Crawling Models
Planning and Defining Objects
Dealing with Different Website Layouts
Structuring Crawlers
Crawling Sites Through Search
Crawling Sites Through Links
Crawling Multiple Page Types
Thinking About Web Crawler Models

8. Scrapy
Installing Scrapy
Initializing a New Spider
Writing a Simple Scraper
Spidering with Rules
Creating Items
Outputting Items
The Item Pipeline
Logging with Scrapy
More Resources

9. Storing Data
Media Files
Storing Data to CSV
MySQL
Installing MySQL
Some Basic Commands
Integrating with Python
Database Techniques and Good Practice
“Six Degrees” in MySQL
Email

II. Advanced Scraping
10. Reading Documents
Document Encoding
Text
Text Encoding and the Global Internet
CSV
Reading CSV Files
PDF
Microsoft Word and .docx

11. Working with Dirty Data
Cleaning Text
Working with Normalized Text
Cleaning Data with Pandas
Cleaning
Indexing, Sorting, and Filtering
More About Pandas

12. Reading and Writing Natural Languages
Summarizing Data
Markov Models
Six Degrees of Wikipedia: Conclusion
Natural Language Toolkit
Installation and Setup
Statistical Analysis with NLTK
Lexicographical Analysis with NLTK
Additional Resources

13. Crawling Through Forms and Logins
Python Requests Library
Submitting a Basic Form
Radio Buttons, Checkboxes, and Other Inputs
Submitting Files and Images
Handling Logins and Cookies
HTTP Basic Access Authentication
Other Form Problems

14. Scraping JavaScript
A Brief Introduction to JavaScript
Common JavaScript Libraries
Ajax and Dynamic HTML
Executing JavaScript in Python with Selenium
Installing and Running Selenium
Selenium Selectors
Waiting to Load
XPath
Additional Selenium WebDrivers
Handling Redirects
A Final Note on JavaScript

15. Crawling Through APIs
A Brief Introduction to APIs
HTTP Methods and APIs
More About API Responses
Parsing JSON
Undocumented APIs
Finding Undocumented APIs
Documenting Undocumented APIs
Combining APIs with Other Data Sources
More About APIs

16. Image Processing and Text Recognition
Overview of Libraries
Pillow
Tesseract
NumPy
Processing Well-Formatted Text
Adjusting Images Automatically
Scraping Text from Images on Websites
Reading CAPTCHAs and Training Tesseract
Training Tesseract
Retrieving CAPTCHAs and Submitting Solutions

17. Avoiding Scraping Traps
A Note on Ethics
Looking Like a Human
Adjust Your Headers
Handling Cookies with JavaScript
TLS Fingerprinting
Timing Is Everything
Common Form Security Features
Hidden Input Field Values
Avoiding Honeypots
The Human Checklist

18. Testing Your Website with Scrapers
An Introduction to Testing
What Are Unit Tests?
Python unittest
Testing Wikipedia
Testing with Selenium
Interacting with the Site

19. Web Scraping in Parallel
Processes Versus Threads
Multithreaded Crawling
Race Conditions and Queues
More Features of the Threading Module
Multiple Processes
Multiprocess Crawling
Communicating Between Processes
Multiprocess Crawling—Another Approach

20. Web Scraping Proxies
Why Use Remote Servers?
Avoiding IP Address Blocking
Portability and Extensibility
Tor
PySocks
Remote Hosting
Running from a Website-Hosting Account
Running from the Cloud
Moving Forward
Web Scraping Proxies
ScrapingBee
ScraperAPI
Oxylabs
Zyte
Additional Resources

Index
About the Author

Managementboek Top 100

Rubrieken

Populaire producten

    Personen

      Trefwoorden

        Web Scraping with Python