Member-only story
Web Scraping From Scratch With 3 Simple Steps
Introduction
Web scraping or crawling refers to the technique to extract the information from a website and transform into structured data for later analysis. There are generally a few reasons that you may need to implement a web scraping scripts to automate the data collection process:
- There isn’t any public API available for you to get data from the source sites
- The information is updated from time to time, such as the exchange rate, you cannot manage it in a manual way
- The final data you need is piecemeal from multiple sites; and so on
Before you decide to implement a scraping script, you will also need to check to be sure that you are not violating the term of use for the data you are going to scrape. Some sites are against the scraping robot. This article is intended for education purpose to help you to understand the overall processes of web scraping, so we will assume you already know the implication of the web scraping and possible legal issues on how the data is used.
Scraping a website sometimes can be difficult depends on how the target website is designed and where the data is resided. But generally you can split the process into 3 steps. Let’s walk through them one by one.
