Semalt Expert: Python and BeautifulSoup. Scrape Sites With Ease

23.05.2018

Semalt Expert: Python and BeautifulSoup. Scrape Sites With Ease

When performing data analysis or machine learning projects, you might need to scrape websites to get the data needed and complete your project. Python programming language has a powerful collection of tools and modules that can be used for this purpose. For instance, you can use the BeautifulSoup module for HTML parsing.

Here, we'll take a look at BeautifulSoup and nd out why it is now being so widely used in web scraping.

BeautifulSoup features - It provides various methods for easy navigation, searching and modifying of parse trees thus allowing you to easily dissect a document and extract everything you need without writing too much code.

- It automatically converts outgoing documents to UTF-8 and incoming documents to Unicode. This means you will not have to worry about encodings provided that the document has speci ed an encoding or Beautiful Soup can autodetect it.

- BeautifulSoup is considered superior to other popular Python parsers such as html5lib and lxml. It allows trying different parsing strategies. One disadvantage of this module, however, is that it provides more exibility at the https://rankexperience.com/articles/article2090.html

1/3

23.05.2018

expense of speed.

What do you need to scrape website with BeautifulSoup?

To start working with BeautifulSoup, you need to have Python programming environment (either local or serverbased) set up on your machine. Python is usually pre-installed in OS X, but if you use Windows, you'll need to download and install the language from the of cial website.

You should have the BeautifulSoup and Requests modules installed.

Lastly, being familiar and comfortable working with HTML tagging and structure is de nitely useful since you'll be working with web-sourced data.

Importing Requests and BeautifulSoup libraries With Python programming environment well set up, you can now create a new le ( using nano, for instance) with any name you like.

The Requests library enables you to use a human-readable form HTTP within your Python programs while BeautifulSoup gets the scraping done at a faster speed. You can use the import statement to get both libraries.

How to collect and parse a web page Use the requests.get() method to collect the URL of the web page from which you want to extract data. Next, create a BeautifulSoup object or parse tree. This object takes the document from Requests as its arguments and then parses it. With the page collected, parsed and set up as a BeautifulSoup object, you can then proceed to collect the data you need.

Extracting the desired text from the parsed web page Whenever you want to collect web data, you need to know how that data is described by the Document Object Model (DOM) of the web page. In your web browser, right-click (if using Windows), or CTRL + click (if using macOS) on one of the items forming part of the data of interest. For instance, if you want to pull out data about students' nationalities, click on one of the names of a student. A context menu pops up, and within it, you'll see a menu item similar to Inspect Element (for Firefox) or Inspect (for Chrome). Click the relevant Inspect menu item, and the web developer tools will appear within your browser.

BeautifulSoup is a simple yet powerful HTML parsing tool that allows you a great deal of exibility when scraping websites. When using it, don't forget to observe general scraping rules such as checking the website's Terms and Conditions; revisiting the site regularly and updating your code as per the changes made on the site. Having this https://rankexperience.com/articles/article2090.html

2/3

23.05.2018

knowledge about scraping websites with Python and BeautifulSoup, you can now easily get the web data you need for your project.

https://rankexperience.com/articles/article2090.html

3/3