Getting Started with Web Scraping Using Selenium and BeautifulSoup

Categories: Python · Tags: python, スクレイピング, プログラミング

What is this?

A technical note on the basic flow of general web scraping using Selenium and BeautifulSoup.

Environment

Apple Silicon M1 MacBook Air
16 Gb
macOS Sequoia 15.0
chromedriver version : ChromeDriver 129.0.6668.58
Google Chrome version : 129.0.6668.59（Official Build）（arm64）
Verified working as of 2024/09/18

Steps

Create a directory and set up a virtual environment

mkdir ScrapingTest
cd ScrapingTest
python -m venv nenv

Install the libraries

Selenium: a library for automating web browsers. It's especially useful for scraping pages that generate content dynamically with JavaScript.
BeautifulSoup: a library for parsing a web page's HTML and extracting the data you specify.

pip install selenium beautifulsoup4 pandas webdriver-manager

For installing the Chrome driver, see this article.

https://coiai.net/blog/2361

First, to control the Chrome browser with Selenium, we set up ChromeDriver. Using webdriver_manager lets us install ChromeDriver automatically and keep it in sync with the latest version.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ChromeDriverのサービス設定
service = Service(ChromeDriverManager().install())

# ChromeDriverを初期化
driver = webdriver.Chrome(service=service)

Next, add the page you want to access. I used my own site here, but feel free to try any site you like.

# ターゲットのWebページにアクセス
driver.get('https://coiai.boy.jp')

Wait up to 20 seconds for the entire page to load. Waiting for a specific element to appear lets you handle dynamically generated content too.

# ページ全体のロードを待つ (20秒まで待機)
WebDriverWait(driver, 20).until(
    EC.presence_of_element_located((By.TAG_NAME, 'body'))
)

Use BeautifulSoup to parse the HTML source and grab the data on the page. In this example, we extract the data inside the h2 tags.

# ページのソースをBeautifulSoupで解析
soup = BeautifulSoup(driver.page_source, 'html.parser')

# h2タグのみを取得
h2_tags = soup.find_all('h2')

Store the data in a list.

# 抽出したデータをリストに格納
test_list = []

# 各h2タグの内容をリストに追加
for h2 in h2_tags:
    test_list.append({
        'テストデータ': h2.text.strip()
    })

Set it up to write the data out to CSV.

# データフレームに変換
df = pd.DataFrame(test_list)

# 結果を表示またはCSVに保存
df.to_csv('test_data.csv', index=False)
print(df)

Finally, close the browser you opened.

# ブラウザを閉じる
driver.quit()

Summary

Putting it all together, you get the code below.
Run this and it should work as expected.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ChromeDriverのサービス設定
service = Service(ChromeDriverManager().install())

# ChromeDriverを初期化
driver = webdriver.Chrome(service=service)

# ターゲットのWebページにアクセス
driver.get('https://coiai.boy.jp')

# ページ全体のロードを待つ (20秒まで待機)
WebDriverWait(driver, 20).until(
    EC.presence_of_element_located((By.TAG_NAME, 'body'))
)

# ページのソースをBeautifulSoupで解析
soup = BeautifulSoup(driver.page_source, 'html.parser')

# h2タグのみを取得
h2_tags = soup.find_all('h2')

# 抽出したデータをリストに格納
test_list = []

# 各h2タグの内容をリストに追加
for h2 in h2_tags:
    test_list.append({
        'テストデータ': h2.text.strip()
    })

# データフレームに変換
df = pd.DataFrame(test_list)

# 結果を表示またはCSVに保存
df.to_csv('test_data.csv', index=False)
print(df)

# ブラウザを閉じる
driver.quit()

In this example, we used Selenium and BeautifulSoup to extract the text inside h2 tags from a web page and save it to a CSV file. You can apply this same approach to automatically pull the data you need from all kinds of websites.

Because Selenium can handle dynamically generated content, this approach also works for scraping more complex pages. Give it a try.

And that wraps up the basics of general-purpose web scraping. Adjust the sample data and specifics to fit your own use case as needed.