Project 07 — Python · Web Scraping

Scraping Fortune 500
from Wikipedia

Built a full web scraping pipeline using Python's BeautifulSoup and Requests to extract, clean, and analyse the largest US companies by revenue from Wikipedia — no API, no CSV download, just raw HTML parsing.

Python BeautifulSoup Requests Pandas Web Scraping Data Cleaning EDA
50+
Companies scraped
7
Data columns extracted
8
Industries covered
$648B
Top company revenue
The Code
scraping_fortune500.py 
# Import libraries
from bs4 import BeautifulSoup
import requests
import pandas as pd

# Target URL — Wikipedia Fortune 500 table
url = 'https://en.wikipedia.org/wiki/List_of_largest_companies_in_the_United_States_by_revenue'
headers = {'User-Agent': 'Mozilla/5.0 ...'}

# Send HTTP request & parse HTML
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.text, 'html')

# Locate the first table on the page
table = soup.find_all('table')[0]

# Extract column headers from  tags
world_titles = table.find_all('th')
world_table_titles = [title.text.strip() for title in world_titles]

# Build DataFrame and populate row by row
df = pd.DataFrame(columns=world_table_titles)
column_data = table.find_all('tr')

for row in column_data[1:]:
    row_data = row.find_all('td')
    individual_row_data = [data.text.strip() for data in row_data]
    length = len(df)
    df.loc[length] = individual_row_data

# Export to CSV
df.to_csv('fortune500_us.csv', index=False)
Process
01
HTTP Request
Sent GET request with browser-like User-Agent headers to avoid bot detection
02
HTML Parsing
Used BeautifulSoup to locate the target <table> and extract <th> header tags
03
Data Extraction
Iterated over all <tr> rows, extracted <td> cell values with text.strip()
04
DataFrame Build
Structured data into a Pandas DataFrame with dynamic column names from headers
05
Export
Saved clean structured dataset as a reusable CSV file for further analysis
Data Findings
// TOP 10 BY REVENUE (USD billions)
// INDUSTRY DISTRIBUTION (Top 20)
Healthcare — 35%
Technology — 15%
Oil & Gas — 15%
Retail — 10%
Other — 25%
Scraped Dataset — Top 20
# Company Industry Revenue (USD M) Growth Employees HQ State
Key Insights
$648B
Walmart leads with the highest revenue — 13% more than Amazon in 2nd place
35%
Healthcare dominates the top 20 with 7 companies, more than any other industry
74.4%
Marathon Petroleum achieved the highest revenue growth among the top 20 companies
2.1M
Walmart employs 2.1 million people — over 5× more than Amazon's 1.5M workforce