Wikipedia_Scraper

Wikipedia Scraper

Description

Extract and display information from Wikipedia articles using the Wikipedia API and web scraping. Search topics, get summaries, full content, infoboxes, and images.

Features

• Search Wikipedia topics
• Get article summaries
• Extract full article content
• Parse infobox data
• Get article images
• Handle disambiguation pages
• Related links extraction
• References listing
• Clean CLI interface

Stack

• Language: Python
• Libraries:
  • wikipedia==1.4.0 (Wikipedia API)
  • beautifulsoup4==4.12.2 (HTML parsing)
  • requests==2.31.0 (HTTP requests)
• Complexity: Intermediate

Installation

pip install -r requirements.txt
python main.py

Features Explained

1. Search Topics

• Search Wikipedia for topics
• Returns top 5 results
• Handles typos and suggestions

2. Article Summary

• Get concise summary (3 sentences)
• Quick overview of topic
• Handles disambiguation

3. Full Article

• Complete article content
• Related links (first 20)
• References (first 10)
• Article URL

4. Infobox Data

• Structured data extraction
• Key facts and figures
• Formatted display

5. Article Images

• List all images from article
• Direct image URLs
• First 10 images shown

How to Use

1. Search topics → Find articles
2. Get article summary → Quick info
3. Get full article → Complete content
4. Get infobox data → Key facts
5. Get article images → Visual content
6. Exit → Close program

Example Usage

Search Topics

Enter search query: Python programming

SEARCH RESULTS
1. Python (programming language)
2. History of Python
3. Python Software Foundation
4. CPython
5. Python syntax

Get Summary

Enter topic name: Python (programming language)

PYTHON (PROGRAMMING LANGUAGE)
Python is a high-level, interpreted programming language. 
It was created by Guido van Rossum and first released in 1991. 
Python emphasizes code readability with significant indentation.

Get Infobox

PYTHON (PROGRAMMING LANGUAGE) - QUICK FACTS

Designer:
  Guido van Rossum

First appeared:
  1991

Stable release:
  3.12.0

Typing discipline:
  Duck, dynamic, strong

License:
  Python Software Foundation License

Learning Outcomes

• Wikipedia API usage
• Web scraping with BeautifulSoup
• HTML parsing
• Data extraction
• Error handling
• CLI application design
• API integration

Use Cases

• Research: Quick information gathering
• Education: Learning tool
• Content Creation: Fact checking
• Data Collection: Wikipedia data extraction
• Reference: Quick lookups

API vs Scraping

Wikipedia API (wikipedia library)

• ✅ Fast and reliable
• ✅ Structured data
• ✅ No rate limiting issues
• ✅ Easy to use

Web Scraping (BeautifulSoup)

• ✅ Access to infobox data
• ✅ Custom data extraction
• ⚠️ May break if HTML changes

Error Handling

Disambiguation Pages

⚠️ 'Python' is ambiguous. Did you mean:
  1. Python (programming language)
  2. Python (genus)
  3. Pythonidae

Page Not Found

❌ Page 'XYZ123' does not exist.

Future Enhancements

• Save articles to file
• Export to PDF
• Multiple language support
• Category browsing
• Random article feature
• Article comparison
• Citation extraction
• Table data extraction
• Image downloading
• Batch processing
• GUI version

Tips

• Use specific topic names
• Check disambiguation options
• Use quotes for exact matches
• Try different search terms
• Explore related links

Limitations

• Requires internet connection
• Wikipedia API rate limits
• Some data may be incomplete
• Infobox format varies
• Large articles truncated in display

Advanced Usage

Custom Summary Length

summary = scraper.get_summary(topic, sentences=5)

More Search Results

results = scraper.search_topic(query, results=10)

Language Selection

wikipedia.set_lang("es")  # Spanish
wikipedia.set_lang("fr")  # French

License

Open source for educational purposes.