Python Language
- Getting started with Python Language
- Awesome Book
- Awesome Community
- Awesome Course
- Awesome Tutorial
- Awesome YouTube
- *args and **kwargs
- 2to3 tool
- Abstract Base Classes (abc)
- Abstract syntax tree
- Accessing Python source code and bytecode
- Alternatives to switch statement from other languages
- ArcPy
- Arrays
- Asyncio Module
- Attribute Access
- Audio
- Basic Curses with Python
- Basic Input and Output
- Binary Data
- Bitwise Operators
- Boolean Operators
- Call Python from C#
- Checking Path Existence and Permissions
- ChemPy - python package
- Classes
- CLI subcommands with precise help output
- Code blocks, execution frames, and namespaces
- Collections module
- Comments and Documentation
- Common Pitfalls
- Commonwealth Exceptions
- Comparisons
- Complex math
- Conditionals
- configparser
- Connecting Python to SQL Server
- Context Managers (“with” Statement)
- Copying data
- Counting
- Create virtual environment with virtualenvwrapper in windows
- Creating a Windows service using Python
- Creating Python packages
- ctypes
- Data Serialization
- Data Visualization with Python
- Database Access
- Date and Time
- Date Formatting
- Debugging
- Decorators
- Defining functions with list arguments
- Deployment
- Deque Module
- Descriptor
- Design Patterns
- Dictionary
- Difference between Module and Package
- Distribution
- Django
- Dynamic code execution with `exec` and `eval`
- Enum
- Exceptions
- Exponentiation
- Files & Folders I/O
- Filter
- Flask
- Functional Programming in Python
- Functions
- Functools Module
- Garbage Collection
- Generators
- getting start with GZip
- graph-tool
- groupby()
- hashlib
- Heapq
- Hidden Features
- HTML Parsing
- Idioms
- ijson
- Immutable datatypes(int, float, str, tuple and frozensets)
- Importing modules
- Incompatibilities moving from Python 2 to Python 3
- Indentation
- Indexing and Slicing
- Input, Subset and Output External Data Files using Pandas
- Introduction to RabbitMQ using AMQPStorm
- IoT Programming with Python and Raspberry PI
- Iterables and Iterators
- Itertools Module
- JSON Module
- kivy - Cross-platform Python Framework for NUI Development
- Linked List Node
- Linked lists
- List
- List comprehensions
- List Comprehensions
- List destructuring (aka packing and unpacking)
- List slicing (selecting parts of lists)
- Logging
- Loops
- Manipulating XML
- Map Function
- Math Module
- Metaclasses
- Method Overriding
- Mixins
- Multidimensional arrays
- Multiprocessing
- Multithreading
- Mutable vs Immutable (and Hashable) in Python
- Neo4j and Cypher using Py2Neo
- Non-official Python implementations
- Operator module
- Operator Precedence
- Optical Character Recognition
- os.path
- Overloading
- Pandas Transform: Preform operations on groups and concatenate the results
- Parallel computation
- Parsing Command Line arguments
- Partial functions
- Performance optimization
- Pickle data serialisation
- Pillow
- pip: PyPI Package Manager
- Plotting with Matplotlib
- Plugin and Extension Classes
- Polymorphism
- PostgreSQL
- Processes and Threads
- Profiling
- Property Objects
- py.test
- pyaudio
- pyautogui module
- pygame
- Pyglet
- PyInstaller - Distributing Python Code
- Python and Excel
- Python Anti-Patterns
- Python concurrency
- Python Data Types
- Python HTTP Server
- Python Lex-Yacc
- Python Networking
- Python Persistence
- Python Requests Post
- Python Serial Communication (pyserial)
- Python Server Sent Events
- Python speed of program
- Python Virtual Environment - virtualenv
- Queue Module
- Raise Custom Errors / Exceptions
- Random module
- Reading and Writing CSV
- Recursion
- Reduce
- Regular Expressions (Regex)
- Searching
- Secure Shell Connection in Python
- Security and Cryptography
- Set
- setup.py
- shelve
- Similarities in syntax, Differences in meaning: Python vs. JavaScript
- Simple Mathematical Operators
- Sockets
- Sockets And Message Encryption/Decryption Between Client and Server
- Sorting, Minimum and Maximum
- Sqlite3 Module
- Stack
- String Formatting
- String Methods
- String representations of class instances: __str__ and __repr__ methods
- Subprocess Library
- sys
- tempfile NamedTemporaryFile
- Templates in python
- The __name__ special variable
- The base64 Module
- The dis module
- The Interpreter (Command Line Console)
- The locale Module
- The os Module
- The pass statement
- The Print Function
- tkinter
- Tuple
- Turtle Graphics
- Type Hints
- Unicode
- Unicode and bytes
- Unit Testing
- Unzipping Files
- urllib
- Usage of "pip" module: PyPI Package Manager
- User-Defined Methods
- Using loops within functions
- Variable Scope and Binding
- virtual environment with virtualenvwrapper
- Virtual environments
- Web scraping with Python
- Basic example of using requests and lxml to scrape some data
- Maintaining web-scraping session with requests
- Modify Scrapy user agent
- Scraping using BeautifulSoup4
- Scraping using Selenium WebDriver
- Scraping using the Scrapy framework
- Scraping with curl
- Simple web content download with urllib.request
- Web Server Gateway Interface (WSGI)
- Webbrowser Module
- Websockets
- Working around the Global Interpreter Lock (GIL)
- Working with ZIP archives
- Writing extensions
- Writing to CSV from String or List
Python Language Web scraping with Python
Introduction
Web scraping is an automated, programmatic process through which data can be constantly 'scraped' off webpages. Also known as screen scraping or web harvesting, web scraping can provide instant data from any publicly accessible webpage. On some websites, web scraping may be illegal.
Remarks
Useful Python packages for web scraping (alphabetical order)
Making requests and collecting data
requests
A simple, but powerful package for making HTTP requests.
requests-cache
Caching for requests; caching data is very useful. In development, it means you can avoid hitting a site unnecessarily. While running a real collection, it means that if your scraper crashes for some reason (maybe you didn't handle some unusual content on the site...? maybe the site went down...?) you can repeat the collection very quickly from where you left off.
scrapy
Useful for building web crawlers, where you need something more powerful than using requests and iterating through pages.
selenium
Python bindings for Selenium WebDriver, for browser automation. Using requests to make HTTP requests directly is often simpler for retrieving webpages. However, this remains a useful tool when it is not possible to replicate the desired behaviour of a site using requests alone, particularly when JavaScript is required to render elements on a page.
HTML parsing
BeautifulSoup
Query HTML and XML documents, using a number of different parsers (Python's built-in HTML Parser,html5lib, lxml or lxml.html)
lxml
Processes HTML and XML. Can be used to query and select content from HTML documents via CSS selectors and XPath.
Web scraping with Python Related Examples
- Basic example of using requests and lxml to scrape some data
- Maintaining web-scraping session with requests
- Modify Scrapy user agent
- Scraping using BeautifulSoup4
- Scraping using Selenium WebDriver
- Scraping using the Scrapy framework
- Scraping with curl
- Simple web content download with urllib.request
