Python Language
- Getting started with Python Language
- Awesome Book
- Awesome Community
- Awesome Course
- Awesome Tutorial
- Awesome YouTube
- *args and **kwargs
- 2to3 tool
- Abstract Base Classes (abc)
- Abstract syntax tree
- Accessing Python source code and bytecode
- Alternatives to switch statement from other languages
- ArcPy
- Arrays
- Asyncio Module
- Attribute Access
- Audio
- Basic Curses with Python
- Basic Input and Output
- Binary Data
- Bitwise Operators
- Boolean Operators
- Call Python from C#
- Checking Path Existence and Permissions
- ChemPy - python package
- Classes
- CLI subcommands with precise help output
- Code blocks, execution frames, and namespaces
- Collections module
- Comments and Documentation
- Common Pitfalls
- Commonwealth Exceptions
- Comparisons
- Complex math
- Conditionals
- configparser
- Connecting Python to SQL Server
- Context Managers (“with” Statement)
- Copying data
- Counting
- Create virtual environment with virtualenvwrapper in windows
- Creating a Windows service using Python
- Creating Python packages
- ctypes
- Data Serialization
- Data Visualization with Python
- Database Access
- Date and Time
- Date Formatting
- Debugging
- Decorators
- Defining functions with list arguments
- Deployment
- Deque Module
- Descriptor
- Design Patterns
- Dictionary
- Difference between Module and Package
- Distribution
- Django
- Dynamic code execution with `exec` and `eval`
- Enum
- Exceptions
- Exponentiation
- Files & Folders I/O
- Filter
- Flask
- Functional Programming in Python
- Functions
- Functools Module
- Garbage Collection
- Generators
- getting start with GZip
- graph-tool
- groupby()
- hashlib
- Heapq
- Hidden Features
- HTML Parsing
- Idioms
- ijson
- Immutable datatypes(int, float, str, tuple and frozensets)
- Importing modules
- Incompatibilities moving from Python 2 to Python 3
- Indentation
- Indexing and Slicing
- Input, Subset and Output External Data Files using Pandas
- Introduction to RabbitMQ using AMQPStorm
- IoT Programming with Python and Raspberry PI
- Iterables and Iterators
- Itertools Module
- JSON Module
- kivy - Cross-platform Python Framework for NUI Development
- Linked List Node
- Linked lists
- List
- List comprehensions
- List Comprehensions
- List destructuring (aka packing and unpacking)
- List slicing (selecting parts of lists)
- Logging
- Loops
- Manipulating XML
- Map Function
- Math Module
- Metaclasses
- Method Overriding
- Mixins
- Multidimensional arrays
- Multiprocessing
- Multithreading
- Mutable vs Immutable (and Hashable) in Python
- Neo4j and Cypher using Py2Neo
- Non-official Python implementations
- Operator module
- Operator Precedence
- Optical Character Recognition
- os.path
- Overloading
- Pandas Transform: Preform operations on groups and concatenate the results
- Parallel computation
- Parsing Command Line arguments
- Partial functions
- Performance optimization
- Pickle data serialisation
- Pillow
- pip: PyPI Package Manager
- Plotting with Matplotlib
- Plugin and Extension Classes
- Polymorphism
- PostgreSQL
- Processes and Threads
- Profiling
- Property Objects
- py.test
- pyaudio
- pyautogui module
- pygame
- Pyglet
- PyInstaller - Distributing Python Code
- Python and Excel
- Python Anti-Patterns
- Python concurrency
- Python Data Types
- Python HTTP Server
- Python Lex-Yacc
- Python Networking
- Python Persistence
- Python Requests Post
- Python Serial Communication (pyserial)
- Python Server Sent Events
- Python speed of program
- Python Virtual Environment - virtualenv
- Queue Module
- Raise Custom Errors / Exceptions
- Random module
- Reading and Writing CSV
- Recursion
- Reduce
- Regular Expressions (Regex)
- Searching
- Secure Shell Connection in Python
- Security and Cryptography
- Set
- setup.py
- shelve
- Similarities in syntax, Differences in meaning: Python vs. JavaScript
- Simple Mathematical Operators
- Sockets
- Sockets And Message Encryption/Decryption Between Client and Server
- Sorting, Minimum and Maximum
- Sqlite3 Module
- Stack
- String Formatting
- String Methods
- String representations of class instances: __str__ and __repr__ methods
- Subprocess Library
- sys
- tempfile NamedTemporaryFile
- Templates in python
- The __name__ special variable
- The base64 Module
- The dis module
- The Interpreter (Command Line Console)
- The locale Module
- The os Module
- The pass statement
- The Print Function
- tkinter
- Tuple
- Turtle Graphics
- Type Hints
- Unicode
- Unicode and bytes
- Unit Testing
- Unzipping Files
- urllib
- Usage of "pip" module: PyPI Package Manager
- User-Defined Methods
- Using loops within functions
- Variable Scope and Binding
- virtual environment with virtualenvwrapper
- Virtual environments
- Web scraping with Python
- Basic example of using requests and lxml to scrape some data
- Maintaining web-scraping session with requests
- Modify Scrapy user agent
- Scraping using BeautifulSoup4
- Scraping using Selenium WebDriver
- Scraping using the Scrapy framework
- Scraping with curl
- Simple web content download with urllib.request
- Web Server Gateway Interface (WSGI)
- Webbrowser Module
- Websockets
- Working around the Global Interpreter Lock (GIL)
- Working with ZIP archives
- Writing extensions
- Writing to CSV from String or List
Python Language Web scraping with Python Modify Scrapy user agent
Example
Sometimes the default Scrapy user agent ("Scrapy/VERSION (+http://scrapy.org)") is blocked by the host. To change the default user agent open settings.py, uncomment and edit the following line to what ever you want.
#USER_AGENT = 'projectName (+http://www.yourdomain.com)'
For example
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
