scrapy Item Pipeline Creating a dynamic pipeline in Python Scrapy

Example

Enable pipelines in your settings.py

ITEM_PIPELINES = {
    'project_folder.pipelines.MyPipeline': 100 
}

Then write this code in items.py

# -*- coding: utf-8 -*-
from scrapy import Item, Field
from collections import OrderedDict

class DynamicItem(Item):
    def __setitem__(self, key, value):
        self._values[key] = value
        self.fields[key] = {}

Then in your `project_folder/spiders/spider_file.py

from project_folder.items import DynamicItem
       def parse(self, response):
               # create an ordered dictionary
               data = OrderedDict()
               data['first'] = ...
               data['second'] = ...
               data['third'] = ...
               .
               .
               .
               # create dictionary as long as you need
               
               # now unpack dictionary
               yield DynamicItem( **data )

               # above line is same as this line
               yield DynamicItem( first = data['first'], second = data['second'], third = data['third'])

What are benefits of this code?

No need to create define each item in items.py one by one.