[PYTHON] Why django-import-export import is so slow and what to do

https://github.com/django-import-export/django-import-export https://django-import-export.readthedocs.org/en/latest/index.html

It's django-import-export, which is so heavy that it can't be helped just by trying to import CSV with about 2000 lines at most, but it seems that a large amount of DB access is taken.

solution

However, if you just reduce the access to the DB, it will be faster. Sample for the time being.

from import_export.admin import ImportExportModelAdmin
from import_export import fields, resources, widgets
from import_export.instance_loaders import CachedInstanceLoader

from .models import Foo


class FooResource(resources.ModelResource):
    related_item = fields.Field(
        column_name='related_item_id',
        attribute='related_item_id',
        widget=widgets.IntegerWidget(),
    )

    class Meta:
        model = Foo
        skip_unchanged = True
        instance_loader_class = CachedInstanceLoader


class FooAdmin(ImportExportModelAdmin):
	resource_class = FooResource
	skip_admin_log = True

Do not use ForeignKeyWidget

If the model fields are defined with ForeignKey, the ImportExport library will now utilize widgets.ForeignKeyWidget.

https://github.com/django-import-export/django-import-export/blob/0.4.5/import_export/resources.py#L671

Since this guy is accessing the DB at the time of import, it will be faster if you replace it with widgets.IntegerWidget.

https://github.com/django-import-export/django-import-export/blob/0.4.5/import_export/widgets.py#L272

Use CachedInstanceLoader

At the time of import, the DB is accessed with the value of PrimaryKey on each line.

https://github.com/django-import-export/django-import-export/blob/0.4.5/import_export/instance_loaders.py#L33

So, if you use CachedInstanceLoader, which will read everything first and then do it, you will only have to access the DB once, which is nice.

Ignore unchanged data (use skip_unchange = True)

If you write skip_unchange = True so that unnecessary UPDATE statements are not executed, the update will be a little faster.

https://github.com/django-import-export/django-import-export/blob/0.4.5/import_export/resources.py#L427

Do not create change log of management screen (use skip_admin_log = True)

It will make a lot of change logs carefully one by one, so please politely decline with skip_admin_log = True.

https://github.com/django-import-export/django-import-export/blob/0.4.5/import_export/admin.py#L163