Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cachalot breaks loading fixtures with no PK #214

Open
Finndersen opened this issue Mar 9, 2022 · 14 comments
Open

Cachalot breaks loading fixtures with no PK #214

Finndersen opened this issue Mar 9, 2022 · 14 comments
Labels

Comments

@Finndersen
Copy link

Cachalot appears to impact fixture loading (for test cases) somehow, I think perhaps for fixtures with no PK specified, or with "pk": null.

Fixture loading works as expected with cachalot disabled.

python3.9, django 3.1.7

@Andrew-Chen-Wang
Copy link
Collaborator

I'm not sure what you mean by fixture loading. Are you talking about pytest with factory boy? Please provide a reproducible example in addition to the version of cachalot's you're using. Thanks!

@Finndersen
Copy link
Author

Just loading fixtures during standard Django TestCases, or using loaddata management command.

I will need some time to try create a minimum reproducible example (will be away for the next week).
But it appears to be only happening when loading multiple fixture files at once, loading a single file directly appears to work.

It appears to be related to fixtures with FK fields specifying the related model using its Natural Key value (not integer PK)
The related model is not being loaded and so the lookup using the Natural Key is failing, for example this occurs when loading a User model fixture:

django.core.serializers.base.M2MDeserializationError: (DoesNotExist('Group matching query does not exist.'), ['MY_GROUP_NAME'])

However the MY_GROUP_NAME group model is defined in the same fixture file before User model is, and loading just that single fixture file by itself appears to work fine...

@Finndersen
Copy link
Author

I'm using cachalot 2.5.1

@Finndersen
Copy link
Author

Note i am also using diskcache.DjangoCache backend, and updating SUPPORTED_CACHE_BACKENDS to add it in to be allowed

@Andrew-Chen-Wang
Copy link
Collaborator

Andrew-Chen-Wang commented Mar 9, 2022

Interesting. An example isn't necessary. Can you post the full stack trace? If cachalot enabled is causing this, then it might have something to do with us loading your models. Have you tried placing cachalot in your INSTALLED_APPS at the very bottom?

@Finndersen
Copy link
Author

Finndersen commented Mar 15, 2022

Still occurs with cachalot at the bottom of INSTALLED_APPS. Here's the stack trace from running a custom management command which just calls loaddata command for every fixture of every app:

Traceback (most recent call last):
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 292, in deserialize_m2m_values
    values.append(m2m_convert(pk))
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 278, in m2m_convert
    return model._default_manager.db_manager(using).get_by_natural_key(*value).pk
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\contrib\auth\models.py", line 87, in get_by_natural_key
    return self.get(name=name)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\query.py", line 429, in get
    raise self.model.DoesNotExist(
django.contrib.auth.models.DoesNotExist: Group matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 239, in save_deferred_fields
    values = deserialize_m2m_values(field, field_value, using, handle_forward_references=False)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 298, in deserialize_m2m_values
    raise M2MDeserializationError(e, pk)
django.core.serializers.base.M2MDeserializationError: (DoesNotExist('Group matching query does not exist.'), ['GROUP_NAME'])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\USERNAME\PycharmProjects\Project\project\core\manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "C:\Users\USERNAME\PycharmProjects\Project\project\core\management\management_command.py", line 34, in handle
    self.run(*args, **options)
  File "C:\Users\USERNAME\PycharmProjects\Project\project\core\management\commands\initialise_local_db.py", line 34, in run
    load_all_app_fixtures()
  File "C:\Users\USERNAME\PycharmProjects\Project\django_unchained\utils\misc.py", line 367, in load_all_app_fixtures
    call_command('loaddata', fixture, database=db, verbosity=verbosity)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 116, in loaddata
    obj.save_deferred_fields(using=self.using)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 241, in save_deferred_fields
    raise DeserializationError.WithData(e.original_exc, label, self.object.pk, e.pk)
django.core.serializers.base.DeserializationError: Group matching query does not exist.: (my_app.customuser:pk=13) field_value was '['GROUP_NAME']'

It appears its due to the Group fixture entry with name 'GROUP_NAME' not being loaded, but works fine with cachalot disabled. I think it's due to the fixture being defined with pk=null, because setting the pk to a valid number fixes it (but I don't want to define PKs for fixtures which can be identified with Natural Keys)

@Finndersen
Copy link
Author

Finndersen commented Mar 15, 2022

Looks like the issue can be recreated by trying to load a fixture which has a model with Natural Key support and no PK specified, and another model with a FK field to the first model, using Natural Key values to refer to the first fixture model. if PK is specified on the first model then it works fine. So it's something about not loading fixtures with no PK properly.
Here's stack trace from loaddata command:

Traceback (most recent call last):
  File "C:\Users\USER\PycharmProjects\Project\project\core\manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 181, in load_label
    obj.save(using=self.using)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\base.py", line 790, in save_base
    updated = self._save_table(
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\base.py", line 933, in _do_insert
    return manager._insert(
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\cachalot\monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\cachalot\monkey_patch.py", line 113, in inner
    return original(write_compiler, *args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\sql\compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 98, in execute
    return super().execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\cachalot\monkey_patch.py", line 137, in inner
    return original(cursor, sql, *args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\sqlite3\base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: Problem installing fixture 'C:\Users\USER\PycharmProjects\Project\project\apps\my_app\fixtures\FixtureName.json': Could not load my_app.CustomUser(pk=None): NOT NULL constraint failed: my_app_customuser.foreign_key_field_id

@Finndersen
Copy link
Author

@Andrew-Chen-Wang any luck with this?

@Andrew-Chen-Wang
Copy link
Collaborator

Hi @Finndersen i've been swamped with tests and work this last month, so i've been unable to find a time to try and resolve this. I'm not really sure if I'll be able to in the near future either, but this is still on my mind. Apologies in advance.

@Finndersen
Copy link
Author

Finndersen commented Apr 11, 2022 via email

@Andrew-Chen-Wang
Copy link
Collaborator

Andrew-Chen-Wang commented Apr 11, 2022

One thing to note is whether you have the model manager set up correctly (I'm sure you do, but just double checking): https://stackoverflow.com/a/25493071

https://docs.djangoproject.com/en/4.0/topics/serialization/ (model manager portion)

@Finndersen
Copy link
Author

I'm using django-natural-keys which should be managing all that fine

@Finndersen
Copy link
Author

Finndersen commented Oct 11, 2022 via email

@milos-u
Copy link

milos-u commented Feb 2, 2023

Maybe this is related: #224

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants