Skip to content

[qa] Use in-memory DB for migration checks #186 #543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BelpHegoR17 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[qa] Use in-memory DB for migration checks #186 #543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion openwisp-qa-check
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ runcheckpendingmigrations() {
if [ ! -f "./tests/manage.py" ]; then
echo "File manage.py not found, skipping Make Migration Check."
else
OUTPUT=$(python tests/manage.py makemigrations --dry-run $MODULE)
OUTPUT=$(OWQA_USE_IN_MEMORY_DB=1 python tests/manage.py makemigrations --dry-run $MODULE)
echo $OUTPUT | grep "No changes detected" &>/dev/null &&
echo "SUCCESS: Migrations check successful!" ||
{
Expand Down
8 changes: 8 additions & 0 deletions tests/openwisp2/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@
"NAME": "openwisp_utils.db",
}
}

# Force in-memory DB for QA migration checks
if os.environ.get("OWQA_USE_IN_MEMORY_DB"):
Copy link
Member

@nemesifier nemesifier Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd have to do this in all the openwisp modules right @pandafy? Or else how would we accomplish this?

Copy link
Member

@stktyagi stktyagi Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I may.
I think we can avoid duplicating this in every module by adding a helper in openwisp_utils that checks for env var and returns the in-memory config. This way, modules can simply wrap their DATABASES setting with that helper.
If I'm missing something you or @pandafy can let me know.

Copy link
Member

@nemesifier nemesifier Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stktyagi wouldn't we have to edit the settings of every test project in each module anyway?

Copy link
Member

@stktyagi stktyagi Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right the changes would be a little small but we would still need to atleast touch and go through every test settings, this makes me think of maybe using DJANGO_SETTINGS_MODULE to point to a dummy settings module in openwisp-utils which dynamically loads the current module settings into it and then overwrites the DATABASES variable dictionary. This whole process can be skipped either at the start or during variable overwriting using OWQA_USE_IN_MEMORY_DB keeping the modules completely untouched.
kind of like a proxy settings module.
what do you think?

Copy link
Member

@nemesifier nemesifier Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would hardly work because each module has different settings which are required for the module to start, unless we solve #408 first.

Copy link
Member

@stktyagi stktyagi Dec 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nemesifier i may have explained it poorly, what i meant was this proxy wouldn't be a 'replacement' for module settings. Instead, it would use an environment variable (set by the QA tool) to dynamically import * from the module's actual settings.py first (using tests/manage.py we can get where the current settings.py are).

This ensures all mandatory, unique settings are loaded so the module can start. The proxy then only overwrites the DATABASES variable in memory.

i asked chatgpt to write a flow for this:-
QA Tool → Proxy Settings → Imports Real Settings (Module starts successfully) → Swaps Database Variable → Success.

Copy link
Member

@nemesifier nemesifier Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could work, if anyone wants to try to come up with a proof of concept we can evaluate whether the approach is really feasible

Copy link
Author

@BelpHegoR17 BelpHegoR17 Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stktyagi While this could work in theory, Django settings often have side effects during import and some modules modify DATABASES themselves, which makes the proxy approach harder. It may also depend on more uniform settings across modules (as discussed in #408).

DATABASES["default"] = {
"ENGINE": "django.db.backends.sqlite3",
"NAME": ":memory:",
}

if TESTING and "--exclude-tag=selenium_tests" not in sys.argv:
DATABASES["default"]["TEST"] = {
"NAME": os.path.join(BASE_DIR, "openwisp_utils_tests.db"),
Expand Down
Loading