[bug] WHOIS run out of queries for the GeoIP2 service and spamming users

[nemesifier]

Describe the bug
Follow up of #1032 #1033 #1037 #1045 #1058.

With @pandafy we have noticed the following issues during testing:

• WHOIS lookup failed. Details: Your account has run out of queries for the GeoIP2 service.
  Probably triggered by the refresh logic.
• When we run out of geoip queries, we start spamming users with notifications:

Steps To Reproduce
I am not sure yet, more investigation is needed to reproduce this.

Expected behavior

1. We do not expect to run out of geoip queries so quickly, we suspect there's a bug which is causing too many requests to be made.
2. If requests run out, we must not spam users, especially because old whois data is still present, which will be in most situations still valid. We should instead just log the error, so that system administrators should get alerted by error monitoring systems like Sentry that this error is popping up a lot.

Screenshots

[pandafy]

This task should not be retried when the we run out of queries

openwisp-controller/openwisp_controller/config/whois/tasks.py

Lines 35 to 40 in 9fd2bb1

	@shared_task(
	bind=True,
	base=WHOISCeleryRetryTask,
	**app_settings.API_TASK_RETRY_OPTIONS,
	)
	def fetch_whois_details(self, device_pk, initial_ip_address):

[DragnEmperor]

This task should not be retried when the we run out of queries

openwisp-controller/openwisp_controller/config/whois/tasks.py

Lines 35 to 40 in 9fd2bb1

@shared_task(
bind=True,
base=WHOISCeleryRetryTask,
**app_settings.API_TASK_RETRY_OPTIONS,
)
def fetch_whois_details(self, device_pk, initial_ip_address):

Autoretry is enabled only for HTTPError.

openwisp-controller/openwisp_controller/config/whois/tasks.py

Line 24 in 9fd2bb1

autoretry_for = (errors.HTTPError,)

Will investigate this

[nemesifier]

Disclaimer: Analysis performed by Claude Haiku 4.5, please double check.

Root Cause Analysis

Issue 1: OutOfQueriesError triggers Celery retry mechanism

File: openwisp_controller/config/whois/tasks.py (lines 20-40)

The WHOISCeleryRetryTask class only includes HTTPError in autoretry_for, but when OutOfQueriesError is raised, it's not explicitly excluded. This means Celery still retries the task up to 5 times (API_TASK_RETRY_OPTIONS default max_retries=5) because the exception is not in the rejection list.

Issue 2: on_failure sends notifications for every attempt

File: openwisp_controller/config/whois/tasks.py (lines 27-30)

The on_failure method sends user notifications for every failure. When a task is retried 5 times, this generates 5 notifications, "spamming" users.

Issue 3: No explicit handling for non-retriable errors

File: openwisp_controller/config/whois/service.py (lines 160-178).

While the service raises OutOfQueriesError properly, Celery doesn't know this is non-retriable. When queries run out, retrying won't help - we'll keep failing and keep creating notifications.

---

Proposed Solution

Add do_not_retry_for to exclude permanent errors from retries, and conditionally send notifications only for temporary errors.

Change 1: Update WHOISCeleryRetryTask in openwisp_controller/config/whois/tasks.py

Replace this block (lines 20-30):

class WHOISCeleryRetryTask(OpenwispCeleryTask):
    """
    Base class for OpenWISP Celery tasks with retry support on failure.
    """

    # this is the exception related to networking errors
    # that should trigger a retry of the task.
    autoretry_for = (errors.HTTPError,)

    def on_failure(self, exc, task_id, args, kwargs, einfo):
        """Notify the user about the failure of the WHOIS task."""
        device_pk = kwargs.get("device_pk")
        send_whois_task_notification(device=device_pk, notify_type="whois_device_error")
        logger.error(f"WHOIS lookup failed. Details: {exc}")
        return super().on_failure(exc, task_id, args, kwargs, einfo)

With:

class WHOISCeleryRetryTask(OpenwispCeleryTask):
    """
    Base class for OpenWISP Celery tasks with retry support on failure.
    """

    # this is the exception related to networking errors
    # that should trigger a retry of the task.
    autoretry_for = (errors.HTTPError,)
    # Don't retry on out of queries or other permanent errors
    do_not_retry_for = (
        errors.OutOfQueriesError,
        errors.AddressNotFoundError,
        errors.AuthenticationError,
        errors.PermissionRequiredError,
    )

    def on_failure(self, exc, task_id, args, kwargs, einfo):
        """Notify the user about the failure of the WHOIS task."""
        device_pk = kwargs.get("device_pk")
        # Only send notifications for temporary/retriable errors
        # For permanent errors like OutOfQueriesError, only log them
        # so system administrators can see them in error monitoring (Sentry)
        if not isinstance(exc, errors.OutOfQueriesError):
            send_whois_task_notification(device=device_pk, notify_type="whois_device_error")
        logger.error(f"WHOIS lookup failed. Details: {exc}")
        return super().on_failure(exc, task_id, args, kwargs, einfo)

---

What This Fixes

1. Prevents query quota waste: No more retrying when queries are exhausted
2. Eliminates notification spam: Users get notified only once per device (or not at all for quota errors)
3. Better error visibility: System admins see quota issues in error logs/Sentry for proper alerting
4. Graceful degradation: Old WHOIS data remains valid and in use while waiting for quota refresh

---

Why The Bug Occurred

The issue was likely triggered by:

• Multiple devices refreshing WHOIS data simultaneously on the same schedule
• The 14-day refresh threshold causing periodic bulk refreshes
• No deduplication or rate-limiting on WHOIS queries across the entire system
• Celery retrying on exceptions not in autoretry_for instead of failing immediately

[codesankalp]

I also think, we should first address all the issues mentioned above to resolve the out of queries error.
​
As a future enhancement, we could also consider using a WHOIS database, which would be faster and will not give above error: #1052