Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to reconnect on error without dropping messages? #9

Open
Commod0re opened this issue Jan 22, 2025 · 0 comments
Open

how to reconnect on error without dropping messages? #9

Commod0re opened this issue Jan 22, 2025 · 0 comments

Comments

@Commod0re
Copy link

Hello! I recently started using aio_statsd to log some telemetry to a datadog agent using DogStatsdClient from a web application backend (python asyncio, using Quart). For the most part it works great but once in a while one worker process will have some kind of issue and lose its connection and I'm trying to figure out what the best way to recover in this scenario is

to start with I'm playing with a subclass like this:

class ReconnectingDogStatsdClient(DogStatsdClient):
    async def _listen(self):
        while True:
            # this is nearly a copy of DogStatsdClient._listen
            try:
                while not self.connection.is_closed:
                    value = self._get_by_queue()
                    if value:
                        await self._real_send(value)
                    else:
                        await asyncio.sleep(0.05)

                    if self.connection.future.done():
                        await self.connection.future
            except asyncio.CancelledError:
                break
            except Exception as e:
                log.error(f"status:{self.is_closed=} {self.connection.is_closed=} error: {e}")
            # try to reconnect
            if self.connection.is_closed:
                await self.connection.connect()

now it doesn't quite feel right to override _listen like this, but I don't see a cleaner way to implement reconnection that also doesn't drop any (or as few as reasonably possible) messages so I'm now here asking for any advice or guidance on this you might be able to offer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant