Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker containerization #355

Open
4 tasks
SiqingYu opened this issue Feb 4, 2020 · 12 comments · May be fixed by #32 or #809
Open
4 tasks

Docker containerization #355

SiqingYu opened this issue Feb 4, 2020 · 12 comments · May be fixed by #32 or #809
Assignees
@SiqingYu
Labels
configuration

Comments

@SiqingYu
Copy link
Contributor

SiqingYu commented Feb 4, 2020 •
edited
Loading

We have a WIP pull request to containerize mygpo for a local development environment - #32

Containerization is a relatively massive undertaking that requires weeks, if not months of effort. We'll take a progressive approach to bring containers onto our production environment.

We will migrate our production environment to a containerized solution before addressing at least the following issues:

  • Local development environment
  • Continuous integration
  • Application migration from vanilla Linux servers to containers
  • Deployment strategy (e.g., rolling updates)

For the moment our server setup on prgmr.com is below:

  • App server for our Nginx, Gunicorn, Django app - 4 vCPUs, 4 GB memory.
  • Database server for our PostgreSQL - 4 vCPUs, 2 GB memory.

The database server is not our primary target for now because we rarely upgrade the database while we regularly deploy the latest master codebase to the app server. Database containerization is much more complicated than stateless web applications. Fortunately, I don't see any direct benefit of a database container for us.

A single app server might not be ideal. As you may know, Kubernetes requires at least three servers for high availability. Meanwhile, lightweight Kubernetes variants focusing on single-node deployment like k8s might be helpful. We might also ask for more resources from prgmr.com or other cloud providers.

For more info about lightweight Kubernetes: K3s, minikube or microk8s? : kubernetes

Why container orchestration?

There are many benefits to enumerate, all of which boil down to smooth application deployment. Our existing infrastructure has many moving parts, so the deployment is error-prone. I have deployed an update last week and caused a half-an-hour outage because of Gunicorn misconfiguration. Not cool; my fault 😔.

Why Kubernetes?

  • I'm not opposed to Docker Swarm and others. I'm inclined to use Kubernetes because I have used Kubernetes but don't know much about Swarm.
  • Although Kubernetes is well-known for its steep learning curve while Swarm is relatively more accessible for new-comers, the main difficulty is on the cluster setup. For development uses the cluster choice is less critical. Kubernetes also has easy-to-use local development variants like Minikube.
  • Kubernetes has good documentation and comes with many out-of-box solutions.
  • There are many managed Kubernetes cloud providers, which is great if we want to relieve the burden of administration and maintenance in the future.
@SiqingYu SiqingYu mentioned this issue Feb 4, 2020
Closed
@SiqingYu SiqingYu self-assigned this Feb 4, 2020
@SiqingYu SiqingYu mentioned this issue Feb 4, 2020
Open
@SiqingYu SiqingYu changed the title Docker Containerization Docker containerization Feb 4, 2020
@SiqingYu SiqingYu linked a pull request Feb 24, 2020 that will close this issue
Open
4 tasks
@SiqingYu
Copy link
Contributor Author

Your advice would be appreciated. 😄
@stefankoegl @JeanFred @morgenroth @fabolhak @elelay

@SiqingYu SiqingYu linked a pull request Feb 24, 2020 that will close this issue
Docker support [WIP] #32
Open
4 tasks
@stefankoegl
Copy link
Member

I think I have a solid understanding of what containerization is, but I have never worked with kubernetes myself. We should keep in mind that we don't make it too complicated to get a test system up and running, to ensure that new contributors can join easily.

More generally, I think gpodder.net's architecture allows for it to be split up into micro services if that's what you're going for. I'm happy to discuss that further (eg on Slack).

@ryoung29
Copy link
Collaborator

I can help with this as well

@SiqingYu
Copy link
Contributor Author

@ryoung29 Thanks. There is a WIP PR that you can follow. #32

@SiqingYu SiqingYu pinned this issue Mar 24, 2020
@zb3
Copy link
Contributor

zb3 commented Apr 12, 2020

I'm currently trying to set up a development environment using docker compose (I'm learning and my goal is to do docker-compose up and have it running).

I saw issue #262 and looked into PR #32, and it appears to me that containerization here is ultimately concerned with the production use (as opposed to #262). The development setup I'm currently trying to create has:

  • DEBUG=true and using manage.py runserver (exceptions are visible and static files are served)
  • source directory mounted as a volume so no need to rebuild the image to test changes
  • test dependencies installed
  • remote debugging support

Since my points above seem to conflict with production use, maybe there could be a separate docker image and compose file for the development setup? If so, I might be able to help with that once I get that running properly.

@oliverkane
Copy link

Hi there. I found this Issue while I was on my way to dockerize this myself since I didn't see anything related to that in the master branch. Glad to see this is in progress. What's the best way for me to connect to folks and see where I could best contribute to this effort? My end goal is to get this into k8s, and I'm pretty familiar with dockerizing apps, though I'm a bit poor at python.

@PascalNoisette
Copy link

Hi,

I came accross the april 2021 fork of @maxolasersquad It is running quite well out of the box and I think it solve #32 and partially this issue (not the k8s part, it's implemented with docker-compose).

@zb3, I am running behind a nginx reverse proxy with DEBUG: "True" the image: maxolasersquad/mygpo so you might be interested as well. (celery must be restarted after postgres though and after migrate, otherwise I get some Relation "django_celery_beat_periodictask" does not exist).

Regards,

@JeanFred JeanFred mentioned this issue Nov 14, 2022
Closed
@nagyv
Copy link

nagyv commented Jan 7, 2023

I created a dockerized version that I use in Kubernetes. It's available at https://gitlab.com/nagyv/gpodder

Please, pick any of my code and contribute it here.

@nagyv nagyv linked a pull request Feb 9, 2023 that will close this issue
Open
@almereyda
Copy link

@nagyv FYI the repository https://gitlab.com/nagyv/gpodder is 404 nowadays. I suspect it is superseded by

@nagyv
Copy link

nagyv commented Feb 10, 2023

@almereyda Correct. the PR includes minimal docs too and it would add all the changes I made to the main mygpo repo.

@albertopasqualetto
Copy link

Is there any update or ETA?

@Ryochan7
Copy link
Contributor

Closest thing you will get would be at https://codeberg.org/atomdmac/mygpo-docker/. It allows running the mygpo project and related dependencies in Docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SiqingYu SiqingYu

Labels
configuration
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Docker support [WIP] gpodder/mygpo
Kubernetes support nagyv/mygpo
10 participants
@nagyv @stefankoegl @Ryochan7 @almereyda @oliverkane @PascalNoisette @zb3 @ryoung29 @SiqingYu @albertopasqualetto