Splitting the Repo (Discussion) please don't delete.

[oblodgett]

I would like to try something different and that is have a discussion under this issue, about splitting the repo into areas of responsibilities, (UI, API, Indexer, Loader).

This way everyone can have well thought out answers and are not totally on the spot in a phone call, please don't delete until after 3/10/2017. The aim for this issue is to come to a conclusion collectively on wether we split or not.

Please state if you are for or against and the reasons.

[oblodgett]

Since I am for splitting this here are the reasons for doing so.

1. Git does does not scale well (as much as we like it).
   a. Number of files
   b. Number of Commits
   c. Number of Refs (Branches, Tags, etc)
   d. Size of files.

As these grow it will get slower, and the .git directory will continually grow. This makes cloning the repo slower. I worked on a repo once that was nearly 1GB in size it took 10+ minutes to clone from github.
Right now agr takes 26 seconds to clone on a (15mb connection) and is 170MB.
Continuous integration clones the repo and checks out HEAD on the branch that it watches, storing all the old versions as history of past builds.
We have already run out of space on the main drive on the dev server. Due to clones into home directories. With CI and developers working on /vol1 its only a matter of time. Even if the decision is made to keep one big repo we will need to remove the data files from the repo and scrub (rewrite) the git history to get that space out of the repo.

2. Areas of responsibility, as this repo grows, there is going to be a lot of code that exists in the areas I have listed. Typically its easier to work on a repo that only has code relevant to what you are working on. Directories are straight forward there is not a bunch of what looks like clutter due to all the other stuff the repo does that does not pertain. We will conceptually be managing four projects in one repo. Splitting gives us repo its own issue tracking (granted some issues will have to be dupped) each repo will have its own wiki. This issue would go away once we get a JIRA or other work management system in place. Right now the README file is getting large and it's not as clear, what you need to do to get everything up and running.

3. Proper project setup, typically when you setup (python, ruby, js, etc) project there is a certain file structure convention of where things are located. We have both (js and python) going on in the same repo. We have a Js application, Flask Application and a python command line application all living in the same space sharing NO code with each other. The convention I see on github is at least projects are split on programming language boundaries.

[adamjohnwright]

I agree with all the points listed above.

I would like to add that I believe that splitting the repos would also make it easier to and somewhat facilitate writing proper documentation. I think it would be relatively clear is a part of the project was not documented correctly if its documentation was not mixed in with the rest of the project. I am sure that it would be possible to write accurate and complete documentation with a single repo as well. I just think that breaking it apart would make it so that it would be clear where to read about the part you are working with. The README was a great example. If we split out the indexer the README would be the clear place to go look to determine how to use the tool.

The second point I would like to add is that the AGR is supposed to be creating reusable code. I think that it would be significantly more likely to have a tool that could be used by another project if we were to split the tools up into their own repos. Someone coming into the AGR organization will likely have a harder time navigating our work if we have it all in on monolithic repo.

These two points aside I can see how a single repo could have its advantages. It would be good for people to list the downsides of splitting as well. This way we can make a fully informed decision.

[cmpich]

I value the pros for splitting the repository. If designed well you could have the UI and the backend piece two independent 'apps'.
My point for keeping it in a single repository is: We still have too much interaction between UI and backend. The so-called API between them, a REST API, is still so much in flux as we add new data types and mature the basic gene detail page. If I make changes to the API I might want to make the UI changes at the same time. That is how we work at ZFIN, we try to keep the complete path, front to end in mind. Everybody is involved in front and back end. Having two repos mean you have to check out twice and check in twice, creating two branches, creating two pull requests, creating two production branches. It doubles those efforts. There are cases when you make changes to the front end that would not require and backend re-built. Yes. But it's harder to keep the two repos in sync. You don't know which version of backend repo goes with which version of UI repo!

In regards to Olin's first point: Here too: If my change affects both front and backend I have to clone both repos and then splitting has not gained anything as you have to download all files including history from two places that make the full application. Once we are in a position where most changes to one does not affect the other you would gain from this split.

Other con: It's harder to write integration tests. Each repo can test it's own functionality but the underlying assumption is the agreed API... But the this API may not be that well defined (including all edge case, such as empty vs null)

A pro is: You are more forced to think about the API with which the UI talks to the backend.

[oblodgett]

Just looking at all commits in the repo I was trying to figure out how much impact this would have from a administrative point of view, with the duplicate entries.

Total commits: 609
Mixed commits: 20

Mixed commits are when there are py files and js or css in the same commit.

dcf5c3b - Travis Sheppard Sep 27, 2016 js => 4 py => 1
221c5b9 - Travis Sheppard Sep 28, 2016 css => 4 js => 13 py => 1
86637ea - Travis Sheppard Sep 28, 2016 js => 1 py => 1
e628e3a - Travis Sheppard Sep 29, 2016 js => 2 py => 1
07df9c0 - Travis Sheppard Oct 24, 2016 js => 10 py => 1
9875168 - Travis Sheppard Oct 25, 2016 js => 1 py => 1
92e3f6c - Travis Sheppard Sep 26, 2016 js => 1 py => 1
881cb31 - Travis Sheppard Oct 27, 2016 css => 1 js => 6 py => 1
1f81832 - Travis Sheppard Oct 28, 2016 js => 1 py => 1
69c5abc - Travis Sheppard Oct 28, 2016 js => 3 py => 1
4fe1117 - Travis Sheppard Oct 31, 2016 js => 1 py => 1
40b63c9 - Travis Sheppard Nov 1, 2016 js => 5 py => 1
f05863c - Travis Sheppard Nov 1, 2016 js => 1 py => 1
2e07668 - ragingsquirrel3 Jan 16, 2017 css => 1 js => 3 py => 1
b0b59b1 - paaatrick 28 days ago css => 1 js => 3 py => 1
4ff1739 - Travis Sheppard 25 days ago js => 8 py => 2
c6eed4e - kevinschaper 20 days ago js => 3 py => 13
c625cf4 - kevinschaper 18 days ago js => 2 py => 2
f229a8e - kevinschaper 6 days ago js => 1 py => 1
3cf731e - kevinschaper 5 days ago js => 3 py => 3

Most of Travis's commits were during the prototype phase, and only involved server.py.
Kevin's commits included mapping.py which effectively is our API at the moment.

[paaatrick]

My two thoughts:

1. I can see (I think) how this simplifies the backend development workflow. You run your Flask application, request your API endpoints, make sure the JSON response looks good, call it a day. No messing around with npm or javascript nonsense. But, what does this world look like for a frontend developer? Do I need to check out the backend repo and run it? Am I somehow pointing my local development site to a shared backend? What version is it running and is it always available? (To be clear, I'm not looking for answers to these questions here; I'm looking for the person(s) splitting the repo to solve these problems and document the solution.)

2. This has already been briefly mentioned, but let me put a finer point on it: I can't see how we can keep using GitHub for issue tracking with separate repos. Sure, developers will have a pretty good idea of which repo to put issues in. But even still there will be mistakes, or issues that at first look like frontend issues that turn out to be backend or vice-versa. And then there are curators! How would they ever know which repo to use for issues?

[oblodgett]

More perspective... coming from a Java / JBoss server environment I didn't understand Flask, I couldn't just do the things I was accustomed to with JBoss. I tried to understand Flask in terms of JBoss. And here is how I would explain it.

It would be like building 1 war file with Jboss and your code inside it. You would start JBoss by java -jar myapp.war. If you wanted to make a change to the code you would have to stop the running war, rebuild the war, and start it again via java -jar myapp.war. The war being the REST endpoints and connections to DB and ES.

From a frontend developer perspective, I rebuild my js file, into the directory that is being served by the running Flask server, and hit refresh in the browser and it works just like deploying a WAR file to JBoss server. And I agree with @paaatrick splitting out the repo might make this process more cumbersome for the frontend devs, and having a good process and documentation should make development be no different then it is now.

Now on the backend side of things Flask does watch some of the code directories and will reload when it sees changes but if there is an error, the server just crashes aka stops running when in development mode. When in production mode, the errors just go to the browser or to log files and the server keeps running.

Yesterday however the code changed under the running server it hung and the kill process for whatever reason did not kill the running server. Which then the start process fails because there is already a server running on that port. If this was in the JBoss realm the server sits outside of the CI process and just acts as a resource for deployed code. Whereas the Flask server IS the deployed code. JBoss is an application server that can serve many applications. Whereas Flask is the application that happens to start a server.

MGI has our Flask install split into 3 repos. Flask and the server code / API pages etc. The models and libraries repo (DB connector API) so other python projects in the future could make use of. JS repo for all the JavaScript that is used on the site. Loads are totally independent repos to this stack.

JS <- Flask Server <- DB Connector / Models <- Database

In our case when the Models change the server has to be restarted so those two repos (Server / Modes) really should be one.

I'm sorry if this information is already known by the group, I had to try to wrap my head around it, when we started using Flask at MGI, just trying to share what I learned.

[christabone]

Also just wanted to chime in regarding issues/tickets real quick.

There has been talk over an over-arching issue tracking system (a la JIRA) to be implemented on top of GitHub and everything else. We could direct curators / PIs / anyone to submit tickets which would then be associated with GitHub issues by developers. In terms of project management, it feels like a necessity at some point (and I believe this is the direction we're heading with AGR? Please feel free to correct me if I'm mistaken).

[LucyHut]

The three points Olin listed in favor to splitting the agr repos into areas of responsibilities are valid and straight forward but a little bit simplistic for frontend developers point of view. As a frontend developer, splitting agr repos into multiple repos will add another layer of complexity into the process - making it harder and harder to setup a fully functional instance of agr in dev environment.

Patrik already made some good points on how life would be like for frontend developers if we were to adopt the multiple repos solution – I would add that even if I were to clone all these repos in my development environment, I could still be faced with the overhead of software dependencies and version conflicts.

To Olin’s first point on running out of space as the files number/size grow – it looks like the size of the repos grows more locally than remotely– since the index files are created locally to the dev/prod server – we could still have that issue even if we were running multiple repos - As for the reduced speed of cloning a large repos, github also gives developers the option to clone a single branch within a repos – so you don’t have to clone the entire repos.

[stuartmiyasato]

There has been talk over an over-arching issue tracking system (a la JIRA) to be implemented on top of GitHub and everything else. We could direct curators / PIs / anyone to submit tickets which would then be associated with GitHub issues by developers. In terms of project management, it feels like a necessity at some point (and I believe this is the direction we're heading with AGR? Please feel free to correct me if I'm mistaken).

Just wanted to tag @kltm and see if he has any comments about this, given that the GO project has just done the exact opposite -- they are dropping JIRA to move exclusively to GitHub for tracking! (Now speaking in the third person since as of today I am no longer officially affiliated with GO...) :)

[kltm]

Also tagging @cmungall .
We tried this (had this tried with JIRA) in another project, and it did not go very well; moreover, nobody used it. One of the issues is the sync and lag between the two. Having a single definitive resource is very useful, and if that ended up being JIRA, it reather defeats the purpose of using GitHub.
If people are interested in higher-level management, it might be worth really exploring what GitHub projects offers, or the use of more "overlay" type systems like waffle.io.
What exactly is the use cases here, except to have more "management"?

[oblodgett]

This topic is kinda going in different directions but I wonder what the GO group is for using JIRA and if JIRA lends better to larger or smaller groups and if so how does AGR compare considering there are a lot of people involved.

[kltm]

Well, the GO group has experimented with JIRA at least two times that I participated in.

The first, and most successful, instance was as an email controlled issue tracker for the GO Helpdesk. The interface was a bit clunky and hard to script, but it did it's job very well (besides clunkiness), with the only downsides being maintenance and overhead.

The other instance was run as an internal management tool, as an attempt to coordinate and manage projects at a higher level. This ended up being...not a good match. There was also the issue of coordinating between the actual issues (say GitHub or Trello) and JIRA.

There is also a third JIRA experience with a non-GO project, where it was to be used as a "managerial wrapper" around GitHub. While it sorta worked, there was little interest in following through due to the reasons above #217 (comment) and a number of edges that were problematic in sanding off.

I don't think it's so much an issue of a large or small group, but more about management style and needs. I also think that at the end of the day there needs to be a single definitive source for actions and history, no matter what is chosen. For us, GitHub is working quite well as both a management and VC system. Also, in my experience, every time there have been multiple ticketing/management systems that are used for different things within an org, there ends up being confusion and dangling ends. YMMV :)