Bug report: yaw optimization verify_convergence fails for large farms and many wind directions/speeds

[Bartdoekemeijer]

Bug description
A memory issue arises with verify_convergence=True when optimizing yaw angles for large farms and many wind directions/speeds. This is because it condenses everything to a single calculate_wake call which can speed things up, but also raises the risk of exceeding memory limitations.

To Reproduce

Expected behavior
Run without memory issues.

Screenshots, if applicable
N/A

Floris Version
FLORIS 3.x

System Information (please complete the following information):
N/A

Additional context
N/A

[Bartdoekemeijer]

I have had some more thoughts on this topic. Essentially, the problem in this issue is that the problem set (dimension space) for floris is too large, and that raises the memory issue. Anyone can trigger such a problem simply by making the number of wind directions and speeds to evaluate in FLORIS very large.

Rather than forcing the user to reduce the problem size, FLORIS could internally split up the problem into multiple calls. As in, rather than evaluating all 360 wind directions in a single shot, you evaluate 120 wind directions at a time. This is pretty much exactly what I have done for code parallelization in #403. I am rolling back on some of the vectorization in benefit of memory management.

Also, the benchmark results in #403 shows that there's a sweet spot in terms of computation time. As the problem becomes bigger, memory management becomes more important and it may actually then be faster to split the problem up into multiple calculate_wake calls which each also require less memory. The sweet spot for into how many calls one must split up calculate_wake will depend on the number of turbines in the farm. My feeling is that the effect on computation time can be really significant. I would not be surprised if you see a 50% reduction in computation time when large problems are split up in the right number of separate calculate_wake calls, just based on running AEP calculations and yaw optimizations in commercial projects.

I'm sidetracking this issue, but would it be valuable to create some sort of table that helps the user select how a problem is best split up for computation time? This could be a pretty easy empirical study. I think essentially you'd want to run the grid:

Dimension 1: Number of wind conditions (n_ws x n_wd)
Dimension 2: Wind farm size
Dimension 3: Memory available on system (?)
Output: computation time

I'd love to hear what @rafmudaf and @bayc think about this.

[paulf81]

Talking to @Bartdoekemeijer I'm persuaded this is a sensible approach, but will require more thinking, and moves the action here over to #403 , but if that was in place, then this issue would be resolved by parallelizing the very large problem verify_convergence sets up when there are a larger number of turbines and wind directions