fenics-DRL :

Repository from the paper A review on Deep Reinforcement Learning for Fluid Mechanics.

List of other repo. with CFD + DRL (if your code is not here, please feel free to make a pull request)

(Explanatations on how to use the code are below)

Fenis + DRL	https://github.com/DonsetPG/fenics-DRL
Flow Control of the 2D Kármán Vortex Street with Deep Reinforcement Learning	https://github.com/jerabaul29/Cylinder2DFlowControlDRL
Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach	https://github.com/jerabaul29/Cylinder2DFlowControlDRLParallel
Deep Reinforcement Learning control of the unstable falling liquid film	https://github.com/vbelus/falling-liquid-film-drl
Direct shape optimization through deep reinforcement learning	https://github.com/jviquerat/drl_shape_optimization
Fluid directed rigid ball balancing using Deep Reinforcement Learning	https://github.com/sahilgupta2105/Deep-Reinforcement-Learning
Efficient collective swimming by harnessing vortices through deep reinforcement learning	https://github.com/cselab/smarties
Training an RL agent to swim at low Reynolds Number	https://github.com/RpDp-git/LearningToSwim-DQN

How to use the code :

Install everything :

CFD :

We used Fenics for this project. The easiest way to install it is by using Docker. Then :

docker run -ti -v $(pwd):/home/fenics/shared -w /home/fenics/shared quay.io/fenicsproject/stable:current

should install Fenics.

DRL :

We are using both Gym - OpenAI and Stable-Baselines. They both can installed with :

pip install gym 
pip install stable-baselines

More generally, everything you need can be installed with :

pip install --user tensorflow keras gym stable-baselines sklearn

Launch an experiment :

An experiment consists of an Environement (based on Gym - OpenAI & Fenics), and an Algorithm from Stable-Baselines. They can be launched with test_main.py. You will only have to precise a few parameters :

*nb_cpu*: the amount of CPU you want to use (e.g. 16)
*agents*: an array of the algorithms you want to use (e.g. ['PPO2','A2C'])
*name_env*: The name of the environment (e.g. 'Control-cylinder-v0')
*total_timesteps*: the amount of timesteps the training will last (e.g. 100000)
*text*: Some precisions you wanna add to you experiment (e.g. '1_step_1_episode_2CPU')

Build your own environment :

The Gym.env environment :

You can find examples of such environments in example 1 : Control Cylinder or example 2 : Flow Control Cylinder. They always share the same architecture :

class FluidMechanicsEnv_(gym.Env):
    metadata = {'render.modes': ['human']}

    def __init__(self,
                    **kwargs):
        ...
        self.problem = self._build_problem()
        self.reward_range = (-1,1)
        self.observation_space = spaces.Box(low=np.array([]), high=np.array([]), dtype=np.float16)
        self.action_space = spaces.Box(low=np.array([]), high=np.array([]), dtype=np.float16)
        
    def _build_problem(self,main_drag):
        ...
        return problem
        
        
    def _next_observation(self):
        ...
        
        
    def step(self, action):
        ...
        return obs, reward, done, {}
    
    def reset(self):
        ...

Here, most of these functions are DRL related, and more informations can be found at this paper (for applications of DRL on fluid mechanics) or here (for more general informations about DRL). The only link with Fenics is made with the

def _build_problem(self,main_drag):
        ...
        return problem

function, where you will be using functions from Fenics.

Fenics functions :

We built several functions to help you use Fenics and build DRL environment with it. Three main classes exist :

• class Channel
• class Obstacles
• class Problem

Channel :

Allows you to create the 'box' where your simulation will take place.

Obstacles :

Allows you to add forms and obstacles (Circle, Square and Polygons) to your environment.

Problem :

Build the simulation with Channl and Obstacles. Also get parameters for the mesh and the solver. Finally, this is a Problem object you will return in the Gym.env class.

What's next :

We built this repository in order to get a code as clean as possible for fluid mechanics with DRL. However, Fenics is not the best solver, especially with very demanding problem. The goal is to keep the same philosophy in mind (DRL and Fluid mechanics coupled easily) but with other (and faster) libraries. Since most of these libraries are C++ based, and using powerful clusters, the architecture will be completely different. We are still working on it and doing our best to release an alpha version as soon as possible.

This repository will be updated when such library finally comes out. Until then, we hope that with this paper and this repository combined, some Fluid Mechanics researcher might want to try to apply Deep Reinforcement Learning to their experiments.

The Team :

• Paul Garnier : MINES Paristech - PSL Research University
• Jonathan Viquerat : MINES Paristech - PSL Research University - CEMEF
• Aurélien Larcher : MINES Paristech - PSL Research University - CEMEF
• Elie Hachem : MINES Paristech - PSL Research University - CEMEF