Centering with MapControls and Orthographic camera

[squiles]

What would be the best way of moving the camera to a certain position programatically while using MapControls and an Orthographic camera? I've tried the following function but for some weird reason the camera rotates and shows a top/side view of the position I'm trying to center.

const centerCamera = (controls, camera) => {
  const point = new THREE.Vector3(0, 500, 1);

  camera.position.copy(point);
  camera.lookAt(point.x, point.y, 10);

  controls.target.copy(point);

  controls.update();

  camera.updateProjectionMatrix();
  camera.updateMatrix();
};

I made a small example to illustrate the problem at: https://codesandbox.io/s/sleepy-dewdney-zuvyx?file=/src/App.js:838-1117

In the example I'm trying to center the camera on a square but I think that when the center function runs it somehow conflicts with the controls and it rotates.