Unable to locate config file dir

[lochhh]

The Pose reader expects to find the config file dir in one of two places:

• local_config_file_dir, where the config file dir is in the same parent dir as file
• shared_config_file_dir, where it falls back to the Pose reader's _model_root attribute (default is /ceph/aeon/aeon/data/processed)

With config dir = /ceph/aeon/aeon/data/ingest/222/..., the following would raise a FileNotFoundError.

aeon_api/swc/aeon/io/reader.py

Lines 308 to 325 in 489b9c2

	def read(self, file: Path, include_model: bool = False) -> pd.DataFrame:
	"""Reads data from the Harp-binarized tracking file."""
	# Get config file from `file`, then bodyparts from config file.
	model_dir = Path(file.stem[self._pattern_offset :].replace("_", "/")).parent
	# Check if model directory exists in local or shared directories.
	# Local directory is prioritized over shared directory.
	local_config_file_dir = file.parent / model_dir
	shared_config_file_dir = Path(self._model_root) / model_dir
	if local_config_file_dir.exists():
	config_file_dir = local_config_file_dir
	elif shared_config_file_dir.exists():
	config_file_dir = shared_config_file_dir
	else:
	raise FileNotFoundError(
	f"""Cannot find model dir in either local ({local_config_file_dir}) \
	or shared ({shared_config_file_dir}) directories"""
	)

MWE

from swc.aeon.io import reader
from pathlib import Path

file = Path("/ceph/aeon/aeon/data/ingest/AEON3/social0.3/2024-07-03T20-00-42/CameraTop/CameraTop_222_gpu-partition_5545029_2024-08-06T06-19-19_fullpose-id_2024-07-03T20-00-00.bin")
model_dir = Path(file.stem[10:].replace("_", "/")).parent 
local_config_file_dir = file.parent / model_dir 
actual_cfg_file_dir = Path("/ceph/aeon/aeon/data/ingest/222/gpu-partition/5545029/2024-08-06T06-19-19/fullpose-id")
shared_config_file_dir = Path(reader.Pose("CameraTop_222*")._model_root) / model_dir
if local_config_file_dir.exists():
    local_config_file_dir
elif shared_config_file_dir.exists():
    shared_config_file_dir
else:
    raise FileNotFoundError(
        "Cannot find model dir in the following directories: \n"
	f"local ({local_config_file_dir})\n"
        f"shared ({shared_config_file_dir})\n"  
        f"Actual config dir is {actual_cfg_file_dir}"
    )

raises

FileNotFoundError: Cannot find model dir in the following directories:
local (\ceph\aeon\aeon\data\ingest\AEON3\social0.3\2024-07-03T20-00-42\CameraTop\222\gpu-partition\5545029\2024-08-06T06-19-19\fullpose-id)
shared (\ceph\aeon\aeon\data\processed\222\gpu-partition\5545029\2024-08-06T06-19-19\fullpose-id)
Actual config dir is \ceph\aeon\aeon\data\ingest\222\gpu-partition\5545029\2024-08-06T06-19-19\fullpose-id

A potential solution is to change the Pose reader default to "/ceph/aeon/aeon/data/ingest" instead of "/ceph/aeon/aeon/data/processed", and then update older schemas assuming the original default in aeon_mecha (e.g. social_01), assuming we move forward with ingesting from "/ceph/aeon/aeon/data/ingest".
Otherwise we can override this in aeon_mecha's social_02 and social_03 Pose readers.

[glopesdev]

@lochhh I feel long-term we should anyway avoid relying on the default directory for the ingestion process or any analysis, to avoid having expectations changing on updates.

For example, if we were to change the default to this new value, any scripts relying on the previous default would have their behaviour silently changed and the corresponding results changed.

So I guess if we ever do change the existing default, my preference would be to remove it altogether.

[lochhh]

Thanks @glopesdev. This makes perfect sense. I've updated PR #21 and opened aeon_mecha/PR#486 accordingly.