Self-contained data pre-processing example binary classification (cancer vs. no cancer)

[moritzknolle]

Dear EMBED OPEN DATA team,
Would it be possible to modify/adapt the 'NUS_example_pipeline.ipynb' notebook so that it is entirely self-contained and can be executed with access to only the .csv files available on AWS? Currently, the notebook makes heavy use of additional columns and information (exam_birads, exam_path_severity, exam_path_desc, exam_outcome, etc.) from complicated pre-processing that are not available in any of the AWS files...

I think having a self-contained example showing how to process EMBED data for cancer vs. no cancer classification, which includes official patient-level train-test split information, would greatly accelerate community adoption of this great resource!

Thanks!

[beatrice-b-m]

Hi Moritz,

I'm sorry for not seeing this issue sooner! This is a really valuable point, I'll discuss this with the team to see how we can improve clarity of our documentation/data for the v2 release.

The NUS_example_pipeline.ipynb notebook was created as reference for a datathon at the National University of Singapore, so they were working from partially-preprocessed data where I'd already derived these features for them. We're planning to remove these files shortly so I'm sorry for any confusion they caused!

I'm not sure if you've already seen, but I recently uploaded a notebook with an end-to-end clinical label assignment process that includes how we derive many of these features in depth screening_label_assignment.ipynb. This notebook uses our internal EMBED-toolkit package which has helper functions based on our common workflows for EMBED.