Finding all runs with flows that can be reinstantiated

[PGijsbers]

You might be interested in retrieving results from OpenML only for those flows which you can locally reinstantiate (a question we recently received in the community). By and large, these are likely just scikit-learn flows. Unfortunately, trying to do this with openml.runs.list_runs with a list of thousands of tasks and/or flows will result in hitting limitations of the server's REST API. We can work around it with some extra steps, and the Python script below wil do just that. Some warnings:

• This script will take long to run (hours), consider extending it with checkpointing and easily loading from the previous progress to do this in parts (especially the third part of the script, as the first part is quick and the second part does benefit from the local openml cache). If you do improve the script that way, please leave a message here with your suggestions :)
• When reinstantiating a flow, the script currently does not require strict version matches (strict_version=False for get_flow). This means the actual model implementation will not necessarily be identical to the one used to produce the results.
• This script will download all flows with "sklearn" in their name by default. As of writing, this is ~10k flows. It's only about 105MB but many separate files.
• The script will perform thousands of calls to OpenML server, a stable internet connection is essential.
• Below the script I'll give a list of flow ids that I could reinstantiate locally (py3.10, openml==0.12.2, scikit-learn==1.1.2). You might want to use that as a starting point instead of testing all sklearn_flow_ids queried from the server locally.

The script will result in a pandas dataframe (and csv file) that looks similar to:

run_id,task_id,setup_id,flow_id,uploader,task_type,upload_time,error_message
10434865,3,8258921,17474,6127,TaskType.SUPERVISED_CLASSIFICATION,2019-12-23 23:49:53,
10228505,3,8153729,12677,8323,TaskType.SUPERVISED_CLASSIFICATION,2019-06-16 01:12:25,
10228510,3,8153734,12677,8323,TaskType.SUPERVISED_CLASSIFICATION,2019-06-16 01:16:55,
10228511,3,8153735,12677,8323,TaskType.SUPERVISED_CLASSIFICATION,2019-06-16 01:20:02,

Here's the script:

import openml
import pandas as pd

# 1: List all Classification tasks and return their meta-data in a dataframe
classification_tasks = openml.tasks.list_tasks(task_type=openml.tasks.TaskType.SUPERVISED_CLASSIFICATION, output_format="dataframe")

# 2: Find all scikit-learn flows that we can reinstantiate
flows = openml.flows.list_flows(output_format="dataframe")
sklearn_flows = flows[flows["name"].str.contains("sklearn")]
sklearn_flow_ids = list(flows[flows["name"].str.contains("sklearn")].id)
reinstantiable_flows = []

print("Downloading scikit-learn flows to see which can be reinstantiated.")
for i, flow_id in enumerate(sklearn_flow_ids):
  print(f"Checked {i:4d}/{len(sklearn_flow_ids):4d} flows, so far {len(reinstantiable_flows)} can be reinstantiated.", end="\r")
  try:
    flow = openml.flows.get_flow(flow_id, reinstantiate=True, strict_version=False)
    reinstantiable_flows.append(flow_id)
  except KeyboardInterrupt:
    raise
  except:
    pass

non_reinstantiable_flows = set(sklearn_flow_ids) - set(reinstantiable_flows)
print(f"Could not reinstantiate {len(non_reinstantiable_flows)} flows: {non_reinstantiable_flows}")
print(f"Ignoring above flows for the remainder and continue with {len(reinstantiable_flows)} reinstantiable flows.")

# 3. Now what we want to do is openml.runs.list_runs with all tasks and flows we found above, but unfortunately
# querying all combinations for flow x task at once is not possible right now, because of limitations in our Rest API
# Running:
# > sklearn_classification_runs = openml.runs.list_runs(flow=list(sklearn_flows.id), task=list(classification_tasks.tid))
# will return an error. So instead we will construct the query piece-by-piece:
classification_task_ids = list(classification_tasks.tid)
chunk_size = 250
sklearn_classification_runs = pd.DataFrame()
for i, task_id in enumerate(classification_task_ids):
  print(f"{i:4d}/{len(classification_task_ids):4d} tasks processed.", end="\r")
  for offset in range(0, len(reinstantiable_flows), chunk_size):
    print(f"{i:4d}/{len(classification_task_ids):4d} tasks processed > {offset:4d}/{len(reinstantiable_flows):4d} flows processed for current task.", end='\r')
    run_batch = openml.runs.list_runs(flow=reinstantiable_flows[offset:offset+chunk_size], task=[task_id], output_format="dataframe")
    sklearn_classification_runs = pd.concat([sklearn_classification_runs, run_batch])
  print(f"{i:4d}/{len(classification_task_ids):4d} tasks processed > {len(reinstantiable_flows):4d}/{len(reinstantiable_flows):4d} flows processed for task {task_id: 6d}.")

  sklearn_classification_runs.to_csv("sklearn_classification_runs.csv", index=False)

Flows I could reinstantiate locally (without strict version matches):
[5513, 5515, 5516, 5642, 6098, 6563, 6564, 6751, 6790, 7021, 7023, 7024, 7028, 7167, 7169, 7175, 7346, 7349, 7367, 7388, 7390, 7391, 7395, 7438, 7440, 7451, 7454, 7468, 7475, 7516, 7533, 7541, 7552, 7562, 7568, 7579, 7585, 7588, 7591, 7673, 7696, 7712, 7814, 7906, 7973, 7992, 8063, 8064, 8165, 8457, 8519, 8575, 8605, 8624, 8738, 8777, 8781, 8790, 8825, 8840, 8848, 8849, 8858, 8864, 8869, 8877, 8878, 8896, 8924, 8946, 8951, 8955, 8966, 8971, 8994, 9016, 9041, 9042, 9047, 9073, 9077, 9150, 9152, 9182, 9213, 9547, 9554, 9556, 9559, 9577, 9633, 9671, 9686, 11053, 12127, 12677, 12736, 12737, 13104, 13296, 13391, 14715, 16358, 17278, 17284, 17285, 17300, 17318, 17320, 17321, 17331, 17338, 17351, 17352, 17353, 17354, 17356, 17362, 17366, 17370, 17372, 17377, 17407, 17412, 17430, 17453, 17457, 17458, 17459, 17460, 17461, 17467, 17474, 17479, 17480, 17481, 17482, 17483, 17484, 17485, 17486, 17487, 17497, 17499, 17500, 17501, 17503, 17577, 17582, 17586, 17629, 17630, 17631, 17635, 17636, 17637, 17638, 17640, 17641, 17642, 17645, 17656, 17657, 17658, 17695, 17696, 17697, 17698, 17699, 17700, 17702, 17703, 17704, 17705, 17706, 17707, 17708, 17709, 17725, 17726, 17728, 17730, 17732, 17733, 17734, 17735, 17738, 17740, 17742, 17744, 17746, 17750, 17751, 17752, 17755, 17756, 17757, 17758, 17760, 17761, 17762, 17764, 17765, 17768, 17770, 17772, 17773, 17774, 17775, 17776, 17778, 17779, 17780, 17782, 17783, 17784, 17785, 17786, 17788, 17789, 17790, 17791, 17793, 17794, 17795, 17796, 17797, 17798, 17800, 17802, 17804, 17806, 17807, 17809, 17810, 17811, 17812, 17814, 17816, 17818, 17819, 17820, 17825, 17827, 17829, 17830, 17832, 17833, 17834, 17835, 17836, 17838, 17839, 17840, 17841, 17842, 17843, 17849, 17850, 17851, 17853, 17854, 17856, 17858, 17859, 17860, 17861, 17862, 17863, 17866, 17867, 17870, 17871, 17872, 17873, 17877, 17880, 17881, 17884, 17888, 17895, 17899, 17902, 17905, 17907, 17911, 17912, 17913, 17914, 17916, 17918, 17919, 17922, 17933, 17934, 17937, 17943, 17951, 17961, 17962, 17966, 17975, 17976, 17979, 17982, 17984, 17985, 17986, 17987, 17988, 17989, 18052, 18283, 18286, 18288, 18289, 18291, 18293, 18297, 18302, 18303, 18304, 18305, 18447, 18448, 18449, 18450, 18451, 18453, 18455, 18458, 18459, 18460, 18461, 18462, 18465, 18467, 18468, 18469, 18470, 18471, 18472, 18473, 18480, 18483, 18485, 18486, 18489, 18490, 18492, 18493, 18495, 18496, 18499, 18501, 18503, 18504, 18506, 18507, 18508, 18509, 18510, 18511, 18514, 18515, 18516, 18517, 18518, 18519, 18520, 18521, 18522, 18523, 18524, 18525, 18526, 18527, 18528, 18529, 18532, 18533, 18534, 18535, 18536, 18537, 18539, 18540, 18541, 18545, 18546, 18552, 18553, 18554, 18560, 18561, 18562, 18563, 18564, 18565, 18566, 18575, 18582, 18583, 18585, 18592, 18595, 18596, 18597, 18598, 18599, 18603, 18604, 18605, 18606, 18684, 18686, 18695, 18696, 18697, 18698, 18699, 18707, 18709, 18710, 18711, 18712, 18713, 18714, 18715, 18716, 18718, 18722, 18723, 18724, 18730, 18733, 18744, 18746, 18749, 18764, 18765, 18766, 18769, 18771, 18772, 18775, 18776, 18777, 18783, 18787, 18789, 18793, 18794, 18795, 18796, 18797, 18813, 18819, 18835, 18841, 18842, 18843, 18844, 18845, 18883, 18885, 18932, 18933, 18957, 18964, 18977, 18985, 18987, 18988, 18991, 18996, 18997, 19000, 19001, 19002, 19003, 19004, 19005, 19006, 19007, 19008, 19009, 19012, 19015, 19016, 19017, 19018, 19019, 19022, 19024, 19025, 19027, 19029, 19030, 19031, 19032, 19033, 19034, 19035, 19036, 19039, 19040, 19043, 19045, 19046, 19047, 19048, 19049, 19070, 19071, 19075, 19077, 19078, 19079, 19083, 19084, 19085, 19093, 19094, 19095, 19096, 19097, 19098, 19099, 19100, 19101, 19107, 19108, 19109, 19115, 19116, 19117, 19119, 19120, 19121, 19122, 19123, 19143, 19145, 19146, 19147, 19148, 19149, 19150, 19152, 19153, 19154, 19155, 19156, 19157, 19158, 19159, 19160, 19161, 19162, 19163, 19164, 19165, 19166, 19167, 19168, 19169, 19170, 19171, 19172, 19173, 19174, 19175, 19176, 19177, 19178, 19179, 19180, 19181, 19182, 19183, 19184, 19185, 19186, 19187, 19188]