ASK/TELL DEVELOP #1307
ASK/TELL DEVELOP #1307
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #1307 +/- ##
===========================================
+ Coverage 78.23% 78.75% +0.52%
===========================================
Files 76 79 +3
Lines 7561 7983 +422
Branches 1116 1195 +79
===========================================
+ Hits 5915 6287 +372
- Misses 1447 1477 +30
- Partials 199 219 +20 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| } | ||
|
|
||
| my_APOSMM = APOSMM(gen_specs) | ||
| my_APOSMM.setup() |
There was a problem hiding this comment.
Does this need to be separate to constructor?
There was a problem hiding this comment.
Perhaps you've seen already, but setup() sets attributes that can't be pickled, but need to be done anyway. So I believe a separate generator.setup() needs to exist.
But perhaps for outside-of-libE purposes, to save a line, this could resemble: my_APOSMM = APOSMM(gen_specs, setup=True).
|
Currently this has two existing user functions refactored to ask/tell, and it is a breaking change for codes using those gens. I don't think this is what we want. So, for now, we could supply these as duplicates (keeping the original), or remove them. Alternative would be to refactor all appropriate gens to ask/tell, but that would hold this up, so I think it may be better to supply these two as a duplicate for now. Also, would it be better for these ask/tell gens (rand sample and gpCAM) to use AskTellGenRunner and not use the wrapper gen_f (given a breaking change anyway). Issues we need to address:
|
Sounds good. Maybe we could raise DeprecationWarnings?
I prefer the AskTellGenRunner myself (I did develop it), but if all the gens move to my runner, then what would you want to do with your wrapper? |
|
I think there is also some opportunity for inheritence with the gpCAM class. And gp_cam_simple is more complicated than gp_cam_asktell. So, if anything the latter would be the base class, and renamed to avoid confusion. See #1316 |
libensemble/libE.py
Outdated
| # ==================== Local version =============================== | ||
|
|
||
|
|
||
| def _retrieve_generator(gen_specs): |
There was a problem hiding this comment.
I don't know why this is needed. If gen is on a thread, should not need to be pickled.
There was a problem hiding this comment.
I've removed on my branch for now, so don't change here
|
Checking random sample when running without Optimas: |
|
With Optimas, currently looks like random sample doesn't match, even when trying to account for seed |
|
Precision fixed in the Optimas example: optimas-org/optimas@08a835b |
|
We may need to update gpCAM gens for latest gpCAM release. Make sure any changes made in gen_f is reflected here before pulling in.
|
libensemble/gen_classes/gpCAM.py
Outdated
| self.all_y = np.vstack((self.all_y, self.y_new)) | ||
|
|
||
| if self.my_gp is None: | ||
| self.my_gp = GP(self.all_x, self.all_y, noise_variances=self.noise * np.ones(len(self.all_y))) |
There was a problem hiding this comment.
I need to update for new gpCAM interface
libensemble/gen_classes/sampling.py
Outdated
| """ | ||
|
|
||
| def __init__(self, _, persis_info, gen_specs, libE_info=None) -> list: | ||
| # self.H = H |
libensemble/gen_classes/gpCAM.py
Outdated
| self.all_y = np.empty((0, 1)) | ||
| np.random.seed(0) | ||
|
|
||
| def __init__(self, H, persis_info, gen_specs, libE_info=None): |
There was a problem hiding this comment.
I will put above _initialize_gpcAM
There was a problem hiding this comment.
and make _initialize_gpcAM _initialize_gpCAM
| self.all_x = np.empty((0, self.n)) | ||
| self.all_y = np.empty((0, 1)) | ||
| np.random.seed(0) | ||
|
|
There was a problem hiding this comment.
We need to decide __init__ interface.
We questioned before whether we keep the same interface - which mirrors the current gen_f, or to rearrange, as H is often not given (basically an H0). So it could be gen_specs first. I'm leaning towards keeping the original ordering as it mirrors our user functions, but this should be discussed.
There was a problem hiding this comment.
Fair enough. My opinion/intuition is a user is more likely to prefer either "classical" gens (e.g. Jeff) or ask/tell gens (e.g. other CAMPA folks). With these gens' interfaces and users being so different, I don't think an arguably simpler rearrangement of the input parameters is too confusing.
Similarly to how some people prefer numpy or pandas; they do similar things, but their interfaces being different isn't a point of contention.
I'd also lean towards if someone were to initialize some object, like a gen, themselves, they'd prefer their specifications be provided as early and clearly as possible:
my_gen = Generator(param=1,
option="two",
)
vs.
my_gen = Generator(None,
{},
{"param": 1, "option": "two"},
{},
)
There was a problem hiding this comment.
If we pull this in, are we treating this as provisional, without documenting gen_classes, so interface (e.g. __init__ option ordering) is mutable. Or are we ready to say a user can use the gen_classes interface for their own gens if desired? This questions applies to the numpy interface (ask_np/tell_np). As for the outer API, we certainly have not finalized that, and it should somehow be noted.
E.g.,
# This feature is in beta and its interface may change in future releases
Note, that currently the two implemented classes (rearranged to ask/tell) are duplicates of gen_f, but in the long term we will not want to have duplicates.
Discussions:
-
Agree current status of gen_classes (internal or public).
-
If provisional - mark somehow in code
-
Agree wrapper paradigm (outer v inner interface)
-
Agree naming ask_np / tell_np
- still not sure
ask_numpy()is better than
@convert_dict_to_numpy ask()
It seems we need separate function names, and the output of an ask() would be ambiguous.
-
Do we allow passing a class for ask/tell (if a libE generator) so initialize internally.
-
this is currently supported with the wrapper method but not internal ask/tell method
-
I think we should probably not support both wrapper and ask/tell approaches.
-
Agree num_points for ask policy (initial and thereafter):
-
Always empty ask and generator has to decide (most similar to now but what about external gens?)
-
intercept:
- init:
gen_specs['user']fieldsinitial_batch_sizeorgen_batch_sizeand else use num workers - after: intercept num points coming in for tell and ask for that many
- Problem with intercepting
gen_specs['user']is names for batch size may be different.
- init:
-
Add fixed gen_specs options
gen_specs['initial_batch_size']- and maybe something like
gen_specs['return_policy'](or better name)- fixed batch size or generate num points got back or let generator decide points - empty ask()
- (at some point may replace alloc_specs
async_returnif we want most user not to worry about allocs)?
- and maybe something like
-
Data conversion - how treat multi-dimensional fields.
-
currently makes a numpy array (as a dictionary value).
-
Agree gpCAM class names (e.g., GP_CAM_Covar) - and match gen_func names (original functions).
-
Naming convention for tests - we have:
- test_gpCAM_class.py
- test_asktell_surmise.py
- test_persistent_surmise_killsims_asktell.py
- test_persistent_aposmm_nlopt_asktell.py
- test_sampling_asktell_gen.py
-
Do our gens all support passing in intiail points? And if so, in standard format?
Interfacer:
- Does it work with executor (consideration of things added to executor in worker.py - including first-tier comm object)
- Does it work when there is no executor
- For user function should we insert comm into libE_info["comm"] inside qcomm_main
- Works with processes and threads - or picked one (and named right in generators.py)
- Works with fork and spawn (for spawn be clear what in calling script needs to be inside main block).
- Sometimes get hang at end of runs using the interfacer - esp. if no final_tell, is there something not getting shut-down? Check this works.
- Need to deal with final_tell when using gen (e.g. optimas)
- Need to sepaarate final_tell as required by standard - tell and extract data function.
- Constructor wrapper approach - demonstrate alternatives in notebook
Todo:
- SH - update gpCAM to reflect current interface.
- Change/rename original gpCAM to match naming in the class.
- Need to run 'extra' tests for gpCAM tests to get run. Check passes.
- Check correct no. points produced each call of gen (usually number sent back).
Check works:
- Running libE ask/tell generator - e.g. gpCAM through Optimas (it will go via ask/tell wrappers).
- Use libEnsemble ask/tell generator standalone (outside of libE)
- must libE be installed to use generators?
- Running libE with our ask_numpy/tell_numpy functions.
- Running libE with ask/tell function (standardised format).
libensemble/generators.py
Outdated
|
|
||
| class Generator(ABC): | ||
| """ | ||
| v 0.7.2.24 |
There was a problem hiding this comment.
Is this version number something we need?
libensemble/generators.py
Outdated
| """ | ||
|
|
||
| def __init__( | ||
| self, gen_specs: dict, History: npt.NDArray = [], persis_info: dict = {}, libE_info: dict = {} |
There was a problem hiding this comment.
Here we have a different constructor signature to those functions in gen_classes.
|
In the full tests: https://github.com/Libensemble/libensemble/actions/runs/10256911231/job/28376940998#step:22:684 |
Update finalize to meet standard
3 participants
See checklist in #1307 (review)