Checkpoint/Restart framework

[smcantab]

I gave this some more thoughts and it will be a pretty serious task, with some issues. Let's start with the one big issue I know of:

there is a bug in the stl rng for gcc4.6 (fixed in gcc4.7) : the state of the rng is not saved or loaded correctly (from stream) so that you can't just continue a calculation on the same random number sequence (hence any test, even if we reload the rng will yield different results). This issue is true at least if the state is read from a stream (I am not sure about other serialization methods, see boost serialize for example). Currently all our computers use gcc4.6.

About the implementation:

each module will have to have a save and load function and MC will have to have a save and load function in turn that saves its own state and calls the respective functions on the modules. I can't see a way around this because some of the modules have their own rng and a set of members that must be reinstated. Then it actually remains the problem of I/O for all this information, it's not obvious to me how to serialize it out of c++, considering that there is also the python layer that needs to be taken care of. Online I found a lot of references to boost serialize to do this sort of things.

Boost serialize would introduce an additional dependency (it could be made optional because check-pointing is a non essential feature). This still requires quite a bit of writing at the c++ level and remains the question of how to make this compatible with the python layer, although it doesn't seem impossible to do (one would have to rebuild the object from python and then call the MC::load function that will make sure everything goes back to the old state).

If we can do this, then re-instating the python layer state is simple because it's just a matter of reloading self.dict before calling mc.load. Unless there will be unforeseen complications.

Oh let's not forget that the potential and optimizer class should be serialized too for proper checkpointing, so the Pele potentials and minimizers would have to undergo this revision too, at least in principle.

This is the only way I can see how to implement a proper check-pointing system without something like BLCR system.

Since this sounds like a ton of work to me I would like to get as many suggestions as possible and get everyone to agree on the best course of action.

[js850]

I really think you're asking for trouble trying to do it that way. If you pickle a class. Then change the class, then try to reload it you're gonna have a bad time. I would explicitly save all the data that needs to be saved.

checkpoint = dict()
checkpoint["niter"] = niter
checkpoint["ncalls"] = ncalls
checkpoint["histogram"] = <numpy array holding the histogram>

etc
then pickle the dictionary. Don't save anything more complicated than a numpy array.

[smcantab]

that's how I thought of it in the first place but then I got stuck on the modules, as each module can have any amount of arbitrary information which you don't want to manage from the MC class.

To do it from python (and manually), the only way around that I see is this: each module pickles itself in a separate pickle file, then when MC.load() is called this tries to reset the states of the modules from their respective pickle files. For example there will be mc.cp.pickle, takestep.cp.pickle and so on to load from.

This is the only way I see to maintain the modularity, since MC does not necessarily know everything the modules know.

[js850]

Yes, that could work. Maybe something like

def save_state(self):
    """return a picklable object"""

@static_method
def restore_state(state):
    """return a new class constructed from state"""

[smcantab]

I started writing my own basic sqlite3 wrapper for c++ but then realised that there is a large number of c++ wrappers out there, see here.
Do you think it might be worth continuing writing our own basic version of an sqlite wrapper or should we just adopt an existing and more exhaustive wrapper?

[js850]

It would be better to find an existing package. Try to find one that is

1. compatible licence (BSD is better than GPL)
2. small and simple.

[smcantab]

this question, however, suggests that there is really no need to write an exhaustive wrapper as one can use sqlite directly from c++. So maybe I should continue writing the the most simple functions myself and then use sqlite directly from a SqliteDB.execute( command ) function (where SqliteDB is my own class)?

[js850]

OK, give it a try and see how far you get. You may run into trouble when you try to put a std::vector into the database.

[kjs73]

I would agree that if in doubt and if we need a wrapper we should not write it ourselves.

[smcantab]

So I have been working on writing an insert vector function and since sql does not understand array one has to turn the vector into a stream and then back. Picking up things around the web I came up with this serialization and deserialization of vectors, please let me know whether you think it can be improved (I am not an expert with stream and string manipulation in c++):

template<typename T>
std::string serialize_vector(std::vector<T> vec)
{
    std::ostringstream oss;
    //store the size of the vector first
    oss << vec.size();
    if (!vec.empty()){
        std::copy(vec.begin(), vec.end()-1, std::ostream_iterator<T>(oss, "\n"));
    }
    return oss.str();
}

template<typename T>
std::vector<T> deserialize_vector(std::string ser_vec)
{
    std::stringstream ss;
    size_t size;
    std::vector<T> vec;

    //turn string into stream
    ss.str(ser_vec);
    //read vector size
    ss>>size;
    ser_vec.resize(size);

    for (size_t i=0; i<size; i++){
        ss >> vec[i];
    }

    return vec;
}

[kjs73]

Probably in the first function the vector could be passed by (const) reference.

[smcantab]

sorry I had to re-edit a couple of times

[js850]

If I read correctly you are turning the vector into a string of numbers, e.g. "1.245 1.434 5.3949".

You will run into problems doing it this way. First, be careful about precision, make sure you are printing at extremely high precision.

It would be better to serialize it in binary. I don't know how to do this, but presumably you can copy the whole block of memory vector.data() and cast it as a char array. you'll have to store the length somhow (in a header?).