Change distribution after create Matrix

[Goon83]

Hi All,
Another question regarding the function availability of DASH.

What will happen if I change the distribution (block size, halo layer) of a existing Matrix ?
Will the DASH support re-distribution the data following new parameters?

Bests,
Bin

[devreal]

What will happen if I change the distribution (block size, halo layer) of a existing Matrix ?
Will the DASH support re-distribution the data following new parameters?

I'm not sure that is possible with the current interface. Once a matrix is allocated there is no way to change its layout (afaics).

[Goon83]

Hi @devreal
Got it and thanks for the clarification.

Bests,
Bin

[devreal]

Could you elaborate what you have in mind for it? Maybe we will get a chance at some point to implement such a feature if there is a good use case.

[Goon83]

Hi @devreal
It is like the "repart" operator in SciDB (doc). . The "repart" changes the chunk size (called tile size in DASH) of an existing array. The "repart" in SciDB mostly involves the disk operation (might be just metadata but not sure so far). A "repart" for DASH array definitely needs to consider lots of communication among processes (units in DASH) to redistribute data.

We might also explore this idea in near future. To avoid repeating work, ping me DBin@lbl.gov
if you are interested in working together. We can have a chat then.

Bests,
Bin

[fuchsto]

@dash-project/developers First, I want to announce my "comeback" as an active contributor to DASH after my hiatus of over a year.
Regarding this feature request, I have prepared concepts for dynamic adaptations such as load-balancing dash::Matrix (aka dash::NArray) which I originally designed.
Also, I'm currently supervising a master's thesis on sparse matrix concepts for DASH.
I will put the respective current states of development in feature branches as soon as possible.

In brief, the NArray / Matrix concepts will not be affected but extended by additional free-function-style concepts. So if you (@Goon83) can use dash::Matrix in its current state and postpone the rebalancing / repartitioning parts in your work, your existing code will be unaffected.

[Goon83]

Hi @fuchsto
Sorry for not monitoring this thread closely.

Welcome back to this wonderful project.

Yes, the rebalancing feature is very interesting to us.

Let me know how can I help.
Bests,
Bin