1.4 In-Memory Storage

[yyin-dev]

This issue is intended to be where all discussions about Section 1.4 In-memory Storage happen (at least before the course is released). Please don't close this.

[yyin-dev]

I think we should explain more in the doc about the following designs:

• Use BitVec + Vec<T> for PrimitiveArray<T> instead of Vec<Option<T>> for memory efficiency;
• A DataChunk contains a number of Array. A DataChunk logically represents a subset of rows in a table, so a table contains a number of DataChunk.

Also, I personally think we can provide more skeletons/guidance in the doc because this section requires writing much more code than previous ones:

• Provide skeleton code for Array and ArrayBuilder;
• More guidance on implementing ArrayIter (e.g. lifetime);
• Provide basic test cases.

I suggest we create the following subtasks for this section:

• Implement PrimitiveArray (and builder), Utf8Array (and builder) and ArrayImpl (and builder);
• Implement ArrayIter, enable collect for PrimitiveArray and Utf8Array;
• Implement DataChunk

[wangrunji0408]

I just opened a draft PR #21 for this section. Feel free to directly commit to this branch!

[yyin-dev]

I just opened a draft PR #21 for this section. Feel free to directly commit to this branch!

Just quickly skimmed through your edits... Looks nice -- I am impressed by how fast you write ;)

[wangrunji0408]

I am impressed by how fast you write ;)

No no no, it was written several months ago. 😅

It really takes time to draft these docs.

[yyin-dev]

No no no, it was written several months ago. 😅

It really takes time to draft these docs.

Just curious, did you consult any paper/textbook/article when writing those texts?

[wangrunji0408]

Not yet. Any recommendations? 👀