Skip to content

Commit f6f8e65

Browse files
authored
Enhance README with AI-native data storage details (#4369)
Expanded on the AI-native data storage capabilities and challenges in data preparation for CarbonData.
1 parent ce99229 commit f6f8e65

File tree

1 file changed

+9
-1
lines changed

1 file changed

+9
-1
lines changed

README.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,16 @@
1717

1818
<img src="/docs/images/CarbonData_logo.png" width="200" height="40">
1919

20-
Apache CarbonData is an indexed columnar data store solution for fast analytics on big data platform, e.g. Apache Hadoop, Apache Spark, etc.
20+
- AI-native data storage for CarbonData's new scope, In AI projects, data scientists and engineers spend 80% of their time on data preparation. Traditional storage presents numerous bottlenecks in this process:
21+
- Data silos: Training data may be scattered across data lakes, data warehouses, file systems, object storage, and other locations, making integration difficult.
22+
- Performance bottlenecks:Training phase: High-speed, low-latency data throughput is required to feed GPUs to avoid expensive GPU resources sitting idle.
23+
- Inference phase: High-concurrency, low-latency vector similarity search capabilities are required.
24+
- Complex data formats: AI processes data types far beyond tables, including unstructured data (images, videos, text, audio) and semi-structured data (JSON, XML). Traditional databases have limited capabilities for processing and querying such data.
25+
- Lack of metadata management: The lack of effective management of rich metadata such as data versions, lineage, annotation information, and experimental parameters leads to poor experimental reproducibility.
26+
- Vectorization requirements: Modern AI models (such as large language models) convert all data into vector embeddings. Traditional storage cannot efficiently store and retrieve high-dimensional vectors.
2127

28+
29+
- In the previous releases Apache CarbonData is an indexed columnar data store solution for fast analytics on big data platform, e.g. Apache Hadoop, Apache Spark, etc.
2230
You can find the latest CarbonData document and learn more at:
2331
[https://carbondata.apache.org](https://carbondata.apache.org/)
2432

0 commit comments

Comments
 (0)