Skip to content

Commit dc391a0

Browse files
liulanzhengliulanzheng
committed
update docs
Signed-off-by: liulanzheng <[email protected]>
1 parent 512059c commit dc391a0

15 files changed

+966
-342
lines changed

‎README.md

+8-342
Original file line numberDiff line numberDiff line change
@@ -1,355 +1,21 @@
11
# OSS Connector for AI/ML
22

3-
## Overview
4-
5-
The OSS Connector for AI/ML is a high-performance Python library specifically designed for AI and ML scenariosis, tailored to work with Alibaba Cloud OSS (Object Storage Service).
6-
7-
OSS Connector for AI/ML provides both [Map-style and Iterable-style datasets](https://pytorch.org/docs/stable/data.html#dataset-types) for loading datasets from OSS.
8-
And also provides a method for loading and saving checkpoints from and to OSS.
9-
10-
The core part of is OSS Connector for AI/ML is implemented in C++ using [PhotonLibOS](https://github.com/alibaba/PhotonLibOS). This repository only contains the code of Python.
11-
12-
13-
## Requirements
14-
15-
- OS: Linux x86-64
16-
- glibc: >= 2.17
17-
- Python: 3.8-3.12
18-
- PyTorch: >= 2.0
19-
20-
## Installation
21-
22-
```shell
23-
pip install osstorchconnector
24-
```
25-
26-
## Configuration
27-
28-
### Credential
29-
30-
For now only JSON format credential file is supported.
31-
```shell
32-
mkdir -p /root/.alibabacloud/
33-
cat <<-EOF | tee /root/.alibabacloud/credentials
34-
{
35-
"AccessKeyId": "<Access-key-id>",
36-
"AccessKeySecret": "<Access-key-secret>",
37-
"SecurityToken": "<Security-Token>",
38-
"Expiration": "2024-08-02T15:04:05Z"
39-
}
40-
EOF
41-
```
42-
`SecurityToken` and `Expiration` are optional.
43-
The credential file must be updated before expiration to avoid authorization errors.
44-
In the future, configuring credentials using the aliyun-oss-python-sdk will be supported.
45-
46-
### Config
47-
48-
```bash
49-
mkdir -p /etc/oss-connector/
50-
cat <<-EOF | tee /etc/oss-connector/config.json
51-
{
52-
"logLevel": 1,
53-
"logPath": "/var/log/oss-connector/connector.log",
54-
"auditPath": "/var/log/oss-connector/audit.log",
55-
"datasetConfig": {
56-
"prefetchConcurrency": 24,
57-
"prefetchWorker": 2
58-
},
59-
"checkpointConfig": {
60-
"prefetchConcurrency": 24,
61-
"prefetchWorker": 4,
62-
"uploadConcurrency": 64
63-
}
64-
}
65-
EOF
66-
```
67-
| Field | Description |
68-
|---------------|-------------------------------------------------------------------------------------------------------|
69-
| logLevel | The log level for log file, 0 - DEBUG, 1 - INFO, 2 - WARN, 3 - ERROR |
70-
| logPath | The path for log file, `/var/log/oss-connector/connector.log` is the default value. |
71-
| auditPath | The path for audit file, `/var/log/oss-connector/audit.log` is the default value. |
72-
| datasetConfig.prefetchConcurrency | The concurrency for perfetching data from Dataset. 24 is the default value. |
73-
| datasetConfig.prefetchWorker | The vcpu number for perfetching data from Dataset. 2 is the default value. |
74-
| checkpointConfig.prefetchConcurrency | The concurrency for perfetching checkpoint . 24 is the default value. |
75-
| checkpointConfig.prefetchWorker | The vcpu number for perfetching checkpoint. 4 is the default value. |
76-
| checkpointConfig.uploadConcurrency | The concurrency for uploading checkpoint. 64 is the default value. |
77-
78-
79-
## Examples
80-
81-
### IterableDataset
82-
83-
```py
84-
from osstorchconnector import OssIterableDataset
85-
86-
ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
87-
CONFIG_PATH = "/etc/oss-connector/config.json"
88-
CRED_PATH = "/root/.alibabacloud/credentials"
89-
OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/"
90-
91-
# 1) from_prefix
92-
iterable_dataset = OssIterableDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
93-
for item in iterable_dataset:
94-
print(item.key)
95-
print(item.size)
96-
content = item.read()
97-
print(len(content))
98-
item.close()
99-
100-
101-
# 2) from_objects
102-
uris = [
103-
"oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00001.png",
104-
"oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00002.png",
105-
"oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00003.png"
106-
]
107-
108-
iterable_dataset = OssIterableDataset.from_objects(uris, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)]
109-
for item in iterable_dataset:
110-
print(item.key)
111-
print(item.size)
112-
content = item.read()
113-
print(len(content))
114-
item.close()
115-
```
116-
117-
### MapDataset
118-
119-
```py
120-
from osstorchconnector import OssMapDataset
121-
122-
ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
123-
CONFIG_PATH = "/etc/oss-connector/config.json"
124-
CRED_PATH = "/root/.alibabacloud/credentials"
125-
OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/"
126-
127-
# 1) from_prefix
128-
map_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
129-
# random access
130-
item = map_dataset[0]
131-
print(item.key)
132-
content = item.read()
133-
print(item.size)
134-
print(len(content))
135-
item.close()
136-
137-
# or
138-
with map_dataset[5] as item:
139-
print(item.key)
140-
content = item.read()
141-
print(item.size)
142-
print(len(content))
143-
144-
# iterable
145-
for item in map_dataset:
146-
print(item.key)
147-
print(item.size)
148-
content = item.read()
149-
print(len(content))
150-
item.close()
151-
152-
153-
# 2) from_objects
154-
uris = [
155-
"oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00001.png",
156-
"oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00002.png",
157-
"oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/img001-00003.png"
158-
]
3+
[ossconnector.github.io](https://ossconnector.github.io/)
1594

160-
map_dataset = OssMapDataset.from_objects(uris, endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
161-
# random access
162-
item = map_dataset[1]
163-
print(item.key)
164-
print(item.size)
165-
content = item.read()
166-
print(len(content))
167-
item.close()
168-
169-
# iterable
170-
for item in map_dataset:
171-
print(item.key)
172-
print(item.size)
173-
content = item.read()
174-
print(len(content))
175-
item.close()
176-
```
177-
178-
Please note that OssMapDataset performs an OSS list objects operation under the given prefix first (which may take some time).
179-
180-
### Manifest file
181-
182-
Manifest file contains objects name (and label) of OSS objects.
183-
Building datasets with manifest file can reduce the overhead of listing objects in OSS, making it suitable for datasets with a large number of objects and repeated dataset loading.
184-
185-
A manifest file must be constructed in advance, and a method for parsing it must be provided during use.
186-
Below are examples of manifest files and loading a dataset with manifest file.
187-
188-
Example manifest file with object name:
189-
```
190-
Img/BadImag/Bmp/Sample001/img001-00001.png
191-
Img/BadImag/Bmp/Sample001/img001-00002.png
192-
Img/BadImag/Bmp/Sample001/img001-00003.png
193-
```
194-
195-
Example manifest file with object name and label:
196-
```
197-
Img/BadImag/Bmp/Sample001/img001-00001.png label1
198-
Img/BadImag/Bmp/Sample001/img001-00002.png label2
199-
Img/BadImag/Bmp/Sample001/img001-00003.png label3
200-
```
201-
202-
```py
203-
from osstorchconnector import OssIterableDataset
204-
205-
ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
206-
CONFIG_PATH = "/etc/oss-connector/config.json"
207-
CRED_PATH = "/root/.alibabacloud/credentials"
208-
OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/"
209-
210-
# manifest_parser
211-
def manifest_parser(reader: io.IOBase) -> Iterable[Tuple[str, str]]:
212-
lines = reader.read().decode("utf-8").strip().split("\n")
213-
for i, line in enumerate(lines):
214-
try:
215-
items = line.strip().split(' ')
216-
if len(items) >= 2:
217-
key = items[0]
218-
label = items[1]
219-
yield (key, label)
220-
elif len(items) == 1:
221-
key = items[0]
222-
yield (key, '')
223-
else:
224-
raise ValueError("format error")
225-
except ValueError as e:
226-
raise e
227-
228-
# from local manifest_file
229-
iterable_dataset = OssIterableDataset.from_manifest_file("manifest_file", manifest_parser, "oss://ossconnectorbucket/EnglistImg/", endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
230-
for item in iterable_dataset:
231-
print(item.key)
232-
print(item.size)
233-
print(item.label)
234-
content = item.read()
235-
print(len(content))
236-
item.close()
237-
238-
# manifest_file on oss
239-
iterable_dataset = OssIterableDataset.from_manifest_file("oss://ossconnectorbucket/manifest_file/EnglistImg/manifest_file", manifest_parser, "oss://ossconnectorbucket/EnglistImg/", endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
240-
```
241-
242-
### Dataset and transform
243-
244-
```py
245-
import sys
246-
import io
247-
import torchvision.transforms as transforms
248-
from PIL import Image
249-
250-
from osstorchconnector import OssIterableDataset, OssMapDataset
251-
252-
ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
253-
CONFIG_PATH = "/etc/oss-connector/config.json"
254-
CRED_PATH = "/root/.alibabacloud/credentials"
255-
OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/"
256-
257-
trans = transforms.Compose([
258-
transforms.Resize(256),
259-
transforms.CenterCrop(224),
260-
transforms.ToTensor(),
261-
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
262-
])
263-
264-
def transform(data):
265-
try:
266-
img = Image.open(io.BytesIO(data.read())).convert('RGB')
267-
val = trans(img)
268-
except Exception as e:
269-
raise e
270-
return val, data.label
271-
272-
iterable_dataset = OssIterableDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH)
273-
274-
for item in iterable_dataset:
275-
print(item[0])
276-
print(item[1])
277-
```
278-
279-
### Pytorch dataloader
280-
```py
281-
import sys
282-
import io
283-
import torch
284-
import torchvision.transforms as transforms
285-
from PIL import Image
286-
from osstorchconnector import OssIterableDataset, OssMapDataset
287-
288-
ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
289-
CONFIG_PATH = "/etc/oss-connector/config.json"
290-
CRED_PATH = "/root/.alibabacloud/credentials"
291-
OSS_URI = "oss://ossconnectorbucket/EnglistImg/Img/BadImag/Bmp/Sample001/"
292-
293-
294-
trans = transforms.Compose([
295-
transforms.Resize(256),
296-
transforms.CenterCrop(224),
297-
transforms.ToTensor(),
298-
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
299-
])
300-
301-
def transform(data):
302-
try:
303-
img = Image.open(io.BytesIO(data.read())).convert('RGB')
304-
val = trans(img)
305-
except Exception as e:
306-
raise e
307-
return val, data.key, data.label
308-
309-
# OssIterableDataset
310-
iterable_dataset = OssIterableDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH)
311-
loader = torch.utils.data.DataLoader(iterable_dataset, batch_size=256, num_workers=32, prefetch_factor=2)
312-
for i, (datas, keys, labels) in enumerate(loader):
313-
print(datas)
314-
print(keys)
315-
316-
# OssMapDataset with shuffle
317-
map_dataset = OssMapDataset.from_prefix(OSS_URI, endpoint=ENDPOINT, transform=transform, cred_path=CRED_PATH, config_path=CONFIG_PATH)
318-
loader = torch.utils.data.DataLoader(map_dataset, batch_size=256, num_workers=32, prefetch_factor=2, shuffle=True)
319-
for i, (datas, keys, labels) in enumerate(loader):
320-
print(datas)
321-
print(keys)
322-
```
323-
324-
When using with DataLoader, the main DataLoader worker responsible for listing from OSS or receiving objects from_prefix/from_manifest_file, all workers obtain their assigned objects from the main worker.
325-
This approach avoids issues of redundant listing and data reading (which may exist in other connectors), allowing better performance from multiple workers. When testing data download speed (excluding transform and other CPU-bound workload) with a large number of small files (e.g., ImageNet), it can exceed 10GB/s.
326-
327-
OssIterableDataset includes prefetch optimization by increasing concurrency. When the DataLoader is configured with multiple workers, the iteration order may not be deterministic (local order might be disrupted).
5+
## Overview
3286

329-
### Checkpoint
7+
OSS Connector for AI/ML contains some high-performance Python libraries specifically designed for AI and ML scenariosis, tailored to work with [Alibaba Cloud OSS (Object Storage Service)](https://www.alibabacloud.com/en/product/object-storage-service).
3308

331-
```py
332-
import torch
333-
from osstorchconnector import OssCheckpoint
9+
Currently, the OSS connector is composed of two libraries: OSS Model Connector and OSS Torch Connector.
33410

335-
ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
336-
CONFIG_PATH = "/etc/oss-connector/config.json"
337-
CRED_PATH = "/root/.alibabacloud/credentials"
11+
- [OSS Torch Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/torchconnector/introduction) is dedicated to AI training scenarios, including loading [datasets](https://pytorch.org/docs/stable/data.html#dataset-types) from OSS and loading/saving checkpoints from/to OSS.
33812

339-
checkpoint = OssCheckpoint(endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
13+
- [OSS Model Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/modelconnector/introduction) focuses on AI inference scenarios, loading large model files from OSS into local AI inference frameworks.
34014

341-
# read checkpoint
342-
CHECKPOINT_READ_URI = "oss://ossconnectorbucket/checkpoint/epoch.0"
343-
with checkpoint.reader(CHECKPOINT_READ_URI) as reader:
344-
state_dict = torch.load(reader)
15+
The core part of is OSS Connector for AI/ML is implemented in C++ using [PhotonLibOS](https://github.com/alibaba/PhotonLibOS). This repository only contains the code of Python.
34516

346-
# write checkpoint
347-
CHECKPOINT_WRITE_URI = "oss://ossconnectorbucket/checkpoint/epoch.1"
348-
with checkpoint.writer(CHECKPOINT_WRITE_URI) as writer:
349-
torch.save(state_dict, writer)
350-
```
17+
For details, please refer to [ossconnector.github.io](https://ossconnector.github.io/) or [aliyun.github.io/oss-connector-for-ai-ml](https://aliyun.github.io/oss-connector-for-ai-ml).
35118

352-
OssCheckpoint can be used for checkpoints, and also for high-speed uploading and downloading of arbitrary objects. In our testing environment, the download speed can exceed 15GB/s.
35319

35420
## Related
35521

‎docs/.nojekyll

Whitespace-only changes.

‎docs/LICENSE.md

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Copyright (c) 2024 aliyun.com
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
4+
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
5+
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
6+
permit persons to whom the Software is furnished to do so, subject to the following conditions:
7+
8+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
9+
Software.
10+
11+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
12+
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
13+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
14+
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎docs/README.md

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# OSS Connector for AI/ML
2+
3+
OSS Connector for AI/ML contains some high-performance Python libraries specifically designed for AI and ML scenariosis, tailored to work with [Alibaba Cloud OSS (Object Storage Service)](https://www.alibabacloud.com/en/product/object-storage-service).
4+
5+
Currently, the OSS connector is composed of two libraries: OSS Model Connector and OSS Torch Connector.
6+
7+
- [OSS Torch Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/torchconnector/introduction) is dedicated to AI training scenarios, including loading [datasets](https://pytorch.org/docs/stable/data.html#dataset-types) from OSS and loading/saving checkpoints from/to OSS.
8+
9+
- [OSS Model Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/modelconnector/introduction) focuses on AI inference scenarios, loading large model files from OSS into local AI inference frameworks.
10+
11+
The core part of is OSS Connector for AI/ML is implemented in C++ using [PhotonLibOS](https://github.com/alibaba/PhotonLibOS). This repository only contains the code of Python.
12+
13+
14+
## License
15+
16+
This project is licensed under the terms of the [MIT License](LICENSE.md).

0 commit comments

Comments
 (0)