Skip to content

Commit d547994

Browse files
alphaprinzalphaprinz
committed
Integrate S3Select to provide "select object content"-like request.
For more information see /docs/s3select.md and /docs/S3Select_Build.md Signed-off-by: Amit Prinz Setter <[email protected]>
1 parent 7316398 commit d547994

20 files changed

+853
-6
lines changed

‎.gitmodules

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "s3select"]
2+
path = submodules/s3select
3+
url = https://github.com/ceph/s3select

‎Makefile

+1-1
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ builder: assert-container-engine
6666

6767
base: builder
6868
@echo "\033[1;34mStarting Base $(CONTAINER_ENGINE) build.\033[0m"
69-
$(DOCKER_BUILDKIT) $(CONTAINER_ENGINE) build $(CPUSET) -f src/deploy/NVA_build/Base.Dockerfile $(CACHE_FLAG) $(NETWORK_FLAG) -t noobaa-base . $(REDIRECT_STDOUT)
69+
$(DOCKER_BUILDKIT) $(CONTAINER_ENGINE) build $(CPUSET) --build-arg BUILD_S3SELECT -f src/deploy/NVA_build/Base.Dockerfile $(CACHE_FLAG) $(NETWORK_FLAG) -t noobaa-base . $(REDIRECT_STDOUT)
7070
$(CONTAINER_ENGINE) tag noobaa-base $(NOOBAA_BASE_TAG)
7171
@echo "\033[1;32mBase done.\033[0m"
7272
.PHONY: base

‎docs/design/s3select.md

+123
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
2+
# S3 Select
3+
4+
5+
## Goal
6+
7+
Provide a feature to run S3 Select queries as close as possible to AWS feature.
8+
9+
10+
## Scenarios
11+
12+
A client sends an S3 Select request according to AWS specification (see [SelectObjectContent - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html)).
13+
14+
The endpoint will retrieve the relevant file (AKA key), process the query, and reply according to specification (see [Appendix: SelectObjectContent Response - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTSelectObjectAppendix.html)).
15+
16+
17+
## Technical Details
18+
19+
20+
### Query Processing Implementation
21+
22+
Use external implementation, as implementing in-house is not a valid option.
23+
24+
25+
26+
* S3Select ([https://github.com/ceph/s3select](https://github.com/ceph/s3select)) implements running SQL on character stream in C++. This requires napi, git and build changes. This is the chosen implementation.
27+
* Partiql ([https://partiql.org/](https://partiql.org/)) is available only in Kotlin implementation. This requires starting a JVM in the endpoint to process SQL, which is cumbersome.
28+
* [https://github.com/partiql/partiql-lang-rust](https://github.com/partiql/partiql-lang-rust) can be considered in the future, but for now it doesn’t have a full implementation yet.
29+
30+
31+
### New S3 OP
32+
33+
A new s3 op, post_object_select will be added.
34+
35+
There are two variations of files to consider:
36+
37+
38+
39+
1. csv, json - These are “stream-friendly”, they are read once from start to finish. So we can get their stream like standard get_object s3 op, pipe the stream to s3select to process, and pipe the result to http res object.
40+
2. Parquet
41+
1. Processing a parquet file requires random seeking. In other words, at any time during the processing of the SQL, s3select can require to read any address of the file. So the stream-piping is not suitable.
42+
2. A general, currently undesigned, solution is to channel the request upwards, back to a callback in post_object_select, which will ask object_sdk for the required address and send it back to s3select.
43+
3. In the simple case of FS namespace buckets, if the required file resides in the endpoint host, the file path (rather than stream) can be sent to s3select for processing. This allows processing Parquet files stored in namespace buckets.
44+
4. Parquet requires Arrow, a library for processing Parquet files.
45+
5. Initial feature will not support Parquet.
46+
47+
48+
### HTTP Response
49+
50+
Response should conform to specification (see [Appendix: SelectObjectContent Response - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTSelectObjectAppendix.html)).
51+
52+
We can adapt S3Select example code that already has implementation to add crc and headers for each chunk.
53+
54+
55+
### Napi
56+
57+
New native code that will implement a transform stream (similar to coder/chunker). This stream will have these functions:
58+
59+
60+
61+
* write(byte array) - that will reply with rows matching the query.
62+
* flush() - that will reply with SQL-aggregate (eg count, sum) result.
63+
64+
Preferably, stream will not encode/decode bytes into/from strings and will not copy buffers.
65+
66+
67+
### Git and Build
68+
69+
70+
71+
1. S3Select source
72+
73+
S3Select is a git repository. It has 3 dependencies:
74+
75+
76+
77+
* Fast Cpp Csv Parser ([https://github.com/ben-strasser/fast-cpp-csv-parser](https://github.com/ben-strasser/fast-cpp-csv-parser))
78+
* RapidJson parser ([https://github.com/Tencent/rapidjson](https://github.com/Tencent/rapidjson))
79+
* Boost
80+
81+
The parser git repositories which are submodules of s3select repository.
82+
83+
Boost is assumed by s3select make to be installed on the builder machine.
84+
85+
86+
87+
2. noobaa-core repository
88+
89+
In noobaa-core repository, a new submodules directory and .gitmodules file are added.
90+
91+
It will have the s3select and boost repositories as submodules, and the two parsers recursively, inside the s3select submodule.
92+
93+
94+
95+
3. Docker build
96+
97+
In docker build, these repositories are fetched and node-gyp uses the sources for compilation.
98+
99+
Only necessary boost submodules are fetched.
100+
101+
All repositories are checkout to a specific commit/tag so updates won’t affect us directly.
102+
103+
When updates are needed, the checkout command needs to be updated to a new commit/tag.
104+
105+
106+
107+
4. Native build
108+
109+
Running native build should still be possible.
110+
111+
Git submodules are needed to be fetched manually (once).
112+
113+
114+
### Tests
115+
116+
117+
118+
* S3 ceph tests
119+
* ..
120+
121+
122+
### Upstream Docs
123+

‎docs/dev_guide/S3Select_Build.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# How to build S3 Select
2+
3+
S3 Select is a native library integrated into noobaa-core.
4+
In noobaa-core git repository, it is present under as git submodule in /submodules/s3select (as defined in .gitmodules file).
5+
6+
## Local Native Build
7+
8+
By default, S3 Select is not build in local native default.
9+
This is to streamline builds of developers who don't need S3 Select feature.
10+
In other words, running "npm run build" and executing S3 Select code will fail.
11+
In order to build native locally with S3 Select, you need to:
12+
13+
1. Init and clone submodule.
14+
15+
These git commands will fetch necessary code into your git reporosity:
16+
-`git submodule init (needed only once)`
17+
-`git submodule update (needed each update of submodule)`
18+
These two commands can be combined:
19+
`git submodule update --init`
20+
21+
Since S3Select has two submodules of its own, you need to repeat above commands.
22+
23+
All of the above can be done with:
24+
`git submodule update --init --recursive`
25+
26+
2. Install "boost-devel" package.
27+
The "boost-devel" package is assumed to be installed by local native build.
28+
It is a relatively widespread package, available in general package repository.
29+
Eg, on a Fedora-based linux:
30+
`yum install boost-devel`
31+
32+
3. Run build command with BUILD_S3SELECT enabled in GYP:
33+
`GYP_DEFINES=BUILD_S3SELECT=1 npm run build`
34+
or, equivalently:
35+
`GYP_DEFINES=BUILD_S3SELECT=1 node-gyp rebuild`
36+
37+
## Docker Build
38+
S3Select is enabled by defualt for docker build.
39+
If you wish to explicitly enable/disable s3select in docker build, you can use BUILD_S3SELECT env parameter. Eg-
40+
`BUILD_S3SELECT=0 make noobaa NOOBAA_TAG=noobaa-core:select`
41+
42+
## Test Native Code
43+
You can test native code with the provide s3select.js. Eg-
44+
`echo -e "1,2,3\n4,5,6" | node noobaa-core/src/tools/s3select.js --query "SELECT sum(int(_2)) from stdin;"`
45+
Which is equivalent to-
46+
`echo -e "1,2,3\n4,5,6" | node src/tools/s3select.js --query "SELECT sum(int(_2)) from stdin;" --input_format CSV --record_delimiter $'\n' --field_delimiter , --file_header_info IGNORE`
47+
48+

‎src/deploy/NVA_build/Base.Dockerfile

+8-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,14 @@ RUN npm install --production && \
1919
##############################################################
2020
COPY ./binding.gyp .
2121
COPY ./src/native ./src/native/
22-
RUN npm run build:native
22+
COPY ./src/deploy/NVA_build/clone_submodule.sh ./src/deploy/NVA_build/
23+
COPY ./src/deploy/NVA_build/clone_s3select_submodules.sh ./src/deploy/NVA_build/
24+
ARG BUILD_S3SELECT=1
25+
#Clone S3Select and its two submodules, but only if BUILD_S3SELECT=1.
26+
RUN ./src/deploy/NVA_build/clone_s3select_submodules.sh
27+
#Pass BUILD_S3SELECT down to GYP native build.
28+
#S3Select will be built only if this parameter is equal to "1".
29+
RUN GYP_DEFINES=BUILD_S3SELECT=$BUILD_S3SELECT npm run build:native
2330

2431
##############################################################
2532
# Layers:

‎src/deploy/NVA_build/NooBaa.Dockerfile

+1
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ ENV ENDPOINT_NODE_OPTIONS ''
5959

6060
RUN dnf install -y epel-release
6161
RUN dnf install -y -q bash \
62+
boost \
6263
lsof \
6364
procps \
6465
openssl \

‎src/deploy/NVA_build/builder.Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ LABEL maintainer="Liran Mauda ([email protected])"
1111
# dnf clean all
1212
RUN dnf update -y -q --nobest && \
1313
dnf clean all
14-
RUN dnf install -y -q wget unzip which vim python2 python3 && \
14+
RUN dnf install -y -q wget unzip which vim python2 python3 boost-devel && \
1515
dnf group install -y -q "Development Tools" && \
1616
dnf clean all
1717
RUN alternatives --set python /usr/bin/python3

‎src/deploy/NVA_build/clone_s3select_submodules.sh

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#Clone S3Select and its two submodules, but only if BUILD_S3SELECT=1.
2+
./src/deploy/NVA_build/clone_submodule.sh submodules/s3select https://github.com/ceph/s3select 58875a5ee76261cc0a3d943bb168f9f9292c34a4 BUILD_S3SELECT
3+
./src/deploy/NVA_build/clone_submodule.sh submodules/s3select/rapidjson https://github.com/Tencent/rapidjson fcb23c2dbf561ec0798529be4f66394d3e4996d8 BUILD_S3SELECT
4+
./src/deploy/NVA_build/clone_submodule.sh submodules/s3select/include/csvparser https://github.com/ben-strasser/fast-cpp-csv-parser 5a417973b4cea674a5e4a3b88a23098a2ab75479 BUILD_S3SELECT
5+

‎src/deploy/NVA_build/clone_submodule.sh

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
enabled=`printf '%s\n' "${!4}"`
2+
echo $enabled
3+
if [ -z "$enabled" ]; then
4+
echo "not en"
5+
exit 0
6+
fi
7+
if [ $enabled -ne 1 ]; then
8+
echo "exit"
9+
exit 0
10+
fi
11+
echo "done"
12+
mkdir -p $1
13+
cd $1
14+
git init
15+
git remote add origin $2
16+
git fetch origin $3 --depth=1
17+
git reset --hard FETCH_HEAD
18+
rm -rf .git

‎src/endpoint/s3/ops/s3_post_object_select.js

+118
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
/* Copyright (C) 2016 NooBaa */
2+
'use strict';
3+
4+
const dbg = require('../../../util/debug_module')(__filename);
5+
const S3Error = require('../s3_errors').S3Error;
6+
const s3_utils = require('../s3_utils');
7+
const { S3SelectStream } = require('../../../util/s3select');
8+
const nb_native = require('../../../util/nb_native');
9+
const stream_utils = require('../../../util/stream_utils');
10+
11+
/**
12+
* https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html
13+
*/
14+
async function post_object_select(req, res) {
15+
16+
if (!nb_native().S3Select) {
17+
throw new S3Error(S3Error.S3SelectNotCompiled);
18+
}
19+
20+
req.object_sdk.setup_abort_controller(req, res);
21+
const agent_header = req.headers['user-agent'];
22+
const noobaa_trigger_agent = agent_header && agent_header.includes('exec-env/NOOBAA_FUNCTION');
23+
const encryption = s3_utils.parse_encryption(req);
24+
const http_req_select_params = req.body.SelectObjectContentRequest;
25+
26+
const md_params = {
27+
bucket: req.params.bucket,
28+
key: req.params.key,
29+
version_id: req.query.versionId,
30+
encryption,
31+
};
32+
const object_md = await req.object_sdk.read_object_md(md_params);
33+
34+
const params = {
35+
object_md,
36+
obj_id: object_md.obj_id,
37+
bucket: req.params.bucket,
38+
key: req.params.key,
39+
content_type: object_md.content_type,
40+
noobaa_trigger_agent,
41+
encryption,
42+
};
43+
44+
//handle ScanRange
45+
if (Array.isArray(http_req_select_params.ScanRange)) {
46+
const scan_range = http_req_select_params.ScanRange[0];
47+
if (scan_range.Start) {
48+
params.start = Number(scan_range.Start);
49+
}
50+
if (scan_range.End) {
51+
if (scan_range.Start) {
52+
params.end = Number(scan_range.End);
53+
} else {
54+
//if only End is specified, start from {End} bytes from the end.
55+
params.start = object_md.size - (Number(scan_range.End));
56+
}
57+
}
58+
}
59+
60+
//prepare s3select stream
61+
const input_serialization = http_req_select_params.InputSerialization[0];
62+
let input_format = null;
63+
if (input_serialization.CSV) {
64+
input_format = 'CSV';
65+
} else if (input_serialization.JSON) {
66+
input_format = 'JSON';
67+
} else {
68+
throw new S3Error(S3Error.MissingInputSerialization);
69+
}
70+
71+
//currently s3select can only output in the same format as input format
72+
if (Array.isArray(http_req_select_params.OutputSerialization)) {
73+
const output_serialization = http_req_select_params.OutputSerialization[0];
74+
if ((output_serialization.CSV && input_format !== 'CSV') ||
75+
(output_serialization.JSON && input_format !== 'JSON')) {
76+
throw new S3Error(S3Error.OutputInputFormatMismatch);
77+
}
78+
}
79+
80+
const select_args = {
81+
query: http_req_select_params.Expression[0],
82+
input_format: input_format,
83+
input_serialization_format: http_req_select_params.InputSerialization[0][input_format][0],
84+
records_header_buf: S3SelectStream.records_message_headers
85+
};
86+
const s3select = new S3SelectStream(select_args);
87+
dbg.log3("select_args = ", select_args);
88+
89+
//pipe s3select result into http result
90+
stream_utils.pipeline([s3select, res], true /*res is a write stream, no need for resume*/);
91+
92+
//send s3select pipe to read_object_stream.
93+
//in some cases (currently nsfs) it will pipe object stream into our pipe (s3select)
94+
const read_stream = await req.object_sdk.read_object_stream(params, s3select);
95+
if (read_stream) {
96+
// if read_stream supports closing, then we handle abort cases such as http disconnection
97+
// by calling the close method to stop it from buffering more data which will go to waste.
98+
if (read_stream.close) {
99+
req.object_sdk.add_abort_handler(() => read_stream.close());
100+
}
101+
read_stream.on('error', err => {
102+
dbg.error('read stream error:', err, req.path);
103+
res.destroy(err);
104+
});
105+
//in other cases, we need to pipe the read stream ourselves
106+
stream_utils.pipeline([read_stream, s3select], true /*no need to resume s3select*/);
107+
}
108+
}
109+
110+
module.exports = {
111+
handler: post_object_select,
112+
body: {
113+
type: 'xml',
114+
},
115+
reply: {
116+
type: 'raw',
117+
},
118+
};

‎src/endpoint/s3/s3_errors.js

+20
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,26 @@ S3Error.InvalidEncodingType = Object.freeze({
512512
http_code: 400,
513513
});
514514

515+
////////////////////////////////////////////////////////////////
516+
// S3 Select //
517+
////////////////////////////////////////////////////////////////
518+
519+
S3Error.S3SelectNotCompiled = Object.freeze({
520+
code: 'S3SelectNotCompiled',
521+
message: 'This server was not compiled with S3 Select support. Recompile with BUILD_S3SELECT=1.',
522+
http_code: 501,
523+
});
524+
S3Error.MissingInputSerialization = Object.freeze({
525+
code: 'MissingRequiredParameter',
526+
message: 'InputSerialization is required. Please check the service documentation and try again.',
527+
http_code: 400,
528+
});
529+
S3Error.OutputInputFormatMismatch = Object.freeze({
530+
code: 'OutputInputFormatMismatch',
531+
message: 'OutputSerialization format must match InputSerializatoin format.',
532+
http_code: 501,
533+
});
534+
515535
S3Error.RPC_ERRORS_TO_S3 = Object.freeze({
516536
UNAUTHORIZED: S3Error.AccessDenied,
517537
BAD_REQUEST: S3Error.BadRequest,

0 commit comments

Comments
 (0)