Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](date_function) fix str_to_date function return wrong microsecond issue #47325

Open
wants to merge 7 commits into
base: branch-2.0
Choose a base branch
from

Conversation

Yulei-Yang
Copy link
Contributor

bp #47129

fix such issue both in legacy & nereids planner

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions bot added area/nereids area/planner Issues or PRs related to the query planner kind/test labels Jan 22, 2025
@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49812 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7335fc4db64d588c7ad57f73e42c6187c8ab175c, data reload: false

------ Round 1 ----------------------------------
q1	17665	4398	4391	4391
q2	2119	165	154	154
q3	10372	1938	1935	1935
q4	10117	1253	1349	1253
q5	8415	3971	3959	3959
q6	231	125	125	125
q7	2065	1618	1587	1587
q8	9301	2740	2724	2724
q9	10569	10230	10184	10184
q10	8660	3556	3540	3540
q11	422	243	254	243
q12	463	297	302	297
q13	18361	3991	4067	3991
q14	356	325	334	325
q15	514	463	456	456
q16	557	462	461	461
q17	1154	971	947	947
q18	7455	6845	6921	6845
q19	1714	1603	1550	1550
q20	517	330	297	297
q21	4396	4179	4151	4151
q22	493	400	397	397
Total cold run time: 115916 ms
Total hot run time: 49812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4340	4338	4341	4338
q2	323	227	223	223
q3	4168	4187	4140	4140
q4	2767	2751	2761	2751
q5	7241	7191	7126	7126
q6	239	121	123	121
q7	3220	2879	2886	2879
q8	4413	4485	4556	4485
q9	14403	14069	14063	14063
q10	4246	4296	4280	4280
q11	771	696	681	681
q12	1034	869	867	867
q13	6821	3813	3762	3762
q14	460	421	429	421
q15	516	466	457	457
q16	660	604	578	578
q17	3887	3888	3869	3869
q18	8843	8757	8909	8757
q19	1740	1656	1705	1656
q20	2370	2135	2138	2135
q21	8555	8534	8498	8498
q22	1069	911	939	911
Total cold run time: 82086 ms
Total hot run time: 76998 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 216629 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7335fc4db64d588c7ad57f73e42c6187c8ab175c, data reload: false

query1	940	431	384	384
query2	6561	2288	2304	2288
query3	6927	203	208	203
query4	24231	22090	22072	22072
query5	19737	6516	6584	6516
query6	286	223	250	223
query7	4153	315	316	315
query8	249	251	228	228
query9	3117	2719	2647	2647
query10	449	306	308	306
query11	15961	15042	15015	15015
query12	139	76	73	73
query13	1030	469	436	436
query14	17354	13906	13839	13839
query15	375	222	241	222
query16	6484	283	269	269
query17	1910	987	928	928
query18	914	326	318	318
query19	217	155	151	151
query20	100	97	104	97
query21	196	100	97	97
query22	5262	5190	5155	5155
query23	34695	33692	33783	33692
query24	7890	6278	6361	6278
query25	520	434	437	434
query26	1274	175	168	168
query27	2496	308	304	304
query28	6081	2287	2279	2279
query29	2881	2803	2643	2643
query30	250	174	168	168
query31	983	785	758	758
query32	75	66	64	64
query33	463	271	280	271
query34	877	485	502	485
query35	1135	964	913	913
query36	1461	1375	1152	1152
query37	173	60	64	60
query38	3128	3008	2961	2961
query39	1388	1329	1333	1329
query40	305	95	102	95
query41	49	39	38	38
query42	82	87	79	79
query43	671	637	531	531
query44	1179	738	725	725
query45	246	243	232	232
query46	1248	995	980	980
query47	1780	1684	1638	1638
query48	516	420	410	410
query49	669	378	394	378
query50	859	660	617	617
query51	4869	4759	4774	4759
query52	99	82	100	82
query53	231	195	205	195
query54	2716	2539	2505	2505
query55	86	86	87	86
query56	226	210	218	210
query57	1257	1320	1210	1210
query58	232	206	217	206
query59	3466	3498	3401	3401
query60	223	217	213	213
query61	100	96	101	96
query62	827	474	481	474
query63	205	185	180	180
query64	3527	1548	1466	1466
query65	3692	3571	3579	3571
query66	774	402	401	401
query67	15923	15444	14951	14951
query68	9265	667	677	667
query69	507	298	275	275
query70	1767	1448	1332	1332
query71	398	319	313	313
query72	6787	4976	4897	4897
query73	785	319	326	319
query74	6305	5943	5798	5798
query75	4909	3746	3800	3746
query76	5016	1161	1213	1161
query77	732	256	268	256
query78	12880	14208	13482	13482
query79	12136	659	657	657
query80	956	406	395	395
query81	507	243	241	241
query82	823	101	107	101
query83	170	133	135	133
query84	259	71	72	71
query85	781	344	326	326
query86	336	289	287	287
query87	3248	3042	3059	3042
query88	5298	2293	2287	2287
query89	382	294	290	290
query90	1863	217	221	217
query91	162	130	130	130
query92	59	55	54	54
query93	4773	548	584	548
query94	690	211	215	211
query95	2064	2020	2028	2020
query96	659	334	336	334
query97	6606	6525	6552	6525
query98	226	221	202	202
query99	2589	976	842	842
Total cold run time: 322357 ms
Total hot run time: 216629 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7335fc4db64d588c7ad57f73e42c6187c8ab175c, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.25	0.05	0.05
query4	1.77	0.07	0.08
query5	0.53	0.52	0.52
query6	1.23	0.63	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.52	0.49	0.48
query10	0.54	0.53	0.53
query11	0.13	0.09	0.09
query12	0.12	0.09	0.10
query13	0.63	0.63	0.61
query14	0.80	0.82	0.80
query15	0.79	0.78	0.76
query16	0.38	0.36	0.36
query17	1.03	1.01	0.98
query18	0.22	0.27	0.24
query19	1.94	1.83	1.78
query20	0.02	0.01	0.01
query21	15.46	0.56	0.55
query22	2.36	2.17	2.00
query23	17.40	1.03	0.85
query24	5.77	0.98	1.64
query25	0.36	0.09	0.07
query26	0.66	0.15	0.17
query27	0.04	0.04	0.04
query28	6.92	0.80	0.72
query29	12.60	2.40	2.39
query30	0.63	0.53	0.55
query31	2.81	0.40	0.37
query32	3.36	0.51	0.49
query33	3.11	3.08	3.07
query34	15.24	4.84	4.84
query35	4.86	4.89	4.87
query36	1.07	1.03	1.04
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.01	0.02
Total cold run time: 104.11 s
Total hot run time: 31.28 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7335fc4db64d588c7ad57f73e42c6187c8ab175c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@Yulei-Yang
Copy link
Contributor Author

run p0

@Yulei-Yang
Copy link
Contributor Author

run external

@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49006 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aafa85f3eb67d030e37d9167adb791f6862b072d, data reload: false

------ Round 1 ----------------------------------
q1	17879	4468	4375	4375
q2	2067	159	151	151
q3	10517	1877	1880	1877
q4	10375	1263	1335	1263
q5	8532	3883	3952	3883
q6	240	123	127	123
q7	2048	1601	1626	1601
q8	9547	2731	2704	2704
q9	13777	9880	9711	9711
q10	8662	3472	3464	3464
q11	420	252	249	249
q12	470	296	302	296
q13	18334	3941	4029	3941
q14	354	320	324	320
q15	516	460	454	454
q16	535	476	464	464
q17	1125	955	969	955
q18	7231	6898	7059	6898
q19	1700	1559	1507	1507
q20	537	300	306	300
q21	4422	4088	4082	4082
q22	518	416	388	388
Total cold run time: 119806 ms
Total hot run time: 49006 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4325	4299	4299
q2	321	228	220	220
q3	4210	4135	4144	4135
q4	2756	2734	2756	2734
q5	7177	7123	7106	7106
q6	238	119	121	119
q7	3227	2864	2824	2824
q8	4361	4468	4480	4468
q9	13684	13688	13673	13673
q10	4225	4256	4257	4256
q11	771	675	674	674
q12	1035	857	835	835
q13	6830	3764	3747	3747
q14	442	416	440	416
q15	499	458	465	458
q16	629	604	589	589
q17	3828	3954	3850	3850
q18	8776	8767	8765	8765
q19	1699	1678	1625	1625
q20	2404	2095	2103	2095
q21	8559	8541	8562	8541
q22	1012	889	939	889
Total cold run time: 80993 ms
Total hot run time: 76318 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 215423 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aafa85f3eb67d030e37d9167adb791f6862b072d, data reload: false

query1	936	393	415	393
query2	6553	2227	2253	2227
query3	6912	201	202	201
query4	23333	21747	21874	21747
query5	19742	6501	6527	6501
query6	288	222	239	222
query7	4324	311	313	311
query8	286	260	273	260
query9	3100	2680	2607	2607
query10	439	309	311	309
query11	15996	15116	15271	15116
query12	139	76	75	75
query13	1027	461	445	445
query14	17575	13688	13544	13544
query15	366	225	238	225
query16	6469	284	267	267
query17	1752	914	908	908
query18	886	317	321	317
query19	215	149	153	149
query20	102	100	100	100
query21	193	97	96	96
query22	5185	5108	5043	5043
query23	34536	33452	33672	33452
query24	7844	6335	6308	6308
query25	537	436	432	432
query26	1270	167	166	166
query27	2440	304	300	300
query28	6068	2275	2258	2258
query29	2853	2859	2752	2752
query30	240	172	174	172
query31	966	710	751	710
query32	74	65	60	60
query33	465	265	267	265
query34	874	507	486	486
query35	1134	910	951	910
query36	1246	1194	1361	1194
query37	180	61	63	61
query38	3093	2946	2927	2927
query39	1391	1372	1365	1365
query40	318	100	99	99
query41	40	37	37	37
query42	95	91	86	86
query43	642	570	573	570
query44	1161	723	727	723
query45	246	232	229	229
query46	1255	955	973	955
query47	1859	2003	2026	2003
query48	515	414	414	414
query49	649	393	394	393
query50	888	645	648	645
query51	4855	4700	4835	4700
query52	99	83	75	75
query53	226	182	193	182
query54	2681	2533	2520	2520
query55	99	86	90	86
query56	232	219	211	211
query57	1350	1206	1301	1206
query58	225	221	203	203
query59	3618	3316	3298	3298
query60	215	199	211	199
query61	100	92	101	92
query62	827	555	489	489
query63	209	180	180	180
query64	3540	1640	1386	1386
query65	3675	3602	3581	3581
query66	1292	431	426	426
query67	17590	17049	16671	16671
query68	12344	1114	707	707
query69	1856	260	269	260
query70	2572	1432	1499	1432
query71	1638	325	359	325
query72	7922	4922	4763	4763
query73	768	316	316	316
query74	6275	5809	5819	5809
query75	5334	3776	3718	3718
query76	6046	1128	1192	1128
query77	1027	253	272	253
query78	12525	11713	12670	11713
query79	16382	639	631	631
query80	1610	381	372	372
query81	470	237	233	233
query82	489	100	95	95
query83	186	135	136	135
query84	262	70	69	69
query85	947	314	310	310
query86	336	290	288	288
query87	3265	3056	2966	2966
query88	4227	2297	2300	2297
query89	482	284	279	279
query90	2762	208	209	208
query91	158	128	125	125
query92	55	53	52	52
query93	6796	574	598	574
query94	1303	209	214	209
query95	1914	1949	1919	1919
query96	660	331	326	326
query97	6441	6387	6304	6304
query98	227	203	200	200
query99	2850	867	894	867
Total cold run time: 339659 ms
Total hot run time: 215423 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aafa85f3eb67d030e37d9167adb791f6862b072d, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.78	0.08	0.07
query5	0.54	0.52	0.53
query6	1.23	0.61	0.68
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.53	0.49	0.48
query10	0.54	0.53	0.54
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.61	0.61	0.60
query14	0.78	0.81	0.79
query15	0.79	0.76	0.75
query16	0.35	0.36	0.37
query17	1.01	1.01	1.01
query18	0.23	0.24	0.25
query19	1.95	1.84	1.84
query20	0.01	0.00	0.00
query21	15.47	0.56	0.55
query22	2.22	2.50	1.76
query23	17.53	0.99	0.82
query24	7.84	0.96	1.12
query25	0.40	0.11	0.06
query26	0.80	0.16	0.15
query27	0.05	0.03	0.04
query28	5.23	0.78	0.77
query29	12.73	2.35	2.39
query30	0.61	0.59	0.53
query31	2.82	0.40	0.37
query32	3.35	0.50	0.49
query33	3.07	3.05	3.10
query34	15.26	4.79	4.78
query35	4.83	4.84	4.83
query36	1.06	1.02	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.66 s
Total hot run time: 30.85 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit aafa85f3eb67d030e37d9167adb791f6862b072d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/nereids area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants