[Enhancement](lock) Optimize CatalogRecycleBin lock granularity to improve concurrency

[felixwluo]

What problem does this PR solve?

Issue Number: close #47997

Ideas

1. Lock particle size optimization

• The original question: Synchronized method locks the entire object, and all operations must be mutually exclusive.
• Optimization solution:
  Use read and write lock to separate read and write operations:
  1-1. Read operations (query, check status, etc.) use read locks.
  1-2. Write operations (add, delete, etc.) use write locks.
  1-3. Multiple read operations can be executed concurrently.

2. Key operations refactoring

• The original question: Time-consuming operations (e.g., onErasePartition) are executed under the lock, resulting in prolonged lock holding time.
• Optimization solution:
  Divide the operation into three stages
  2-1. Before acquiring the lock: Preparation work.
  2-2. While holding the lock: Perform only necessary data structure modifications.
  2-3. After releasing the lock: Execute time-consuming operations.

3. Implementation strategy
   3-1. Data preparation stage:
   - Collect the data to be processed before acquiring the lock
   - Pre-allocate necessary data structures

   3-2. Critical section operations:
   - Modify only shared data structures
   - Keep the lock hold time as short as possible

   3-3. Subsequent processing:
   - Move I/O operations outside the lock
   - Log-related operations should also be outside the lock

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[Thearas]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[felixwluo]

run buildall

[doris-robot]

TPC-H: Total hot run time: 31505 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93108059113e4eb19edb4eb34291155ea60fc56a, data reload: false

------ Round 1 ----------------------------------
q1	17621	5223	5074	5074
q2	2048	308	171	171
q3	10446	1255	734	734
q4	10203	1012	534	534
q5	7553	2378	2318	2318
q6	186	166	129	129
q7	907	740	601	601
q8	9294	1288	1028	1028
q9	4900	4618	4875	4618
q10	6831	2305	1904	1904
q11	493	270	262	262
q12	344	359	224	224
q13	17768	3689	3104	3104
q14	233	220	205	205
q15	525	475	462	462
q16	640	615	590	590
q17	568	884	331	331
q18	6693	6319	6206	6206
q19	1424	957	559	559
q20	325	321	185	185
q21	2721	2151	1956	1956
q22	361	343	310	310
Total cold run time: 102084 ms
Total hot run time: 31505 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5150	5133	5455	5133
q2	240	330	245	245
q3	2196	2691	2308	2308
q4	1463	1823	1419	1419
q5	4252	4135	4190	4135
q6	200	164	122	122
q7	1870	1816	1680	1680
q8	2630	2677	2579	2579
q9	7183	7187	7227	7187
q10	3018	3193	2819	2819
q11	601	516	488	488
q12	724	804	673	673
q13	3431	3857	3320	3320
q14	283	284	281	281
q15	524	472	464	464
q16	659	698	640	640
q17	1137	1619	1344	1344
q18	7634	7362	7483	7362
q19	771	784	828	784
q20	1975	2049	1914	1914
q21	5377	5047	4654	4654
q22	624	565	577	565
Total cold run time: 51942 ms
Total hot run time: 50116 ms

[doris-robot]

TPC-DS: Total hot run time: 189844 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93108059113e4eb19edb4eb34291155ea60fc56a, data reload: false

query1	1297	989	957	957
query2	6276	1928	1859	1859
query3	10951	4581	4397	4397
query4	56556	24469	23614	23614
query5	5000	536	481	481
query6	337	213	180	180
query7	4978	503	292	292
query8	317	254	233	233
query9	6245	2519	2527	2519
query10	427	321	257	257
query11	15341	15389	14816	14816
query12	163	106	106	106
query13	1111	526	392	392
query14	10713	6472	6284	6284
query15	216	199	191	191
query16	7102	648	481	481
query17	1049	829	569	569
query18	1566	436	306	306
query19	203	208	171	171
query20	128	129	120	120
query21	218	124	107	107
query22	4318	4353	4421	4353
query23	33882	33318	33364	33318
query24	5629	2443	2423	2423
query25	463	492	405	405
query26	709	274	161	161
query27	1742	474	348	348
query28	2836	2470	2433	2433
query29	570	593	456	456
query30	214	191	160	160
query31	884	881	817	817
query32	79	60	60	60
query33	445	356	300	300
query34	767	853	503	503
query35	813	834	747	747
query36	1015	962	909	909
query37	125	99	76	76
query38	4293	4478	4251	4251
query39	1506	1445	1432	1432
query40	210	116	110	110
query41	54	49	51	49
query42	120	108	106	106
query43	507	518	498	498
query44	1348	799	791	791
query45	182	175	172	172
query46	876	1071	678	678
query47	1862	1895	1786	1786
query48	395	444	337	337
query49	688	535	412	412
query50	720	752	422	422
query51	4364	4310	4189	4189
query52	110	106	96	96
query53	227	264	199	199
query54	470	480	428	428
query55	83	77	87	77
query56	266	270	256	256
query57	1162	1218	1140	1140
query58	248	247	236	236
query59	2818	2859	2612	2612
query60	286	296	260	260
query61	118	118	121	118
query62	744	729	688	688
query63	228	199	198	198
query64	1836	1041	681	681
query65	3194	3147	3152	3147
query66	720	390	298	298
query67	15916	15732	15222	15222
query68	5515	782	524	524
query69	526	297	262	262
query70	1206	1124	1118	1118
query71	455	296	254	254
query72	5960	3690	3800	3690
query73	1216	750	362	362
query74	9015	9203	8917	8917
query75	3299	3149	2690	2690
query76	3920	1173	753	753
query77	539	374	284	284
query78	10018	10003	9325	9325
query79	2307	796	590	590
query80	605	539	481	481
query81	498	282	243	243
query82	483	133	95	95
query83	179	169	148	148
query84	285	97	78	78
query85	839	356	303	303
query86	405	307	279	279
query87	4395	4436	4347	4347
query88	3854	2208	2179	2179
query89	403	323	285	285
query90	1844	194	189	189
query91	133	139	106	106
query92	77	57	57	57
query93	1852	1010	581	581
query94	650	407	304	304
query95	334	267	256	256
query96	475	562	275	275
query97	2777	2834	2718	2718
query98	216	211	207	207
query99	1356	1392	1254	1254
Total cold run time: 295872 ms
Total hot run time: 189844 ms

[doris-robot]

ClickBench: Total hot run time: 30.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93108059113e4eb19edb4eb34291155ea60fc56a, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.40	0.42	0.40
query6	1.17	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.61	0.52	0.51
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.61	0.60
query14	2.72	2.70	2.70
query15	0.93	0.86	0.83
query16	0.36	0.38	0.38
query17	1.07	1.06	1.01
query18	0.21	0.20	0.21
query19	1.94	1.78	2.06
query20	0.01	0.01	0.01
query21	15.35	0.88	0.54
query22	0.75	1.29	0.71
query23	14.78	1.39	0.67
query24	11.76	1.03	0.39
query25	0.34	0.20	0.07
query26	0.96	0.18	0.15
query27	0.05	0.04	0.05
query28	6.20	0.79	0.44
query29	12.55	3.99	3.24
query30	0.25	0.09	0.05
query31	2.81	0.58	0.39
query32	3.23	0.55	0.47
query33	3.02	3.00	3.02
query34	15.89	5.15	4.46
query35	4.56	4.54	4.58
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.59 s
Total hot run time: 30.09 s

[Vallishp]

Can synchronized removed for this function?

[felixwluo]

That’s fine. I still need to refactor. Thank you.

[felixwluo]

run buildall

[doris-robot]

TPC-H: Total hot run time: 31630 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56a5a982ebdc8007d4b4a20fa4ba47ad36fc2f88, data reload: false

------ Round 1 ----------------------------------
q1	17586	5418	5088	5088
q2	2053	301	171	171
q3	10417	1262	752	752
q4	10204	1007	531	531
q5	7520	2488	2336	2336
q6	187	170	135	135
q7	916	752	594	594
q8	9299	1362	1203	1203
q9	4866	4698	4664	4664
q10	6845	2316	1882	1882
q11	483	282	255	255
q12	355	363	225	225
q13	17761	3732	3070	3070
q14	227	228	206	206
q15	519	472	445	445
q16	618	604	564	564
q17	590	879	333	333
q18	6783	6369	6168	6168
q19	1220	956	574	574
q20	317	323	198	198
q21	2801	2133	1931	1931
q22	364	336	305	305
Total cold run time: 101931 ms
Total hot run time: 31630 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5126	5209	5158	5158
q2	255	337	229	229
q3	2209	2705	2331	2331
q4	1428	1824	1390	1390
q5	4213	4157	4144	4144
q6	213	170	128	128
q7	1886	1816	1686	1686
q8	2618	2681	2637	2637
q9	7285	7265	7158	7158
q10	3014	3221	2799	2799
q11	575	523	498	498
q12	692	737	672	672
q13	3446	3895	3238	3238
q14	267	288	274	274
q15	503	472	475	472
q16	645	689	641	641
q17	1166	1624	1323	1323
q18	7605	7331	7448	7331
q19	797	808	894	808
q20	1998	2062	1916	1916
q21	5562	5095	4697	4697
q22	629	590	551	551
Total cold run time: 52132 ms
Total hot run time: 50081 ms

[doris-robot]

TPC-DS: Total hot run time: 190797 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56a5a982ebdc8007d4b4a20fa4ba47ad36fc2f88, data reload: false

query1	1305	973	958	958
query2	6137	1966	1902	1902
query3	11118	4532	4633	4532
query4	57341	25622	23478	23478
query5	5144	505	488	488
query6	392	182	188	182
query7	5302	497	286	286
query8	324	235	237	235
query9	7112	2537	2534	2534
query10	431	310	253	253
query11	15450	15237	14942	14942
query12	155	106	105	105
query13	1256	563	416	416
query14	10155	6344	6383	6344
query15	215	206	191	191
query16	7123	685	481	481
query17	1076	750	596	596
query18	1549	417	327	327
query19	209	204	183	183
query20	147	136	124	124
query21	208	131	102	102
query22	4202	4605	4313	4313
query23	34119	33334	33316	33316
query24	6771	2466	2426	2426
query25	486	461	409	409
query26	689	295	152	152
query27	2166	509	361	361
query28	3094	2500	2450	2450
query29	597	548	429	429
query30	209	182	149	149
query31	879	875	792	792
query32	73	69	68	68
query33	459	359	304	304
query34	735	864	515	515
query35	801	843	778	778
query36	942	1006	897	897
query37	119	97	81	81
query38	4342	4270	4285	4270
query39	1488	1434	1430	1430
query40	227	117	100	100
query41	51	55	48	48
query42	123	112	115	112
query43	524	516	510	510
query44	1351	857	840	840
query45	180	176	174	174
query46	886	1063	673	673
query47	1802	1913	1786	1786
query48	384	426	308	308
query49	714	525	419	419
query50	701	766	437	437
query51	4242	4277	4216	4216
query52	112	108	94	94
query53	238	277	203	203
query54	503	506	426	426
query55	85	86	80	80
query56	264	281	283	281
query57	1136	1163	1143	1143
query58	241	247	238	238
query59	2890	2921	2716	2716
query60	284	293	276	276
query61	122	118	119	118
query62	752	734	670	670
query63	225	193	203	193
query64	1726	1069	672	672
query65	3210	3175	3127	3127
query66	750	399	291	291
query67	15948	15521	15338	15338
query68	5357	787	526	526
query69	544	297	273	273
query70	1212	1150	1165	1150
query71	449	317	270	270
query72	6278	3618	3707	3618
query73	973	765	352	352
query74	9231	9252	9047	9047
query75	3403	3181	2683	2683
query76	3897	1164	741	741
query77	545	384	274	274
query78	10024	10131	9379	9379
query79	1795	830	678	678
query80	664	531	448	448
query81	496	282	241	241
query82	326	132	92	92
query83	178	170	162	162
query84	286	99	70	70
query85	733	353	293	293
query86	335	322	274	274
query87	4528	4622	4391	4391
query88	2904	2236	2259	2236
query89	404	322	293	293
query90	1909	197	197	197
query91	142	133	106	106
query92	78	59	56	56
query93	2432	1019	595	595
query94	642	401	307	307
query95	347	272	262	262
query96	506	527	283	283
query97	2809	2875	2715	2715
query98	229	215	202	202
query99	1315	1554	1295	1295
Total cold run time: 298962 ms
Total hot run time: 190797 ms

[doris-robot]

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56a5a982ebdc8007d4b4a20fa4ba47ad36fc2f88, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.25	0.06	0.06
query4	1.62	0.10	0.10
query5	0.42	0.41	0.40
query6	1.16	0.68	0.68
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.60	0.52	0.51
query10	0.57	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.69	2.70	2.69
query15	0.93	0.87	0.86
query16	0.38	0.38	0.37
query17	1.02	1.01	1.05
query18	0.21	0.20	0.20
query19	1.93	2.03	1.86
query20	0.01	0.01	0.01
query21	15.35	0.92	0.54
query22	0.76	1.24	0.81
query23	14.81	1.42	0.63
query24	11.04	1.52	0.51
query25	0.44	0.13	0.28
query26	0.88	0.17	0.15
query27	0.05	0.04	0.04
query28	6.19	0.79	0.44
query29	12.53	3.98	3.34
query30	0.25	0.09	0.06
query31	2.82	0.61	0.38
query32	3.23	0.56	0.47
query33	2.95	3.04	3.01
query34	15.73	5.18	4.49
query35	4.57	4.56	4.58
query36	0.67	0.49	0.50
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.62 s
Total hot run time: 30.62 s

[felixwluo]

run buildall

[doris-robot]

TPC-H: Total hot run time: 31193 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8fd364c50983636cec7964b1e7b2465fc9a0aa57, data reload: false

------ Round 1 ----------------------------------
q1	17593	5259	5066	5066
q2	2051	305	174	174
q3	10404	1200	761	761
q4	10218	1002	524	524
q5	7524	2476	2229	2229
q6	182	166	132	132
q7	899	755	594	594
q8	9299	1312	1138	1138
q9	4877	4547	4642	4547
q10	6818	2310	1887	1887
q11	473	271	253	253
q12	343	360	222	222
q13	17773	3637	3056	3056
q14	217	229	209	209
q15	513	456	448	448
q16	614	607	584	584
q17	557	848	327	327
q18	6573	6168	6122	6122
q19	1207	958	527	527
q20	339	323	179	179
q21	2754	2174	1905	1905
q22	381	336	309	309
Total cold run time: 101609 ms
Total hot run time: 31193 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5129	5066	5096	5066
q2	232	327	230	230
q3	2163	2641	2295	2295
q4	1419	1801	1348	1348
q5	4226	4128	4141	4128
q6	205	167	124	124
q7	1836	1800	1659	1659
q8	2508	2559	2520	2520
q9	7248	7178	7246	7178
q10	3031	3175	2762	2762
q11	569	506	495	495
q12	711	770	665	665
q13	3597	3735	3266	3266
q14	273	294	265	265
q15	511	464	465	464
q16	660	670	632	632
q17	1129	1570	1329	1329
q18	7594	7359	7343	7343
q19	770	792	868	792
q20	2007	2006	1925	1925
q21	5351	4963	4884	4884
q22	630	564	550	550
Total cold run time: 51799 ms
Total hot run time: 49920 ms

[doris-robot]

TPC-DS: Total hot run time: 190172 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8fd364c50983636cec7964b1e7b2465fc9a0aa57, data reload: false

query1	1299	943	949	943
query2	6232	1966	1965	1965
query3	11089	4531	4556	4531
query4	26082	23721	23252	23252
query5	4881	669	493	493
query6	301	219	196	196
query7	4000	506	311	311
query8	295	254	238	238
query9	8503	2512	2520	2512
query10	505	322	249	249
query11	15625	15091	14956	14956
query12	174	109	115	109
query13	1541	507	388	388
query14	9961	6030	6386	6030
query15	196	190	178	178
query16	7475	593	474	474
query17	1080	722	575	575
query18	2027	403	326	326
query19	187	214	157	157
query20	120	122	121	121
query21	203	121	105	105
query22	4301	4496	4175	4175
query23	34189	33426	33365	33365
query24	7590	2454	2478	2454
query25	506	471	412	412
query26	724	273	154	154
query27	2341	498	336	336
query28	4351	2453	2429	2429
query29	585	566	430	430
query30	213	186	160	160
query31	920	876	810	810
query32	69	61	61	61
query33	518	347	298	298
query34	769	866	558	558
query35	841	827	743	743
query36	985	999	924	924
query37	123	93	72	72
query38	4323	4371	4340	4340
query39	1567	1460	1504	1460
query40	203	118	101	101
query41	53	49	55	49
query42	121	111	104	104
query43	521	533	515	515
query44	1339	825	823	823
query45	182	170	162	162
query46	871	1058	669	669
query47	1840	1895	1781	1781
query48	398	474	326	326
query49	699	508	420	420
query50	702	735	423	423
query51	4281	4343	4242	4242
query52	111	106	95	95
query53	247	258	191	191
query54	505	494	419	419
query55	81	83	82	82
query56	265	291	248	248
query57	1150	1201	1109	1109
query58	242	240	244	240
query59	2876	3016	2747	2747
query60	304	290	290	290
query61	137	155	143	143
query62	784	758	667	667
query63	228	198	191	191
query64	3050	1016	663	663
query65	3375	3253	3217	3217
query66	821	391	303	303
query67	16172	15666	15557	15557
query68	2950	785	546	546
query69	472	298	270	270
query70	1228	1135	1121	1121
query71	398	289	251	251
query72	6146	3617	3674	3617
query73	717	741	356	356
query74	9310	8973	8941	8941
query75	3161	3193	2663	2663
query76	3059	1178	734	734
query77	469	358	273	273
query78	9919	10150	9220	9220
query79	1034	876	615	615
query80	1235	544	438	438
query81	534	271	245	245
query82	418	124	92	92
query83	270	177	154	154
query84	238	97	70	70
query85	765	417	391	391
query86	370	326	286	286
query87	4501	4505	4360	4360
query88	2973	2245	2230	2230
query89	404	322	285	285
query90	1576	188	184	184
query91	129	137	105	105
query92	58	57	60	57
query93	1124	978	575	575
query94	665	398	298	298
query95	346	258	253	253
query96	484	523	271	271
query97	2716	2852	2773	2773
query98	228	207	198	198
query99	1317	1425	1233	1233
Total cold run time: 267463 ms
Total hot run time: 190172 ms

[doris-robot]

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8fd364c50983636cec7964b1e7b2465fc9a0aa57, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.40	0.41	0.41
query6	1.22	0.64	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.61	0.52	0.53
query10	0.57	0.57	0.58
query11	0.15	0.10	0.11
query12	0.14	0.10	0.11
query13	0.66	0.60	0.61
query14	2.66	2.71	2.71
query15	0.92	0.86	0.86
query16	0.38	0.37	0.38
query17	1.02	1.06	1.01
query18	0.21	0.20	0.20
query19	1.88	1.83	2.03
query20	0.01	0.01	0.02
query21	15.35	0.89	0.55
query22	0.75	1.21	0.72
query23	14.89	1.37	0.65
query24	10.51	2.57	0.52
query25	0.31	0.20	0.11
query26	0.91	0.18	0.13
query27	0.06	0.04	0.04
query28	6.06	0.75	0.44
query29	12.56	3.97	3.34
query30	0.25	0.09	0.06
query31	2.82	0.57	0.38
query32	3.22	0.55	0.47
query33	3.00	3.06	3.04
query34	15.91	5.20	4.59
query35	4.58	4.52	4.58
query36	0.66	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.05	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.18 s
Total hot run time: 30.56 s

[felixwluo]

run p0

[felixwluo]

run feut