Problem about a BE node reporting lots of HTTP_ERROR for task_type:CLONE error #26020

patrickdung2022 · 2023-10-27T07:46:37Z

patrickdung2022
Oct 27, 2023

We had installed Doris 2.0.2 3xFE and 3xBE on 3 VM

It is find when I install the single FE and BE
Then I add two more FE into the cluster, it is ok
When I add the two BE:
maybe I did not wait for enough time when adding the nodes in the SQL client, I type below commands quickly..

ALTER SYSTEM ADD BACKEND "10.58.179.55:9051";
ALTER SYSTEM ADD BACKEND "10.58.179.91:9051";

Then in the 10.58.179.91 node, the be warning log has appeared lots of HTTP 403 ERROR. like below:

From BE (node3, 10.58.179.91)
E1026 09:01:31.285338 2671988 variable.cpp:179] Already exposed `doris_cache_tablet_version_cache' whose value is `0'
E1026 09:01:31.285368 2671988 variable.cpp:179] Already exposed `doris_cache_tablet_version_cache_persecond' whose value is `0'
W1026 09:01:31.347836 2672572 olap_server.cpp:669] Have not get FE Master heartbeat yet
W1026 09:02:36.954288 2672659 http_client.cpp:173] fail to execute HTTP client, errmsg=The requested URL returned error: 403, trace=0. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000ba1f197 in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
1. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000ba1d72d in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::HttpClient::execute(std::function<bool (void const*, unsigned long)> const&) @ 0x000000000be98f1f in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:244: doris::HttpClient::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) @ 0x000000000be98d1f in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
4. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291: std::_Function_handler<doris::Status (doris::HttpClient*), doris::EngineCloneTask::_download_files(doris::DataDir*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::$_0>::_M_invoke(std::_Any_data const&, doris::HttpClient*&&) @ 0x000000000b79b107 in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
5. /root/src/doris-2.0/be/src/common/status.h:348: doris::HttpClient::execute_with_retry(int, int, std::function<doris::Status (doris::HttpClient*)> const&) @ 0x000000000be992fc in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
6. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:244: doris::EngineCloneTask::_download_files(doris::DataDir*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) @ 0x000000000b7978bc in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
7. /root/src/doris-2.0/be/src/common/status.h:348: doris::EngineCloneTask::_make_and_download_snapshots(doris::DataDir&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::TBackend*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::vector<doris::Version, std::allocator<doris::Version> > const&, bool*) @ 0x000000000b796a58 in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
8. /root/src/doris-2.0/be/src/olap/task/engine_clone_task.cpp:198: doris::EngineCloneTask::_do_clone() @ 0x000000000b79524e in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
9. /root/src/doris-2.0/be/src/olap/storage_engine.h:87: doris::EngineCloneTask::execute() @ 0x000000000b794b88 in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
10. /root/src/doris-2.0/be/src/common/status.h:432: doris::StorageEngine::execute_task(doris::EngineTask*) @ 0x000000000b64c739 in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
11. /root/src/doris-2.0/be/src/agent/task_worker_pool.cpp:0: doris::CloneTaskPool::_clone_worker_thread_callback() @ 0x000000000b696f1e in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
12. /root/src/doris-2.0/be/src/util/threadpool.cpp:0: doris::ThreadPool::dispatch_thread() @ 0x000000000ba5bdaf in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
13. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba51d3c in /data/doris/deploy/apache-doris-2.0.2-bin-x64/be/lib/doris_be
14. start_thread @ 0x00000000000081ca in /usr/lib64/libpthread-2.28.so
15. __clone @ 0x0000000000039e73 in /usr/lib64/libc-2.28.so
W1026 09:02:37.262897 2672659 status.h:383] meet error status: [HTTP_ERROR]The requested URL returned error: 403

From FE ():

2023-10-27 07:29:02,147 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14212, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:06,169 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14027, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:10,207 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14027, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:14,153 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14027, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:18,161 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14130, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:22,172 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14130, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:26,176 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14130, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:30,181 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14097, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:34,190 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14097, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:38,201 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14097, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:42,202 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13977, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:46,205 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13977, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:50,209 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13977, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:54,214 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14408, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:29:58,212 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14408, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:02,221 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14408, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:06,225 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14308, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:10,225 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14308, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:14,226 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14308, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:18,231 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14099, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:22,233 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14099, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:26,237 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14099, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:30,240 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14114, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:34,240 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14114, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:38,254 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14114, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:42,253 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13977, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:46,252 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13977, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:50,255 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13977, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:54,259 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14339, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:30:58,263 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14339, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:02,271 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14339, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:06,271 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14097, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:10,274 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14097, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:14,278 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14097, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:18,277 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13876, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:22,294 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13876, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:26,284 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13876, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:30,292 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14139, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:34,294 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14139, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:38,297 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14139, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:42,297 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14243, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:46,301 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14243, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:50,304 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14243, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:54,304 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14210, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:31:58,309 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14210, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:32:02,316 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:14210, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:32:06,320 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13960, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:32:10,328 WARN (thrift-server-pool-6|255) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13960, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:32:14,326 WARN (thrift-server-pool-8|257) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13960, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))
2023-10-27 07:32:18,328 WARN (thrift-server-pool-7|256) [MasterImpl.finishTask():93] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.58.179.91, be_port:9061, http_port:8041), task_type:CLONE, signature:13897, task_status:TStatus(status_code:HTTP_ERROR, error_msgs:[(10.58.179.91)[HTTP_ERROR]The requested URL returned error: 403]))

Here's the info of show FE and show BE:

mysql> show frontends;
+-----------------------------------------+--------------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
| Name                                    | Host         | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | Version                  | CurrentConnected |
+-----------------------------------------+--------------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
| fe_85f37b8b_e29e_43aa_8c47_173e4f6c91d5 | 10.58.179.54 | 9011        | 8031     | 9031      | 9021    | FOLLOWER | false    | 89135519  | true | true  | 78726             | 2023-10-27 07:45:13 | true     |        | doris-2.0.2-rc05-ae923f7 | Yes              |
| fe_fa302beb_cd40_41d2_92ab_64e32640bda8 | 10.58.179.55 | 9011        | 8031     | 9031      | 9021    | FOLLOWER | false    | 89135519  | true | true  | 78726             | 2023-10-27 07:45:13 | true     |        | doris-2.0.2-rc05-ae923f7 | No               |
| fe_6825db3b_9b79_4302_86c0_244145dd32a3 | 10.58.179.91 | 9011        | 8031     | 9031      | 9021    | FOLLOWER | true     | 89135519  | true | true  | 78727             | 2023-10-27 07:45:13 | true     |        | doris-2.0.2-rc05-ae923f7 | No               |
+-----------------------------------------+--------------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+--------------------------+------------------+
3 rows in set (0.05 sec)

mysql> show backends
    -> ;
+-----------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+--------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| BackendId | Host         | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | TabletNum | DataUsedCapacity | TrashUsedCapcacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | RemoteUsedCapacity | Tag                      | ErrMsg | Version                  | Status                                                                                                                        | HeartbeatFailureCounter | NodeRole |
+-----------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+--------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
| 12189     | 10.58.179.54 | 9051          | 9061   | 8041     | 8061     | 2023-10-27 07:20:21 | 2023-10-27 07:45:43 | true  | false                | 135       | 1.059 GB         | 0.000              | 56.933 GB     | 79.990 GB     | 28.83 % | 28.83 %        | 0.000              | {"location" : "default"} |        | doris-2.0.2-rc05-ae923f7 | {"lastSuccessReportTabletsTime":"2023-10-27 07:45:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 12260     | 10.58.179.55 | 9051          | 9061   | 8041     | 8061     | 2023-10-27 07:20:15 | 2023-10-27 07:45:43 | true  | false                | 106       | 1.202 GB         | 0.000              | 62.412 GB     | 79.983 GB     | 21.97 % | 21.97 %        | 0.000              | {"location" : "default"} |        | doris-2.0.2-rc05-ae923f7 | {"lastSuccessReportTabletsTime":"2023-10-27 07:45:35","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
| 12262     | 10.58.179.91 | 9051          | 9061   | 8041     | 8061     | 2023-10-27 07:19:51 | 2023-10-27 07:45:43 | true  | false                | 91        | 997.679 MB       | 0.000              | 62.440 GB     | 79.983 GB     | 21.93 % | 21.93 %        | 0.000              | {"location" : "default"} |        | doris-2.0.2-rc05-ae923f7 | {"lastSuccessReportTabletsTime":"2023-10-27 07:45:38","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
+-----------+--------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+--------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
3 rows in set (0.02 sec)

Thanks in advance.

ixzc · 2023-10-27T08:56:42Z

ixzc
Oct 27, 2023

Any questions now? I see that the result of show backends is normal.

3 replies

patrickdung2022 Oct 27, 2023
Author

We do not know why the HTTP ERROR 403 appeared so frequently and what is the impact and problem is.

komov-r Jul 9, 2025

@patrickdung2022 Were you able to fix this problem?

patrickdung2022 Jul 9, 2025
Author

@komov-r We upgraded to Doris 3.0.4 and is not aware of this problem now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Problem about a BE node reporting lots of HTTP_ERROR for task_type:CLONE error #26020

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Problem about a BE node reporting lots of HTTP_ERROR for task_type:CLONE error #26020

Uh oh!

Uh oh!

patrickdung2022 Oct 27, 2023

Replies: 1 comment · 3 replies

Uh oh!

ixzc Oct 27, 2023

Uh oh!

patrickdung2022 Oct 27, 2023 Author

Uh oh!

komov-r Jul 9, 2025

Uh oh!

patrickdung2022 Jul 9, 2025 Author

patrickdung2022
Oct 27, 2023

Replies: 1 comment 3 replies

ixzc
Oct 27, 2023

patrickdung2022 Oct 27, 2023
Author

patrickdung2022 Jul 9, 2025
Author