-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Open
Labels
Description
Is this a new bug in dbt-core?
- I believe this is a new bug in dbt-core
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
We need to run some dbt commands on a machine without internet connection (but with access to a private git repository).
We have dependencies defined in packages.yml that either point to the git repository, or can be provided locally.
In this case we don't need to download dependencies from hub.dbt.com
If we run dbt deps
it fails because it cannot contact hub.dbt.com.
This happens no matter if we only define "local" or only "git" ressources in packages.yml
Expected Behavior
If there are no dependencies defined from hub.dbt.com in the packages.yml file, then there should be no call to that URL.
In particular a timeout for this call should not block the dependencies to load.
Steps To Reproduce
- Define a e.g. local package in packages.yml
packages:
- local: path/to/package
- Disconnect from internet
- Run dbt deps
- After a long wait time the command will fail
Relevant log output
[info ] [MainThread]: Running with dbt=1.9.4
[debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'fail_fast': 'False', 'log_path': '/home/user/data-platform/logs', 'debug': 'False', 'profiles_dir': '/home/user/data-platform', 'use_colors': 'True', 'use_experimental_parser': 'False', 'empty': 'None', 'quiet': 'False', 'no_print': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'introspect': 'True', 'invocation_command': 'dbt deps', 'static_parser': 'True', 'target_path': 'None', 'log_format': 'default', 'send_anonymous_usage_stats': 'True'}
[debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '2125813c-e369-42a6-85c2-cd1527c0d5f5', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x78f1716e0860>]}
[debug] [MainThread]: Set downloads directory='/tmp/dbt-downloads-65ju5nbo'
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16f002cc0>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 0 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16f003590>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 1 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16f003c80>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 2 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16ee3c380>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 3 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16ee3ca70>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 4 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[error] [MainThread]: Encountered an error:
[error] [MainThread]: Traceback (most recent call last):
File "/opt/conda/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/opt/conda/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
TimeoutError: timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/opt/conda/lib/python3.12/site-packages/urllib3/connection.py", line 616, in connect
self.sock = sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x78f16f002cc0>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)')
Environment
- OS: Debian 12
- Python: 3.12.3
- dbt: 1.9.4
Which database adapter are you using with dbt?
No response
Additional Context
No response