Skip to content

[Bug] Can't run dbt deps in an offline environment #11641

@alangner

Description

@alangner

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

We need to run some dbt commands on a machine without internet connection (but with access to a private git repository).
We have dependencies defined in packages.yml that either point to the git repository, or can be provided locally.
In this case we don't need to download dependencies from hub.dbt.com

If we run dbt deps it fails because it cannot contact hub.dbt.com.
This happens no matter if we only define "local" or only "git" ressources in packages.yml

Expected Behavior

If there are no dependencies defined from hub.dbt.com in the packages.yml file, then there should be no call to that URL.
In particular a timeout for this call should not block the dependencies to load.

Steps To Reproduce

  1. Define a e.g. local package in packages.yml
    packages:
  • local: path/to/package
  1. Disconnect from internet
  2. Run dbt deps
  3. After a long wait time the command will fail

Relevant log output

[info ] [MainThread]: Running with dbt=1.9.4
[debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'fail_fast': 'False', 'log_path': '/home/user/data-platform/logs', 'debug': 'False', 'profiles_dir': '/home/user/data-platform', 'use_colors': 'True', 'use_experimental_parser': 'False', 'empty': 'None', 'quiet': 'False', 'no_print': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'introspect': 'True', 'invocation_command': 'dbt deps', 'static_parser': 'True', 'target_path': 'None', 'log_format': 'default', 'send_anonymous_usage_stats': 'True'}
[debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '2125813c-e369-42a6-85c2-cd1527c0d5f5', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x78f1716e0860>]}
[debug] [MainThread]: Set downloads directory='/tmp/dbt-downloads-65ju5nbo'
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16f002cc0>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 0 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16f003590>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 1 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16f003c80>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 2 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16ee3c380>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 3 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[debug] [MainThread]: External call exception: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x78f16ee3ca70>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)'))
[debug] [MainThread]: Retrying external call. Attempt: 4 Max attempts: 5
[debug] [MainThread]: Making package index registry request: GET https://hub.getdbt.com/api/v1/index.json
[error] [MainThread]: Encountered an error:
 
[error] [MainThread]: Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/conda/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: timed out
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/urllib3/connection.py", line 207, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x78f16f002cc0>, 'Connection to hub.getdbt.com timed out. (connect timeout=30)')

Environment

- OS: Debian 12
- Python: 3.12.3
- dbt: 1.9.4

Which database adapter are you using with dbt?

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions