Reusable "environment" packages to share dependencies and configuration

[gluxon]

Edit: Draft RFC in discussion here pnpm/rfcs#3

Background

Starting a brainstorming discussion for details of an upcoming "environments" feature. (Final name to be determined.) This was first proposed by @zkochan here: #2713 (comment)

This was previously discussed in:

• Ensure/check single version of a dependency package throughout workspace #2713
• RFC: Catalogs — Shareable dependency version specifiers rfcs#1

This discussion will be short-lived and close when there's enough concrete ideas for a new RFC. https://github.com/pnpm/rfcs

Concept

Environment packages will allow other packages to reference them for different purposes. A @comp/react-env environment package may be defined as:

{
  "name": "@comp/react-env",
  "version": "1.0.0",
  "author": "Example Team <[email protected]>",
  "dependencies": {
    "react": "16",
    "eslint": "4",
    "react-dom": "17",
    "jest": "22"
  },
  "peerDependencies": {
    "react-dom": "17",
    "react": "16"
  }
}

Other packages would be able to reference them through pnpm.environment:

{
  "name": "@comp/foo",
  "pnpm": {
     "environment": "@comp/[email protected]"
  }
}

The rough goals this feature should accomplish are:

1. Allow packages to reference a consistent version catalog when defining dependencies. This should provide better guarantees that an entire pnpm workspace uses the same version of a given dependency
   • RFC: Catalogs — Shareable dependency version specifiers rfcs#1
   • Ensure/check single version of a dependency package throughout workspace #2713
2. Provide a mechanism to inherit a subset of package.json fields such as engines, license, authors, scripts, pnpm.overrides between packages.
   • RFC: Catalogs — Shareable dependency version specifiers rfcs#1 (comment)
3. Provide a mechanism to inherit dependencies as if they were your own. Similar to the above, this involves inherting fields such as dependencies, devDependencies, etc.
   • RFC: Catalogs — Shareable dependency version specifiers rfcs#1 (comment)

[gluxon]

The version catalog portion of this feature feels different than the inheritance portions. I wonder if we should define them differently and reserve the "environments" terminology specifically as an inheritance mechanism.

Here's my take:

• Explicit inheritance: Using the pnpm.catalog config never automatically inherits. This makes it more clear that users need to add new entries to dependencies, devDependencies, peerDependencies, etc.
• Implicit inheritance: Using the pnpm.environment config, fields such as licenses, author, devDependencies would appear as if they had been defined in that same package.json file.

Since inheritance can be surprising, I think it makes sense to draw a line between the features that do this implicitly.

Configuring Version Catalog

Here's what the version catalog would look like:

{
  "name": "@comp/foo",
  "pnpm": {
     "catalog": "@comp/[email protected]"
  },
  "dependencies": {
    "react": "catalog:",
    "react-dom": "catalog:"
  },
  "devDependencies": {
    "eslint": "catalog",
    "jest": "catalog:"
  }
}

Configuring Environment Inheritance

Here's what environment inheritance would look like:

{
  "name": "@comp/foo",
  "pnpm": {
     "environment": "@comp/[email protected]"
  }
}

In-memory this would become the following. Note that author, dependencies, and peerDependencies have been inherited.

{
  "name": "@comp/foo",
  "pnpm": {
     "environment": "@comp/[email protected]"
  },
  "author": "Example Team <[email protected]>",
  "dependencies": {
    "react": "16",
    "eslint": "4",
    "react-dom": "17",
    "jest": "22"
  },
  "peerDependencies": {
    "react-dom": "17",
    "react": "16"
  }
}

Configuring both...?

I do see a use case for configuring both. Users may want to share a version catalog across the entire company, but scope common fields to the specific product team within a company.

{
  "name": "@team/env",
  "author": "Team Within @comp <[email protected]>",
  "license": "BSD-3-Clause"
}

{
  "name": "@comp/foo",
  "pnpm": {
     "environment": "@team/[email protected]",
     "version-catalog": "@comp/[email protected]"
  },
  "dependencies": {
    "react": "version-catalog:",
    "react-dom": "version-catalog:"
  }
}

But I don't think we should split the config specifically for this purpose. My concern was more around implicit/explicit inheritance.

Alternative

From pnpm/rfcs#1 (comment), the other alternative would be to configure this on the environment package itself. For example alwaysIncludedDevDeps would add jest and typescript as dev dependencies to any package defining @comp/utils as the environment.

On the environment package itself:

{
  "name": "@comp/utils",
  "version": "1.0.0",
  "alwaysIncludedDevDeps": ["jest", "typescript"],
  "dependencies": {
    "jest": "17",
    "typescript": "4",
    "ramda": "15"
}

[zkochan]

Ideally all fields would be inherited. The issue is that pnpm currently cannot automatically detect what dependencies are used in the project. In the case of Bit environments, Bit performs code analysis and automatically inherits only the needed dependencies and adds them to either prod or dev deps.

I think environment is a good name for the version catalog because it stores versions for peer dependencies. And peer dependencies define in what environment the package should be executed.

I do see a use case for configuring both. Users may want to share a version catalog across the entire company, but scope common fields to the specific product team within a company.

They could have one base environment inherited by other environments.

[gluxon]

Thanks. Let's stick with environment terminology for now then.

I think environment is a good name for the version catalog because it stores versions for peer dependencies. And peer dependencies define in what environment the package should be executed.

A bit confused about this though: "what environment the package should be executed"? Does the importer automatically have access to all the peer dependencies of an environment, regardless of whether they're explicitly defined in the importer's package.json?

More generally, how do the peer dependencies declared in an environment package affect an importer, if at all? I was imagining the environment package would have no effect.

[zkochan]

The peer dependencies from the environment will not be automatically applied because pnpm cannot detect if a dependency is used in the code. But we can disallow setting peer dependency versions manually if the same peer dependency is also present in the environment. We can also force a dependency to be included in peer dependencies if the user adds it to other types of dependencies.

[gluxon]

Okay that matches my understanding. Agree a package shouldn't be declaring peer dependencies that aren't actually used in its source.

[gluxon]

Question

When a package specifies a version from an environment package, should the original specifier be used, or its resolution?

For example, suppose the following environment package:

{
  "name": "@comp/react-env",
  "version": "1.0.0",
  "dependencies": {
    "react": "^16.0.0",
    "react-dom": "^16.0.0",
  }
}

Gets installed as (^16.0.0 → 16.14.0):

# pnpm-lock.yaml
  packages:

    /@comp/[email protected]:
      dependencies:
        react: 16.14.0
        react-dom: 16.14.0

And the following package references that:

{
  "name": "@comp/app",
  "pnpm": {
    "env": "@comp/[email protected]"
  },
  "dependencies": {
    "react": "pnpm-env:",
    "react-dom": "pnpm-env:"
  }
}

Should the pnpm-env: specifiers in-memory become ^16.0.0 or 16.14.0?

I also noticed that most of the examples given so far only specify the major version number. #2713 (comment). Was that intentional?

{
  "name": "@comp/react-env",
  "version": "1.0.0",
  "dependencies": {
    "react": "16",
    "eslint": "4",
    "react-dom": "17",
    "jest": "22"
  },
  "peerDependencies": {
    "react-dom": "17",
    "react": "16"
  }
}

My Thoughts

I could see reasons for approaches, but I think using the same resolution as @comp/react-env would be preferable. Using the original specifier (e.g. ^16.0.0) could lead to surprises as different importers get resolved to different versions that satisfy ^16.0.0.

[zkochan]

When a package specifies a version from an environment package, should the original specifier be used, or its resolution?

The original specifier should be used.

By the way, I also plan to change the resolution algorithm in pnpm v8, so that direct dependencies will always be resolved to the lowest version. So to update a version, the user will need to make a change in the environment.

I also noticed that most of the examples given so far only specify the major version number. #2713 (comment). Was that intentional?

Any specifier may be used.

[gluxon]

By the way, I also plan to change the resolution algorithm in pnpm v8, so that direct dependencies will always be resolved to the lowest version. So to update a version, the user will need to make a change in the environment.

I see how the pnpm v8 behavior would better guarantee the same resolution across all packages in the workspace referencing the environment then. That mitigates the concern of a specifier like ^16.0.0 resolving to different versions in different packages.

[gluxon]

Question

What should pnpm do if an environment package is referenced, but the consuming package uses pnpm-env: in a different dependencies block?

Suppose @comp/react-env defines a dependencies block:

{
  "name": "@comp/react-env",
  "version": "1.0.0",
  "dependencies": {
    "react": "16",
    "react-dom": "16",
  }
}

But the consuming package incorrectly uses this in peerDependencies?

{
  "name": "@comp/app",
  "pnpm": {
    "env": "@comp/[email protected]"
  },
  "peerDependencies": {
    "react": "pnpm-env:",
    "react-dom": "pnpm-env:"
  }
}

My Thoughts

We probably need to throw an error in this case and push environment package authors towards:

{
  "name": "@comp/react-env",
  "version": "1.0.0",
  "dependencies": {
    "react": "16",
    "react-dom": "16",
  },
  "peerDependencies": {
    "react": "16",
    "react-dom": "16",
  }
}

But I could see this getting messy as environment packages end up duplicating the same list across dependencies, devDependencies, and peerDependencies.

[zkochan]

In the environment you can only declare dependencies and peerDependencies. The specs from dependencies may be used in dependencies/devDependencies/optionalDependencies. The specs from peer dependencies may be used in peerDependencies only.

[gluxon]

That's helpful. Thanks!

[zkochan]

I did an error at the end

The specs for optional dependencies may be used in peerDependencies.

I mean to say the specs for peer dependencies may be used in peerDependencies only.

[gluxon]

The specs for optional dependencies may be used in peerDependencies.

Was "optional dependencies" here meant to be "peer dependencies" given the first sentence?

In the environment you can only declare dependencies and peerDependencies.

[zkochan]

yes

[gluxon]

Question

Should auto-install inheritance be specified on the environment package or the consuming package?

Approach 1:

{
  "name": "@comp/utils",
  "version": "1.0.0",
  "alwaysIncludedDevDeps": ["jest", "typescript"],
  "dependencies": {
    "jest": "17",
    "typescript": "4",
    "ramda": "15"
}

Approach 2:

{
  "name": "@comp/app",
  "pnpm": {
    "env": {
      "name": "@comp/[email protected]",
      "autoInstallAsDevDeps": true
    }
  }
}

I have the same question for regular metadata fields such as author, repository, etc.

My Thoughts

I see how approach 1 is easier, but it could be a surprise for developers that are only looking at the consuming app's package.json (e.g. @comp/app).

[zkochan]

I think the first approach is better. It is like "extends" in tools like eslint and typescript.

[gluxon]

I like the extends concept in ESLint configs and tsconfig.json files too. Bringing this feature to pnpm definitely modernizes package.json.

---

ESLint might be a good example to talk about more deeply.

• ESLint configs automatically inherit when they're added to the extends list.
• ESLint plugins do nothing by default when they're added to the plugins array. Users still need to list the specific rule from the plugin for it to apply.

I think this is similar to extends → pnpm.env and plugins → pnpm.catalog in an earlier question. #5974 (comment)

One of the reasons we need alwaysIncludedDevDeps is because we're overloading the new pnpm.env config for both inheritance and a version catalog. If we make pnpm.env works like extends in ESLint out of the box without an additional alwaysIncludedDevDeps setting, a lot of things might get simpler? But that requires a separate config for the catalog, like plugins in ESLint.

I know you weren't in favor of a separate pnpm.catalog config previously. #5974 (reply in thread). Was the problem there the catalog naming, or the idea of splitting the config options itself? If the problem was the name, perhaps we could do pnpm.env.extends and pnpm.env.references?

The reason I bring that up again is because alwaysIncludedDevDeps feels a bit strange and pnpm.env.extends / pnpm.env.references (à la ESLint extends / plugins) avoids it while making it more clear when an environment is being inherited/extended.

[zkochan]

ok, I am personally fine naming it "catalog"

[gluxon]

Thanks! I'm not married to the catalog naming. If you think of a more appropriate name we can definitely take that.

[gluxon]

Been a bit busy with work. Getting the RFC together will be my top priority this weekend.

[evelant]

Love where this is headed, these ideas will be a big step forward (yet another led by pnpm) for package management in the JS ecosystem!

[gluxon]

Hey folks! Thanks for following along. I think we can close this discussion since it served the initial purpose of answering many of my initial questions. See progress in pnpm/rfcs#3.