yarn-deduplicate

Cleans up yarn.lock by removing duplicates.

Builds:
![Node.js CI](https://github.com/scinos/yarn-deduplicate/actions/workflows/node.js.yml)

This package only works with Yarn v1. Yarn v2 supports package deduplication
natively!

A duplicate package is when two dependencies are resolved to a different version, even when a single
version matches the range specified in the dependencies. See the
Deduplication strategies section for a few examples.

Installation

Install the package globally:

``sh npm install -g yarn-deduplicate`

or

`sh yarn global add yarn-deduplicate`

This package also works wth npx, so you don't need to install it. For example, to recreate the most common scenario below withnpx, run:

`sh npx yarn-deduplicate yarn.lock`

---

`Usage`

The most common scenario is to run

`sh yarn-deduplicate yarn.lock`

This will use the default strategy to remove duplicated packages in yarn.lock.

If you do not specify the yarn.lock path, it defaults to yarn.lock.

Check all available options with:

`sh yarn-deduplicate --help`

---

`Duplicated packages`

yarn.lockcontains a list of all the dependencies required by your project (including transitive dependencies), and the actual package version installed to satisfy those dependencies.

For the context of this project, a "duplicated package" is a package that appears on multiple nodes of the dependency tree with overlapping version ranges but resolved to different versions.

For example, imagine that your project directly depends on lodash and babel, and babeldepends onlodash as well. Specifically, your project depends on lodash@^1.0.0 and babeldepends onlodash@^1.1.0. Because how the resolution algorithm works in Yarn, you might end up with two different copies oflodash (for example, version 1.0.1 and 1.2.0) in your project, even when1.2.0 will suffice to satisfy both requirements for lodash. That's a "duplicated package".

It is important to note that we do not consider duplicated packages when the version ranges don't overlap. For example, if your project depends onunderscore@^1.0.0 and underscore@^2.0.0. Your project will end up with two versions ofunderscore, and yarn-deduplicate won't change that.

When using yarn-deduplicateremember that it will change your dependency tree. There are certain code paths that now will run with a different set of dependencies. It is highly recommended that you review each change toyarn.lock. If the change is too big, use the flag --packagesto deduplicate them gradually.

`$3`

Yarn documentation seems to suggest this package shouldn't be necessary. For example, in https://classic.yarnpkg.com/en/docs/cli/dedupe/, it says

> The dedupe command isn’t necessary. yarn install will already dedupe.

This is, however, not exactly true. There are cases where yarn will _not_ deduplicate existing packages. For example, this scenario:

- Install libA. It depends on libB ^1.1.0. At this point, the latest version of libBis1.1.2, so it gets installed as a transitive dependency in your repo

- After a few days, install libC. It also depends on libB ^1.1.0. But this time, the latestlibB version is 1.1.3.

In the above scenario, you'll end up with libB@1.1.2 and libB@1.1.3 in your repo.

Find more examples in:

- yarn-deduplicate — The Hero We Need - De-duplicating yarn.lock - https://github.com/yarnpkg/yarn/issues/3778

`$3`

--strategy

highestwill try to use the highest installed version. For example, with the followingyarn.lock:

`text library@^1.1.0: version "1.2.0"

library@^1.2.0: version "1.2.0"

library@^1.3.0: version "1.3.0"`

It will deduplicate library@^1.1.0 and library@^1.2.0 to 1.3.0

fewerwill try to minimize the number of installed versions by trying to deduplicate to the version that satisfies most of the ranges first. For example, with the followingyarn.lock:

`text library@*: version "2.0.0"

library@>=1.1.0: version "3.0.0"

library@^1.2.0: version "1.2.0"`

It will deduplicate library@* and library@>=1.1.0 to 1.2.0.

Note that this may cause some packages to be downgraded. Be sure to check the changelogs between all versions and understand the consequences of that downgrade. If unsure, don't use this strategy.

It is not recommended to use different strategies for different packages. There is no guarantee that the strategy will be honored in subsequent runs ofyarn-deduplicateunless the same set of flags is specified again.

`$3`

--packages

Receives a list of packages to deduplicate. It will ignore any other duplicated package not in the list. This option is recommended when the number of duplicated packages inyarn.lockis too big to be easily reviewed by a human. This will allow for a more controlled and progressive deduplication ofyarn.lock.

--scopes

Receives a list of scopes to deduplicate. It will ignore any other duplicated package not in the list. This option is recommended when deduplicating a large number of inter-dependent packages from a single scope, such as @babel. This will allow for a more controlled and progressive deduplication ofyarn.lock without specifying each package individually.

`$3`

--exclude

--exclude-scopes

With these commands you can exclude certain packages/scopes from the deduplication process. This is specially useful if you want to apply a different strategy for a scope, for example.

`$3`

By default, yarn-deduplicatewill only match pre-release versions if they share they share the samemajor, minor and patch versions (example: ^1.2.3-alpha.1 and 1.2.3-alpha.2can be deduplicated, but^1.2.3 and 1.2.4-alpha.1can't). This matches the behaviour of semver.

To change this behaviour you can use the flag --includePrerelease. This will treat all pre-release versionas as if they were normal versions (^1.2.3 and 1.2.4-alpha.1 can be deduplicated).

`$3`

This tool can be used as part of a CI workflow. Adding the flag --failwill force the process to exit with status 1 if there are duplicated packages. Example:

`sh

`Print the list of duplicated packages and exit with status 1`


yarn-deduplicate --list --fail
Deduplicate yarn.lock and exit with status 1 if changes were required

yarn-deduplicate --fail


---
Migration guide
$3
In this version we have adopted variadic arguments from commander.js. These are the equivalent
commands:

`sh #Old yarn-deduplicate --packages libA,libB yarn-deduplicate --scopes @scopeA,@scopeB yarn-deduplicate --exclude libA,libB

#New yarn-deduplicate --packages libA libB yarn-deduplicate --scopes @scopeA @scopeB yarn-deduplicate --exclude libA libB`

A consequence of this change is that if you were using one or more of the affected options (--packages, --scopes or --exclude) and a custom path for yarn.lock, you need to use --to "stop" package/scope/exclude parsing:

`sh yarn-deduplicate --packages libA libB -- path/to/yarn.lock`

`$3`

In this version we have renamed the project and refactored the CLI. These are the equivalent commands:

#### Installation

`sh

`Old`


npm install -g yarn-tools
New

npm install -g yarn-deduplicate


#### List duplicates

`sh

`Old`


yarn-tools list-duplicates path/to/yarn.lock
New

yarn-deduplicate --list path/to/yarn.lock

$3

`sh

`Old`


yarn-tools fix-duplicates path/to/yarn.lock > tmp
mv tmp path/to/yarn.lock
New

yarn-deduplicate path/to/yarn.lock

$3

`sh

`Old`


yarn-tools fix-duplicates path/to/yarn.lock package1 package2

New

yarn-deduplicate --packages package1,package2 path/to/yarn.lock

License

yarn-deduplicate

Cleans up yarn.lock by removing duplicates.

Builds:
![Node.js CI](https://github.com/scinos/yarn-deduplicate/actions/workflows/node.js.yml)

This package only works with Yarn v1. Yarn v2 supports package deduplication
natively!

Installation

Install the package globally:

``sh npm install -g yarn-deduplicate`

or

`sh yarn global add yarn-deduplicate`

This package also works wth npx, so you don't need to install it. For example, to recreate the most common scenario below withnpx, run:

`sh npx yarn-deduplicate yarn.lock`

---

`Usage`

The most common scenario is to run

`sh yarn-deduplicate yarn.lock`

This will use the default strategy to remove duplicated packages in yarn.lock.

If you do not specify the yarn.lock path, it defaults to yarn.lock.

Check all available options with:

`sh yarn-deduplicate --help`

---

`Duplicated packages`

yarn.lockcontains a list of all the dependencies required by your project (including transitive dependencies), and the actual package version installed to satisfy those dependencies.

For the context of this project, a "duplicated package" is a package that appears on multiple nodes of the dependency tree with overlapping version ranges but resolved to different versions.

`$3`

Yarn documentation seems to suggest this package shouldn't be necessary. For example, in https://classic.yarnpkg.com/en/docs/cli/dedupe/, it says

> The dedupe command isn’t necessary. yarn install will already dedupe.

This is, however, not exactly true. There are cases where yarn will _not_ deduplicate existing packages. For example, this scenario:

- Install libA. It depends on libB ^1.1.0. At this point, the latest version of libBis1.1.2, so it gets installed as a transitive dependency in your repo

- After a few days, install libC. It also depends on libB ^1.1.0. But this time, the latestlibB version is 1.1.3.

In the above scenario, you'll end up with libB@1.1.2 and libB@1.1.3 in your repo.

Find more examples in:

- yarn-deduplicate — The Hero We Need - De-duplicating yarn.lock - https://github.com/yarnpkg/yarn/issues/3778

`$3`

--strategy

highestwill try to use the highest installed version. For example, with the followingyarn.lock:

`text library@^1.1.0: version "1.2.0"

library@^1.2.0: version "1.2.0"

library@^1.3.0: version "1.3.0"`

It will deduplicate library@^1.1.0 and library@^1.2.0 to 1.3.0

fewerwill try to minimize the number of installed versions by trying to deduplicate to the version that satisfies most of the ranges first. For example, with the followingyarn.lock:

`text library@*: version "2.0.0"

library@>=1.1.0: version "3.0.0"

library@^1.2.0: version "1.2.0"`

It will deduplicate library@* and library@>=1.1.0 to 1.2.0.

Note that this may cause some packages to be downgraded. Be sure to check the changelogs between all versions and understand the consequences of that downgrade. If unsure, don't use this strategy.

`$3`

--packages

--scopes

`$3`

--exclude

--exclude-scopes

With these commands you can exclude certain packages/scopes from the deduplication process. This is specially useful if you want to apply a different strategy for a scope, for example.

`$3`

To change this behaviour you can use the flag --includePrerelease. This will treat all pre-release versionas as if they were normal versions (^1.2.3 and 1.2.4-alpha.1 can be deduplicated).

`$3`

This tool can be used as part of a CI workflow. Adding the flag --failwill force the process to exit with status 1 if there are duplicated packages. Example:

`sh

`Print the list of duplicated packages and exit with status 1`


yarn-deduplicate --list --fail
Deduplicate yarn.lock and exit with status 1 if changes were required

yarn-deduplicate --fail


---
Migration guide
$3
In this version we have adopted variadic arguments from commander.js. These are the equivalent
commands:

`sh #Old yarn-deduplicate --packages libA,libB yarn-deduplicate --scopes @scopeA,@scopeB yarn-deduplicate --exclude libA,libB

#New yarn-deduplicate --packages libA libB yarn-deduplicate --scopes @scopeA @scopeB yarn-deduplicate --exclude libA libB`

`sh yarn-deduplicate --packages libA libB -- path/to/yarn.lock`

`$3`

In this version we have renamed the project and refactored the CLI. These are the equivalent commands:

#### Installation

`sh

`Old`


npm install -g yarn-tools
New

npm install -g yarn-deduplicate


#### List duplicates

`sh

`Old`


yarn-tools list-duplicates path/to/yarn.lock
New

yarn-deduplicate --list path/to/yarn.lock

$3

`sh

`Old`


yarn-tools fix-duplicates path/to/yarn.lock > tmp
mv tmp path/to/yarn.lock
New

yarn-deduplicate path/to/yarn.lock

$3

`sh

`Old`


yarn-tools fix-duplicates path/to/yarn.lock package1 package2

New

yarn-deduplicate --packages package1,package2 path/to/yarn.lock