Skip to content

Clarify PP split point documentation. #1054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
githubsgi opened this issue Apr 3, 2025 · 1 comment
Open

Clarify PP split point documentation. #1054

githubsgi opened this issue Apr 3, 2025 · 1 comment
Labels
question Further information is requested

Comments

@githubsgi
Copy link
Contributor

Bug description

The current documentation is as follows.

        self.parser.add_argument(
            "--parallelism.pipeline_parallel_split_points",
            type=string_list,
            nargs="+",
            default=[],
            help="""
                Specify comma-separated names of modules to use as the beginning of a split point.

                e.g. "layers.0,layers.2" will cause the model to be split into 3 stages,
                the first containing all the layers up to layers.0,
                the second containing layers.0 and up to layers.2,
                the third containing layers.2 and all the remaining layers.

                Note: fully-automated splitting may be enabled in the future,
                but currently the split points must be specified manually.""",
        )

The above description seems to indicate that layer.0 is present in both the first and second stages, layer.2 is present in both second and third stages. Can someone please clarify inclusivity ?

Versions

head of master

@tianyu-l
Copy link
Contributor

tianyu-l commented Apr 3, 2025

cc @H-Huang

@tianyu-l tianyu-l added the question Further information is requested label Apr 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants