Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use c10d broadcast_object in Zero #79452

Open
kumpera opened this issue Jun 13, 2022 · 1 comment
Open

Use c10d broadcast_object in Zero #79452

kumpera opened this issue Jun 13, 2022 · 1 comment
Labels
better-engineering good first issue module: bootcamp oncall: distributed pt_distributed_rampup triaged

Comments

@kumpera
Copy link
Contributor

@kumpera kumpera commented Jun 13, 2022

🚀 The feature, motivation and pitch

The current implementation of Zero Redundancy optimizer has its own implementation of object broadcasting.

We should replace it with c10d broadcast_object_list.

To verify the result, run the tests in test/distributed/optim/test_zero_redundancy_optimizer.py.

Alternatives

No response

Additional context

No response

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501

@kumpera kumpera added oncall: distributed module: bootcamp good first issue triaged labels Jun 13, 2022
@rohan-varma rohan-varma added pt_distributed_rampup better-engineering labels Jun 13, 2022
@awgu
Copy link
Contributor

@awgu awgu commented Jun 13, 2022

Does broadcast_object_list() still load Tensors in the broadcasted objects to the sender's GPU, or has that been changed?

This was an issue previously, so I tried landing #61539 to add a map_location argument similar to torch.load() to specify which device to load the Tensors in the broadcasted objects. However, since I changed the underlying serialization to use torch.save() / torch.load() instead of pickle.Pickler / pickle.Unpickler, it broke internal workloads and was reverted, where the hypothesis for the breakage was the difference in default pickle protocol level between pickle.Pickle and torch.save().

The ZeRO _broadcast_object() uses torch.save() / torch.load() and uses the map_location argument to avoid loading received Tensors to the sender's GPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
better-engineering good first issue module: bootcamp oncall: distributed pt_distributed_rampup triaged
Projects
None yet
Development

No branches or pull requests

3 participants