Unable to run code on Multiple GPUs in PyTorch - Usage shows only 1 GPU is being utilized

Question

I am training a Transformer Encoder-Decoder based model for Text summarization. The code works without any errors but uses only 1 GPU when checked with nvidia-smi. However, I want to run it on all the available GPUs (I can access as many as I want). I wrapped my model in Dataparallel. Here’s how I have wrapped the model:

if torch.cuda.device_count() > 1:
        print(f"Using {torch.cuda.device_count()} GPUs")
        model = nn.DataParallel(model)

    model.to(device)

This is how I am calling the functions from my model:

if torch.cuda.device_count() > 1:
                encoder_output = model.module.encode(
                    encoder_input, encoder_mask
                )  # (B, input_len, d_model)
                decoder_output = model.module.decode(
                    encoder_output, encoder_mask, decoder_input, decoder_mask
                )  # (B, seq_len, d_model)
                proj_output = model.module.project(
                    decoder_output
                )

I am using Python 3.12 and Torch 2.3.0. MWE is available on GitHub.

I have also checked if GPUs are configured correctly. Tried this example from PyTorch Documentation to test my GPUs configuration and working.

Maicon Melo Alves · Accepted Answer · 2024-06-19 11:40:54Z

0

You can find a complete code example to run on multiple GPUs at here. Besides adjusting the code to run on multiple GPUs, you need to execute the Python script with a program launcher like torchrun.

answered Jun 19, 2024 at 11:40

Maicon Melo Alves

1

Sign up to request clarification or add additional context in comments.

Comments

Abid Meraj · Accepted Answer · 2024-06-20 00:24:15Z

0

Someone has answered to this on another forum as follows, reproducing the same here:

In your provided code snippets you are using the deprecated nn.DataParallel module first, but are then skipping it by accessing the internal .module. Call the model directly and let DataParallel handle the data splits as well as model copies before it calls the forward.

Indeed, I was missing on forward method and there was no need to call .module.

answered Jun 20, 2024 at 0:24

Abid Meraj

114 bronze badges

Collectives™ on Stack Overflow

Unable to run code on Multiple GPUs in PyTorch - Usage shows only 1 GPU is being utilized

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related