PeftModel.from_pretrained adapter not applied when using device="cuda:1"

[Xxxxsir]

When I load a base model with AutoModelForCausalLM and then apply a LoRA adapter using PeftModel.from_pretrained, the adapter works correctly on GPU0, but on GPU1 the model behaves like the original base model (no adapter effect).

No error is raised — inference runs normally, but outputs match the base model instead of the adapted model.

Here is my code：

        print("Using PEFT model for inference.")
        tokenizer = AutoTokenizer.from_pretrained(adapter_model_path, token=hf_token, trust_remote_code=True, use_fast=True)
        model = AutoModelForCausalLM.from_pretrained(model_name, token=hf_token, dtype=dtype, device_map=device)
        model.resize_token_embeddings(len(tokenizer))

        adapter_path = adapter_model_path + "/adapter_model"
        print(f"Loading adapter model from {adapter_path}")
        model = PeftModel.from_pretrained(model, adapter_path)
        model.eval()

Environment

Thu Sep 18 09:02:23 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.247.01 Driver Version: 535.247.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100 80GB PCIe Off | 00000000:27:00.0 Off | 0 |
| N/A 62C P0 292W / 300W | 67975MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100 80GB PCIe Off | 00000000:38:00.0 Off | 0 |
| N/A 65C P0 291W / 300W | 18560MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+

Expected behavior
When specifying device="cuda:1", the adapter should be applied and inference results should match the adapted model (just like on GPU0).

[BenjaminBossan]

Thanks for the report. However, I cannot reproduce your findings. This is the code I used for testing:

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, PeftModel, get_peft_model

path = "/tmp/peft/2787"

torch.manual_seed(0)
device = 0
x = torch.arange(10).unsqueeze(0).to(device)
model_id = "trl-internal-testing/tiny-random-LlamaForCausalLM"
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
with torch.inference_mode():
    base_out = model(x).logits
    print("base model output:\n", base_out[0])

config = LoraConfig(init_lora_weights=False)
model = get_peft_model(model, config)
with torch.inference_mode():
    peft_out = model(x).logits
    print(f"output after applying LoRA, device {device} (should be != base output):\n", peft_out[0])

model.save_pretrained(path)
del model

device = 1
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
model = PeftModel.from_pretrained(model, path)
with torch.inference_mode():
    peft_out_loaded = model(x.to(device)).logits
    print(f"output after loading LorA, device {device} (should be == previous LoRA output):\n", peft_out_loaded[0])

It prints the following, which is what would be expected if everything works correctly:

base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
                                                                                                                                                                                                         
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0168, -0.0252,  0.0742,  ..., -0.0183, -0.0264,  0.0524],
        [ 0.0585, -0.0056,  0.0790,  ..., -0.0394,  0.0573,  0.1350],
        [ 0.1186,  0.0182,  0.0130,  ..., -0.0780,  0.0979,  0.0565],
        ...,
        [ 0.0740, -0.0514, -0.0078,  ..., -0.1429,  0.0050,  0.0454],
        [ 0.0755,  0.0077,  0.0021,  ..., -0.0312,  0.0025,  0.1476],
        [ 0.1275, -0.0228, -0.0605,  ..., -0.1178,  0.0023,  0.1093]],
       device='cuda:0')

output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[ 0.0168, -0.0252,  0.0742,  ..., -0.0183, -0.0264,  0.0524],
        [ 0.0585, -0.0056,  0.0790,  ..., -0.0394,  0.0573,  0.1350],
        [ 0.1186,  0.0182,  0.0130,  ..., -0.0780,  0.0979,  0.0565],
        ...,
        [ 0.0740, -0.0514, -0.0078,  ..., -0.1429,  0.0050,  0.0454],
        [ 0.0755,  0.0077,  0.0021,  ..., -0.0312,  0.0025,  0.1476],
        [ 0.1275, -0.0228, -0.0605,  ..., -0.1178,  0.0023,  0.1093]],
       device='cuda:1')

Could you please confirm that this reproducer also works on your system? If it does, please try to see how this code may differ from your code, or provide a complete reproducer for me to try.

[Xxxxsir]

Thanks for the report. However, I cannot reproduce your findings. This is the code I used for testing:

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, PeftModel, get_peft_model

path = "/tmp/peft/2787"

torch.manual_seed(0)
device = 0
x = torch.arange(10).unsqueeze(0).to(device)
model_id = "trl-internal-testing/tiny-random-LlamaForCausalLM"
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
with torch.inference_mode():
base_out = model(x).logits
print("base model output:\n", base_out[0])

config = LoraConfig(init_lora_weights=False)
model = get_peft_model(model, config)
with torch.inference_mode():
peft_out = model(x).logits
print(f"output after applying LoRA, device {device} (should be != base output):\n", peft_out[0])

model.save_pretrained(path)
del model

device = 1
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
model = PeftModel.from_pretrained(model, path)
with torch.inference_mode():
peft_out_loaded = model(x.to(device)).logits
print(f"output after loading LorA, device {device} (should be == previous LoRA output):\n", peft_out_loaded[0])
It prints the following, which is what would be expected if everything works correctly:

base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
                                                                                                                                                                                                         
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0168, -0.0252,  0.0742,  ..., -0.0183, -0.0264,  0.0524],
        [ 0.0585, -0.0056,  0.0790,  ..., -0.0394,  0.0573,  0.1350],
        [ 0.1186,  0.0182,  0.0130,  ..., -0.0780,  0.0979,  0.0565],
        ...,
        [ 0.0740, -0.0514, -0.0078,  ..., -0.1429,  0.0050,  0.0454],
        [ 0.0755,  0.0077,  0.0021,  ..., -0.0312,  0.0025,  0.1476],
        [ 0.1275, -0.0228, -0.0605,  ..., -0.1178,  0.0023,  0.1093]],
       device='cuda:0')

output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[ 0.0168, -0.0252,  0.0742,  ..., -0.0183, -0.0264,  0.0524],
        [ 0.0585, -0.0056,  0.0790,  ..., -0.0394,  0.0573,  0.1350],
        [ 0.1186,  0.0182,  0.0130,  ..., -0.0780,  0.0979,  0.0565],
        ...,
        [ 0.0740, -0.0514, -0.0078,  ..., -0.1429,  0.0050,  0.0454],
        [ 0.0755,  0.0077,  0.0021,  ..., -0.0312,  0.0025,  0.1476],
        [ 0.1275, -0.0228, -0.0605,  ..., -0.1178,  0.0023,  0.1093]],
       device='cuda:1')

Could you please confirm that this reproducer also works on your system? If it does, please try to see how this code may differ from your code, or provide a complete reproducer for me to try.

Thanks for the quick reply and the reproducer.
I ran your code on my machine, but I see different results on GPU1 compared to what you reported.

Here is the log from my run:

CUDA_VISIBLE_DEVICES=0,1 python test_peft_gpu.py
config.json: 100%|███████████████████████████████████████████████████████████████████████| 531/531 [00:00<00:00, 3.19MB/s]
model.safetensors: 100%|██████████████████████████████████████████████████████████████| 4.13M/4.13M [00:05<00:00, 767kB/s]
The following generation flags are not valid and may be ignored: ['pad_token_id']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
generation_config.json: 100%|█████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 855kB/s]
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        ...,
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497]],
       device='cuda:1')

[BenjaminBossan]

Oh wow, very strange. Can you add this snippet once before and once after loading and share the results?

print(model)
print({p.device for p in model.parameters()})
print(sum(p.sum() for p in model.parameters()))
print(model.get_model_status())
print(model.get_layer_status())

[BenjaminBossan]

Also, please share the versions of relevant packages: PyTorch, transformers, accelerate, PEFT. Are you using flash attention or other special kernels? Do you have some environment flags that could affect the backend (env | egrep 'CUDA|CUBLAS|CUDNN|TF32|PYTORCH')?

[Xxxxsir]

Oh wow, very strange. Can you add this snippet once before and once after loading and share the results?

print(model)
print({p.device for p in model.parameters()})
print(sum(p.sum() for p in model.parameters()))
print(model.get_model_status())
print(model.get_layer_status())

Here is the results

=== DEBUG: before loading adapter ===
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 16, padding_idx=31999)
    (layers): ModuleList(
      (0-1): 2 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=16, out_features=16, bias=False)
          (k_proj): Linear(in_features=16, out_features=16, bias=False)
          (v_proj): Linear(in_features=16, out_features=16, bias=False)
          (o_proj): Linear(in_features=16, out_features=16, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=16, out_features=64, bias=False)
          (up_proj): Linear(in_features=16, out_features=64, bias=False)
          (down_proj): Linear(in_features=64, out_features=16, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((16,), eps=1e-06)
        (post_attention_layernorm): LlamaRMSNorm((16,), eps=1e-06)
      )
    )
    (norm): LlamaRMSNorm((16,), eps=1e-06)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=16, out_features=32000, bias=False)
)
{device(type='cuda', index=0)}
tensor(57.7938, device='cuda:0', grad_fn=<AddBackward0>)
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')

=== DEBUG: after loading adapter ===
PeftModel(
  (base_model): LoraModel(
    (model): LlamaForCausalLM(
      (model): LlamaModel(
        (embed_tokens): Embedding(32000, 16, padding_idx=31999)
        (layers): ModuleList(
          (0-1): 2 x LlamaDecoderLayer(
            (self_attn): LlamaAttention(
              (q_proj): lora.Linear(
                (base_layer): Linear(in_features=16, out_features=16, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=16, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=16, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (k_proj): Linear(in_features=16, out_features=16, bias=False)
              (v_proj): lora.Linear(
                (base_layer): Linear(in_features=16, out_features=16, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=16, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=16, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (o_proj): Linear(in_features=16, out_features=16, bias=False)
            )
            (mlp): LlamaMLP(
              (gate_proj): Linear(in_features=16, out_features=64, bias=False)
              (up_proj): Linear(in_features=16, out_features=64, bias=False)
              (down_proj): Linear(in_features=64, out_features=16, bias=False)
              (act_fn): SiLU()
            )
            (input_layernorm): LlamaRMSNorm((16,), eps=1e-06)
            (post_attention_layernorm): LlamaRMSNorm((16,), eps=1e-06)
          )
        )
        (norm): LlamaRMSNorm((16,), eps=1e-06)
        (rotary_emb): LlamaRotaryEmbedding()
      )
      (lm_head): Linear(in_features=16, out_features=32000, bias=False)
    )
  )
)
{device(type='cuda', index=0)}
tensor(57.4667, device='cuda:0', grad_fn=<AddBackward0>)
Model status: TunerModelStatus(base_model_type='LlamaForCausalLM', adapter_model_type='LoraModel', peft_types={'default': 'LORA'}, trainable_params=1024, total_params=1033296, num_adapter_layers=4, enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': True}, available_adapters=['default'], devices={'default': ['cuda']})
Layer status: [TunerLayerStatus(name='model.model.layers.0.self_attn.q_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': True}, available_adapters=['default'], devices={'default': ['cuda']}), TunerLayerStatus(name='model.model.layers.0.self_attn.v_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': True}, available_adapters=['default'], devices={'default': ['cuda']}), TunerLayerStatus(name='model.model.layers.1.self_attn.q_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': True}, available_adapters=['default'], devices={'default': ['cuda']}), TunerLayerStatus(name='model.model.layers.1.self_attn.v_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': True}, available_adapters=['default'], devices={'default': ['cuda']})]
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')

=== DEBUG: after loading adapter ===
PeftModel(
  (base_model): LoraModel(
    (model): LlamaForCausalLM(
      (model): LlamaModel(
        (embed_tokens): Embedding(32000, 16, padding_idx=31999)
        (layers): ModuleList(
          (0-1): 2 x LlamaDecoderLayer(
            (self_attn): LlamaAttention(
              (q_proj): lora.Linear(
                (base_layer): Linear(in_features=16, out_features=16, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=16, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=16, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (k_proj): Linear(in_features=16, out_features=16, bias=False)
              (v_proj): lora.Linear(
                (base_layer): Linear(in_features=16, out_features=16, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=16, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=16, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (o_proj): Linear(in_features=16, out_features=16, bias=False)
            )
            (mlp): LlamaMLP(
              (gate_proj): Linear(in_features=16, out_features=64, bias=False)
              (up_proj): Linear(in_features=16, out_features=64, bias=False)
              (down_proj): Linear(in_features=64, out_features=16, bias=False)
              (act_fn): SiLU()
            )
            (input_layernorm): LlamaRMSNorm((16,), eps=1e-06)
            (post_attention_layernorm): LlamaRMSNorm((16,), eps=1e-06)
          )
        )
        (norm): LlamaRMSNorm((16,), eps=1e-06)
        (rotary_emb): LlamaRotaryEmbedding()
      )
      (lm_head): Linear(in_features=16, out_features=32000, bias=False)
    )
  )
)
{device(type='cuda', index=1)}
tensor(57.4667, device='cuda:1')
Model status: TunerModelStatus(base_model_type='LlamaForCausalLM', adapter_model_type='LoraModel', peft_types={'default': 'LORA'}, trainable_params=0, total_params=1033296, num_adapter_layers=4, enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': False}, available_adapters=['default'], devices={'default': ['cuda']})
Layer status: [TunerLayerStatus(name='model.model.layers.0.self_attn.q_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': False}, available_adapters=['default'], devices={'default': ['cuda']}), TunerLayerStatus(name='model.model.layers.0.self_attn.v_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': False}, available_adapters=['default'], devices={'default': ['cuda']}), TunerLayerStatus(name='model.model.layers.1.self_attn.q_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': False}, available_adapters=['default'], devices={'default': ['cuda']}), TunerLayerStatus(name='model.model.layers.1.self_attn.v_proj', module_type='lora.Linear', enabled=True, active_adapters=['default'], merged_adapters=[], requires_grad={'default': False}, available_adapters=['default'], devices={'default': ['cuda']})]
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:1')

and here is the versions of relevant packages:

torch: 2.4.1+cu121
transformers: 4.56.1
accelerate: 1.10.1
peft: 0.12.0

(judge) chenchen@chenchen:~/gjx/Judge$ env | egrep 'CUDA|CUBLAS|CUDNN|TF32|PYTORCH'
(judge) chenchen@chenchen:~/gjx/Judge$ env | egrep 'CUDA|CUBLAS|CUDNN|TF32|PYTORCH'
(judge) chenchen@chenchen:~/gjx/Judge$ 

[BenjaminBossan]

Thanks for the additional info. When you look at the model output, this time it works! After loading the LoRA adapter, the results are the same as before, which is what we would expect. I don't think that the additional print statements can be the cause of this change, but please double check if that's the case: If you comment them out, does the model output change?

Other than that, the results are identical, the only difference being that the first time, the PEFT model is in train mode (with requires_grad=True on LoRA adapters) and the second time, it's in eval mode. For consistency, you could add a model.eval() after get_peft_model to put it in eval mode and see if that changes the outcome.

Regarding your versions, there is nothing in particular that stands out. Your torch and PEFT versions are a bit old, so you could try upgrading them and see if there is any change.

[Xxxxsir]

Thanks for the additional info. When you look at the model output, this time it works! After loading the LoRA adapter, the results are the same as before, which is what we would expect. I don't think that the additional print statements can be the cause of this change, but please double check if that's the case: If you comment them out, does the model output change?

Other than that, the results are identical, the only difference being that the first time, the PEFT model is in train mode (with requires_grad=True on LoRA adapters) and the second time, it's in eval mode. For consistency, you could add a model.eval() after get_peft_model to put it in eval mode and see if that changes the outcome.

Regarding your versions, there is nothing in particular that stands out. Your torch and PEFT versions are a bit old, so you could try upgrading them and see if there is any change.

Strange，Here are the outputs from several consecutive runs I did，seems like random：

(judge) chenchen@chenchen:~/gjx/Judge$ CUDA_VISIBLE_DEVICES=0,1 python test_peft_gpu.py
The following generation flags are not valid and may be ignored: ['pad_token_id']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        ...,
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497]],
       device='cuda:1')
(judge) chenchen@chenchen:~/gjx/Judge$ CUDA_VISIBLE_DEVICES=0,1 python test_peft_gpu.py
The following generation flags are not valid and may be ignored: ['pad_token_id']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:1')
(judge) chenchen@chenchen:~/gjx/Judge$ CUDA_VISIBLE_DEVICES=0,1 python test_peft_gpu.py
The following generation flags are not valid and may be ignored: ['pad_token_id']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        ...,
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497]],
       device='cuda:1')
(judge) chenchen@chenchen:~/gjx/Judge$ CUDA_VISIBLE_DEVICES=0,1 python test_peft_gpu.py
The following generation flags are not valid and may be ignored: ['pad_token_id']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        ...,
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497],
        [-0.0920, -0.0163,  0.0668,  ..., -0.0331, -0.1508, -0.0497]],
       device='cuda:1')
(judge) chenchen@chenchen:~/gjx/Judge$ CUDA_VISIBLE_DEVICES=0,1 python test_peft_gpu.py
The following generation flags are not valid and may be ignored: ['pad_token_id']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
base model output:
 tensor([[-0.0682, -0.0175, -0.0077,  ...,  0.0170, -0.0623, -0.0240],
        [-0.0542, -0.0649, -0.0617,  ..., -0.1379,  0.0427,  0.0846],
        [ 0.0315,  0.0083, -0.0798,  ..., -0.0822,  0.0893, -0.0965],
        ...,
        [ 0.0190, -0.0499, -0.0092,  ..., -0.0988, -0.0121, -0.0791],
        [ 0.0428,  0.0370,  0.0197,  ...,  0.0427, -0.0087,  0.1056],
        [ 0.1015, -0.0016, -0.0331,  ..., -0.0334,  0.0029,  0.0050]],
       device='cuda:0')
output after applying LoRA, device 0 (should be != base output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:0')
output after loading LorA, device 1 (should be == previous LoRA output):
 tensor([[ 0.0299, -0.0452,  0.0530,  ..., -0.0207, -0.0032,  0.0378],
        [ 0.0449, -0.0803,  0.0368,  ..., -0.1109,  0.0422,  0.0946],
        [ 0.0770, -0.0409,  0.0045,  ..., -0.1055,  0.0776,  0.0034],
        ...,
        [-0.0024, -0.0782, -0.0391,  ..., -0.1278, -0.0582, -0.0062],
        [ 0.0389, -0.0075, -0.0176,  ..., -0.0227, -0.0622,  0.1843],
        [ 0.0940, -0.0295, -0.1351,  ..., -0.1036, -0.0684,  0.1012]],
       device='cuda:1')

[BenjaminBossan]

This is a new one for me. On my machine, I always get the same results. I can't imagine what might cause this random behavior. I assume you run the exact same script back to back, right? Are you sharing your machine with other users?

One thing I noticed with the wrong result is that it just repeats the same output for each row, which is also very strange and does not correspond to the output of the base model. There must be something else going on entirely.

As I have no idea what the issue is, it's hard to provide further help. One thing I'd recommend is to create a completely new Python environment and ensure that the latest versions of Python packages are being used. If that doesn't help, consider upgrading the CUDA driver if it's out of date.

[Xxxxsir]

This is a new one for me. On my machine, I always get the same results. I can't imagine what might cause this random behavior. I assume you run the exact same script back to back, right? Are you sharing your machine with other users?

One thing I noticed with the wrong result is that it just repeats the same output for each row, which is also very strange and does not correspond to the output of the base model. There must be something else going on entirely.

As I have no idea what the issue is, it's hard to provide further help. One thing I'd recommend is to create a completely new Python environment and ensure that the latest versions of Python packages are being used. If that doesn't help, consider upgrading the CUDA driver if it's out of date.

In fact, I do share the machine with other users, but I also cannot figure out why this would cause such a result.

[BenjaminBossan]

In fact, I do share the machine with other users, but I also cannot figure out why this would cause such a result.

I'm just guessing, but perhaps there is some kind of hidden optimization going on depending on available memory or so, and a bug in this code path affects the output.