Why doesn't the bing squad bert example or cifar example work with the zero optimizer?

[SantoshGuptaML]

I can get the bing squad bert example to run as is, but when I want to experiment with the zero optimizer and add

  "zero_optimization": {
    "stage": 1
  }

to https://github.com/microsoft/DeepSpeedExamples/blob/master/BingBertSquad/deepspeed_bsz24_config.json

The program seems to freeze during or after loading the weights. These are the last couple lines before it freezes.

04/22/2021 05:38:24 - INFO - convert_bert_ckpt_to_deepspeed -   Loading Huggingface weight bert.encoder.layer.21.attention.self.query.weight with shape (1024, 1024)
04/22/2021 05:38:24 - INFO - convert_bert_ckpt_to_deepspeed -   Loading Huggingface weight bert.encoder.layer.21.attention.self.query.bias with shape (1024,)
04/22/2021 05:38:24 - INFO - convert_bert_ckpt_to_deepspeed -   Loading Huggingface weight bert.encoder.layer.21.attention.self.key.weight with shape (1024, 1024)
04/22/2021 05:38:24 - INFO - convert_bert_ckpt_to_deepspeed -   Loading Huggingface weight bert.encoder.layer.21.attention.self.key.bias with shape (1024,)
04/22/2021 05:38:24 - INFO - convert_bert_ckpt_to_deepspeed -   Loading Huggingface weight bert.encoder.layer.21.attention.self.value.weight with shape (1024, 1024)

For the cifar example, first I tried to add the zero optimizer stage 1 to the config, but then I get an error message saying that I need fp16 enabled, so after I add

  "fp16": {
    "enabled": true,
    "loss_scale": 0,
    "initial_scale_power": 32,
    "loss_scale_window": 1000,
    "hysteresis": 2,
    "min_loss_scale": 1
  }

I then get an error message saying

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same      

So it seems that there's something in the code for these examples that are preventing either the zero optimizer or fp16 from working, I'm very curious about what exactly it is, so I can avoid the issue with my models.

[SantoshGuptaML]

It was mostly due to this issue, after I updated it, I've had no problem

#968

[tjruwase]

@SantoshGuptaML, so can this issue be closed?

[SantoshGuptaML]

Yes. I am looking for an option to close this, but I don't see it. It may be different since this is a discussion.