Problem using the same script in cloudml

[hrampadarath]

Hi, I am getting a strange issue whilist using the same script to train with cloudml. Last week I was training multiple models with the same script. Starting from a base model that gave pretty good results (acc = 0.9973; val_acc = 0.9925), I would add/remove elements or change parameters to find a preferred model architecture. All training runs were pretty much good, and nothing to be concerned.
Continuing my investigations today, the results are really bad. Restarted from the base model, but that also gave really bad results. Regardless of any changes to the model, the results are the same: after 1 epoch the accuracy flatlined at approx. 0.62. Training the same script on my linux machine, returns similar results to what I got last week. Changing the working directory do not help, but renaming the file seems to have worked (although the results were not as great, but acceptable).

Is this normal? Has this been noticed before? I think this is a cache problem, or is it the way I'm using cloudml? I really don't want to be using a different script when making small tweaks to the model.

Note: The data is saved to a local sub-directory named "data" in the working directory with the script, and not on Google storage bucket.

My keras/cloudml codes

library(keras)
library(caret)
library(e1071)


df <- readRDS("data/dataset.rds")

set.seed(111)
val_indices <- createDataPartition(df$y_train, p = 0.2, list = FALSE)

partial_y_train <- df$y_train[-val_indices]
partial_x_train <- cbind(df$x_train$x_trainX[-val_indices,],
                         df$x_train$x_trainY[-val_indices,],
                         df$x_train$x_trainZ[-val_indices,])

partial_y_valid <- df$y_train[val_indices]
partial_x_valid <- cbind(df$x_train$x_trainX[val_indices,],
                         df$x_train$x_trainY[val_indices,],
                         df$x_train$x_trainZ[val_indices,])



y_test <- df$y_test
x_test <- cbind(df$x_test$x_testX, df$x_test$x_testY, df$x_test$x_testZ)

inputXYZ <- layer_input(shape = c(1200))

outputXYZ <- inputXYZ %>% 
  layer_reshape(target_shape = c(400,3), input_shape = c(1200)) %>%
  layer_conv_1d(filters=64, kernel_size = 7, activation='relu', input_shape = c(400,3), batch_size = 1) %>%
  layer_conv_1d(filters=64, kernel_size = 7, activation='relu') %>%
  #layer_conv_1d(filters=64, kernel_size = 7, activation='relu') %>%
  #layer_max_pooling_1d(pool_size = 3) %>%
  layer_flatten() %>%
  layer_dense(units = 32, activation = 'sigmoid')%>% 
  layer_dense(units = 1, activation = "sigmoid")

model <- keras_model(inputs = inputXYZ, outputs = outputXYZ)

model %>% compile(optimizer = optimizer_rmsprop(lr = 0.01),
                  loss = "binary_crossentropy", 
                  metrics = c("accuracy"))


model %>% fit(
  x = partial_x_train,
  y = partial_y_train,
  epochs = 30,
  batch_size = 1500,
  validation_data = list(partial_x_valid, partial_y_valid),
  verbose=1, callbacks=NULL
)


NN_cm_fnol <- function(model, test_data, y_test) {
  
  pred <- model %>% predict(test_data)
  
  test_pred_class <- ifelse(pred >= 0.5, "fnol", "not.fnol")
  test_class <- ifelse(y_test == 1, "fnol", "not.fnol")
  
  actual_prediction <- table(actual = test_class, prediction = test_pred_class)
  print(caret::confusionMatrix(actual_prediction, mode = "everything"))
}

NN_cm_fnol(model, x_test, y_test)