Error in py_call_impl(callable, call_args$unnamed, call_args$named) : OverflowError: Python int too large to convert to C long

[ducdonghiem]

Hi,

I was running the code for training the MNIST dataset and got an error. Below is my code and the error:

library(keras3)

# Load the MNIST dataset
mnist <- dataset_mnist()
train_images <- mnist$train$x
train_labels <- mnist$train$y
test_images <- mnist$test$x
test_labels <- mnist$test$y

# Inspect the data
str(train_images)
str(train_labels)
str(test_images)
str(test_labels)

# Build the model
model <- keras_model_sequential() %>%
  layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
  layer_dropout(rate = 0.4) %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.3) %>%
  layer_dense(units = 10, activation = 'softmax')

# Compile the model
compile(model,
        optimizer = "rmsprop",
        loss = "sparse_categorical_crossentropy",
        metrics = "accuracy")

# Reshape and normalize the data
train_images <- array(train_images, dim = c(60000, 28 * 28))
train_images <- train_images / 255
test_images <- array(test_images, dim = c(10000, 28 * 28))
test_images <- test_images / 255

# Train the model
fit(model, train_images, train_labels, epochs = 5, batch_size = 128)

I got:

Error in py_call_impl(callable, call_args$unnamed, call_args$named) : 
  OverflowError: Python int too large to convert to C long
Run `reticulate::py_last_error()` for details.

reticulate::py_last_error()

── Python Exception Message ────────────────────────────────────────────────────────────────────────────────
Traceback (most recent call last):
  File "C:\Users\DucDo\ONEDRI~1\DOCUME~1\VIRTUA~1\R-TENS~1\Lib\site-packages\keras\src\utils\traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\DucDo\ONEDRI~1\DOCUME~1\VIRTUA~1\R-TENS~1\Lib\site-packages\keras\src\backend\tensorflow\numpy.py", line 2079, in signbit
    tf.constant(0x80000000, dtype=tf.int32),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: Python int too large to convert to C long

── R Traceback ─────────────────────────────────────────────────────────────────────────────────────────────
    ▆
 1. ├─generics::fit(...)
 2. └─keras3:::fit.keras.src.models.model.Model(...)
 3.   ├─base::do.call(object$fit, args)
 4.   └─reticulate (local) `<python.builtin.method>`(...)
 5.     └─reticulate:::py_call_impl(callable, call_args$unnamed, call_args$named)
See `reticulate::py_last_error()$r_trace$full_call` for more details.

reticulate::py_last_error()$r_trace$full_call

[[1]]
fit(model, train_images, train_labels, epochs = 5, batch_size = 128)

[[2]]
fit.keras.src.models.model.Model(model, train_images, train_labels, 
    epochs = 5, batch_size = 128)

[[3]]
do.call(object$fit, args)

[[4]]
(function (x = NULL, y = NULL, batch_size = NULL, epochs = 1L, 
    verbose = "auto", callbacks = NULL, validation_split = 0, 
    validation_data = NULL, shuffle = TRUE, class_weight = NULL, 
    sample_weight = NULL, initial_epoch = 0L, steps_per_epoch = NULL, 
    validation_steps = NULL, validation_batch_size = NULL, validation_freq = 1L) 
{
    cl <- sys.call()
    cl[[1L]] <- list2
    call_args <- split_named_unnamed(eval(cl, parent.frame()))
    result <- py_call_impl(callable, call_args$unnamed, call_args$named)
    if (py_get_convert(callable)) 
        result <- py_to_r(result)
    if (is.null(result)) 
        invisible(result)
    else result
})(x = <environment>, y = <environment>, batch_size = 128L, epochs = 5L, 
    verbose = "auto", callbacks = list(<environment>))

[[5]]
py_call_impl(callable, call_args$unnamed, call_args$named)

How to fix this problem? Thanks.

[t-kalinowski]

Hi, thanks for reporting.

What version of TF and keras?

How were the python packages installed?

Does the error happen if you install the development version of keras3 and reticulate, and set Sys.setenv("RETICULATE_PYTHON" = "managed")?

[ducdonghiem]

I just reinstalled as instructed here
I run this script in Rstudio launched from Anaconda.

keras 2.15.0
keras3 1.3.0.9000
tensorflow 2.16.0

Installed by:

install.packages("keras3")
keras3::install_keras(backend = "tensorflow")

And got (the same with remotes::install_github("rstudio/keras3")):

Error: Error installing package(s): "scipy", "pandas", "Pillow", "pydot", "ipython", "tensorflow_datasets", "\"numpy<2\""
In addition: Warning message:
In file.info(x, extra_cols = FALSE) :
  cannot open file 'C:/Users/DucDo/AppData/Local/Microsoft/WindowsApps/python3.exe': The file cannot be accessed by the system

reticulate::py_config()

python:         C:/Users/DucDo/OneDrive - University of Manitoba/Documents/.virtualenvs/r-keras/Scripts/python.exe
libpython:      C:/Users/DucDo/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python39.dll
pythonhome:     C:/Users/DucDo/OneDrive - University of Manitoba/Documents/.virtualenvs/r-keras
version:        3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/DucDo/OneDrive - University of Manitoba/Documents/.virtualenvs/r-keras/Lib/site-packages/numpy
numpy_version:  1.26.4
keras:          C:\Users\DucDo\ONEDRI~1\DOCUME~1\VIRTUA~1\r-keras\lib\site-packages\keras

NOTE: Python version was forced by import("keras")

When I run fit(model, train_images, train_labels, epochs = 5, batch_size = 128), I got:

Error: RuntimeError: Function readRStudioPreference not found in RStudio
Run `reticulate::py_last_error()` for details.
Error during wrapup: 'length = 17' in coercion to 'logical(1)'
Error: no more error handlers available (recursive errors?); invoking 'abort' restart

Hope you can help!

[t-kalinowski]

Those error messages suggest that multiple python installations are loaded in the same R session, which is invalid.

Try this:

1. Close all open windows / files in RStudio
2. Restart the R session (Ctrl + shift + f10)
3. Run this R code:

remotes::install_github(c("rstudio/reticulate", "rstudio/keras3", "rstudio/tensorflow"), force = TRUE)

If there were errors, try to fix or ask for help. If there were no errors
4. Restart the R session (Ctrl + shift + f10)
5. Run this R code:

Sys.setenv("RETICULATE_PYTHON"="managed")
library(keras3)
op_convert_to_tensor("Hello World!")

If the above succeeds, then try to run the mnist script.

[ducdonghiem]

All the above succeeded until I ran the mnist script and got the same error:

Error: RuntimeError: Function readRStudioPreference not found in RStudio
Run `reticulate::py_last_error()` for details.
Error during wrapup: 'length = 17' in coercion to 'logical(1)'
Error: no more error handlers available (recursive errors?); invoking 'abort' restart

[t-kalinowski]

That's coming from the RStudio IDE. What version of RStudio? Is it the latest RStudio?

[ducdonghiem]

No it was not. I have retried with the latest RStudio and got this error at step 5:

Sys.setenv("RETICULATE_PYTHON"="managed")
library(keras3)
op_convert_to_tensor("Hello World!")

Downloading cpython-3.11.11-windows-x86_64-none (24.3MiB)
 Downloaded cpython-3.11.11-windows-x86_64-none
Installed 60 packages in 8.16s
Error: ImportError: Traceback (most recent call last):
  File "C:\Users\DucDo\AppData\Local\R\cache\R\RETICU~1\uv\cache\ARCHIV~1\L0OXVK~1\Lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 73, in <module>
<...truncated...>  module = hook()
             ^^^^^^
  File "C:\Users\DucDo\AppData\Local\R\win-library\4.4\reticulate\python\rpytools\loader.py", line 120, in _hook
    return _find_and_load(name, import_)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: A dynamic link library (DLL) initialization routine failed.


Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors for some common causes and solutions.
If you need help, create an issue at https://github.com/tensorflow/tensorflow/issues and include the entire stack trace above this error message.
Run `reticulate::py_last_error()` for details.