Installation Issue: keras3::install_keras() creates a virtual environment outside of Miniconda

[polizzi4757]

The problem is that

keras3::install_keras(backend = "tensorflow")

creates a virtual environment "r-keras" stored in "D:/Data Folders/Documents/.virtualenvs/r-keras" outside of Miniconda

tensorflow::install_tensorflow()

creates a virtual environment "r-tensorflow" stored in "D:/Data Folders/Documents/.virtualenvs/r-tensorflow" outside of Miniconda

I do not know how to fix that. The immediate consequence is that both installations of keras and tensorFlow are not avalaible.

Here are my installation's steps:

1. I couldn't install the development version of packages reticulate and keras3 because their compilation ended with errors. I have R version 4.1.1 and rtools40 installed.

2. I solved the missing rpytools problem with the environmental variable

   PYTHONPATH = C:/Program Files/R/R-4.1.1/library/reticulate/python
   

3. I removed the conflicting installation at each of the countless new installation tries.

4. I installed the R package reticulate and Miniconda

   install.packages("reticulate")
   reticulate::install_miniconda()

5. I installed the R package keras3 and its dependence tensorflow

   install.packages("keras3") # installs also the R package tensorflow
   keras3::install_keras(backend = "tensorflow")
   tensorflow::install_tensorflow()

6. I tested the keras3 installation

   library(keras3); library(tensorflow)
   mnist <- dataset_mnist()

   Error: ModuleNotFoundError: No module named 'keras'
   

7. I tested the tensorflow installation

   tensorflow::as_tensor("Hello World")

   Error: Valid installation of TensorFlow not found.
   

8. reticulate::py_config()

   python:         C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
   libpython:      C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python310.dll
   pythonhome:     C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate
   version:        3.10.16 | packaged by conda-forge | (main, Dec  5 2024, 14:07:43) [MSC v.1942 64 bit (AMD64)]
   Architecture:   64bit
   numpy:          C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
   numpy_version:  2.2.4
   keras:          [NOT FOUND]
   
   NOTE: Python version was forced by RETICULATE_PYTHON
   

9.   tensorflow::tf_config()

   Valid installation of TensorFlow not found.
   
   Python environments searched for 'tensorflow' package:
     C:\Users\Giuseppe\AppData\Local\r-miniconda\envs\r-reticulate\python.exe
   
   Python exception encountered:
     Traceback (most recent call last):
     File "C:\Program Files\R\R-4.1.1\library\reticulate\python\rpytools\loader.py", line 122, in _find_and_load_hook
       return _run_hook(name, _hook)
     File "C:\Program Files\R\R-4.1.1\library\reticulate\python\rpytools\loader.py", line 96, in _run_hook
       module = hook()
     File "C:\Program Files\R\R-4.1.1\library\reticulate\python\rpytools\loader.py", line 120, in _hook
       return _find_and_load(name, import_)
   ModuleNotFoundError: No module named 'tensorflow'
   

10. reticulate::py_last_error()

    -- Python Exception Message --------------------------------------------------------------------------------------------
    Traceback (most recent call last):
      File "C:\Program Files\R\R-4.1.1\library\reticulate\python\rpytools\loader.py", line 122, in _find_and_load_hook
        return _run_hook(name, _hook)
      File "C:\Program Files\R\R-4.1.1\library\reticulate\python\rpytools\loader.py", line 96, in _run_hook
        module = hook()
      File "C:\Program Files\R\R-4.1.1\library\reticulate\python\rpytools\loader.py", line 120, in _hook
        return _find_and_load(name, import_)
    ModuleNotFoundError: No module named 'tensorflow'
    
    -- R Traceback ---------------------------------------------------------------------------------------------------------
    x
    1. \-reticulate::import("tensorflow")
    2.   \-reticulate:::py_module_import(module, convert = convert)
    See `reticulate::py_last_error()$r_trace$full_call` for more details.
    

11.   sessionInfo()

    R version 4.1.1 (2021-08-10)
    Platform: x86_64-w64-mingw32/x64 (64-bit)
    Running under: Windows 10 x64 (build 19045)
    
    Matrix products: default
    
    locale:
      [1] LC_COLLATE=Italian_Italy.1252  LC_CTYPE=Italian_Italy.1252    LC_MONETARY=Italian_Italy.1252
    [4] LC_NUMERIC=C                   LC_TIME=Italian_Italy.1252
    
    attached base packages:
      [1] stats     graphics  grDevices utils     datasets  methods   base
    
    other attached packages:
      [1] tensorflow_2.16.0 keras3_1.3.0
    
    loaded via a namespace (and not attached):
      [1] Rcpp_1.0.14         rstudioapi_0.17.1   whisker_0.4.1       magrittr_2.0.3      lattice_0.20-44
    [6] rlang_1.1.5         dotty_0.1.0         fansi_0.5.0         tools_4.1.1         grid_4.1.1
    [11] png_0.1-8           utf8_1.2.2          cli_3.6.4           withr_3.0.2         tfruns_1.5.3
    [16] lifecycle_1.0.4     Matrix_1.5-4        vctrs_0.6.5         base64enc_0.1-3     codetools_0.2-18
    [21] zeallot_0.1.0       glue_1.8.0          compiler_4.1.1      pillar_1.9.0        generics_0.1.3
    [26] reticulate_1.41.0.1 jsonlite_1.9.1
    

```

I really hope you could help me resolve the mentioned problem because I already invested a lot of time in this project with no results. So please help. Thank you.

[t-kalinowski]

It looks like you have a few python installations, and reticulate is not finding the one you expect. Reticulate's order of python discovery can be found here: https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery

The most surefire way to point reticulate at a python installation is with RETICULATE_PYTHON.

# in a new R session before reticulate has initialized python
if (reticulate::py_available()) 
  stop("reticulate has already selected a python installation. Restart the R session and try again")
Sys.setenv("RETICULATE_PYTHON" = "path/to/python")
reticulate::py_config()

However, we have not fully updated the docs yet for keras, but the following method will work and does not require manually creating a python environment:

Sys.setenv("RETICULATE_PYTHON" = "managed")
library(keras3)
reticulate::py_config()

[polizzi4757]

Thank you very much for your prompt answer. I waited to respond because setting to "managed" the environment variable RETICULATE_PYTHON produced other types of incomprehensible things (for me) and I wanted to make sense of them before to respond to you. I still need your help.

I made some "tries" to see what happens. I always restarted R before a new try.

1st try

    reticulate::py_config()

      Downloading cpython-3.11.11-windows-x86_64-none
       Downloaded cpython-3.11.11-windows-x86_64-none
      Downloading numpy
       Downloaded numpy
      Installed 1 package in 2.38s

At this point the system interrupted processing and gave the following Alert:

     R Session Disconnected
       This browser was disconnected from the R session because another browser connected
       (only one browser at a time may be connected to an RStudio session)

2nd try

   library(keras3)

     Warning message:
       The request to `use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe")`
       will be ignored because the environment variable RETICULATE_PYTHON is set to "managed"

  mnist <- dataset_mnist()

     Downloading setuptool
     Downloading jedi
     Downloading pygments
     Downloading setuptools
     Downloading scipy
     Downloading pandas)
     Downloading tensorboard
     Downloading pillow
     Downloading grpcio
     Downloading tensorflow-io-gcs-filesystem
     Downloading libclang
     Downloading numpy
     Downloading h5py
     Downloading keras
     Downloading tensorflow
      Downloaded tensorflow-io-gcs-filesystern
      Downloaded pygments
      Downloaded pillow
      Downloaded setuptools
      Downloaded h5py
      Downloaded keras
      Downloaded grpcio
      Downloaded tensorboard
      Downloaded jedi
      Downloaded pandas
      Downloaded numpy
      Downloaded libclang
      Downloaded scipy
      Downloaded tensorflow
     Installed 60 packages in 33.37s

At this point the system interrupted processing and gave the following Alert:

      R Session Disconnected
        This browser was disconnected from the R session because another browser connected
        (only one browser at a time may be connected to an RStudio session)

3rd try

   library(keras3)

     Warning message:
       The request to `use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe")`
       will be ignored because the environment variable RETICULATE_PYTHON is set to "managed"

   reticulate::py_config()

      R Session Disconnected
        This browser was disconnected from the R session because another browser connected
        (only one browser at a time may be connected to an RStudio session)

4th try

   library(tensorflow)

 Warning message:
   The request to `use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe")`
   will be ignored because the environment variable RETICULATE_PYTHON is set to "managed"

```R
  tensorflow::tf_config()

      R Session Disconnected
        This browser was disconnected from the R session because another browser connected
        (only one browser at a time may be connected to an RStudio session)

5th try

   library(tensorflow)

     Warning message:
       The request to `use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe")`
       will be ignored because the environment variable RETICULATE_PYTHON is set to "managed"

   reticulate::import("tensorflow")

  Python path configuration:
    PYTHONHOME = 'C:/Users/Giuseppe/AppOata/Local/R/cache/R/reticulate/uv/cache/archive-vO/KZy9DSKRuq-uwCn2Sq-UM'
    PYTHONPATH = (not set)
    program name = 'C:/Users/Giuseppe/AppOata/Local/R/cache/R/reticulate/uv/cache/archive-vO/KZy9DSKRuq-uwCn2Sq-UM/s........
    isolated = 0
    enviroment = 1
    usersite = 1
    safe_path = 0
    import site = 1
    is in build tree = 0
    stdlib dir = 'C:\Users\Giuseppe\AppOata\Local\R\cache\R\reticulate\uv\cache\archive-vO\KZy9DSKRuq-uwCn2Sq-UM\Lib........
    sys.base_executable = 'C:\\Program Files\\RStudio\\resources\\app\\bin\\rsession.exe'
    sys.base_prefix = 'C:/Users/Giuseppe/AppOata/Local/R/cache/R/reticulate/uv/cache/archive-vO/KZy9DSKRuq-uwCn2Sq-U........
    sys.base_exec_prefix = 'C:/Users/Giuseppe/AppOata/Local/R/cache/R/reticulate/uv/cache/archive-vO/KZy9DSKRuq-uwCn........
    sys.platlibdir = 'DLLs'
    sys.executable = 'C:\\Program Files\\RStudio\\resources\\app\\bin\\rsession.exe
    sys.prefix = 'C:/Users/Giuseppe/AppData/Local/R/cache/R/reticulate/uv/cache/archive-v0/KZy90SKRuq-uwCn2Sq-UM
    sys.exec_prefix = 'C:/Users/Giuseppe/AppData/Local/R/cache/R/reticulate/uv/cache/archive-v0/KZy90SKRuq-uwCn2Sq-U........
    sys.path = [
      'C:\\Users\\Giuseppe\\AppData\\Roaming\\UV\\python\\cpython-3.11.11-windows-x86_64-none\\python311.zip',
      'C:\\Users\\Giuseppe\\AppData\\Local\\R\\cache\\R\\reticulate\\uv\\cache\\archive-vO\\KZy9DSKRuq-uwCn2Sq-UM\D.........
      'C:\\Users\\Giuseppe\\AppData\\Local\\R\\cache\\R\\reticulate\\uv\\cache\\archive-vO\\KZy9DSKRuq-uwCn2Sq-UM\U.........
      'C:\\Program Files\\RStudio\\resources\\app\\bin',
    ]
  Fatal Python error: init_fs_encoding: failed to get the PYhon codec of the filesystem encoding
  Python runtime state: core initialized
  ModuleNotFoundError: No module named 'encodings'
  
  current thread 0x0000zfcc (most recent call first) :
    <no Python frame>

    R Session Disconnected
      This browser was disconnected from the R session because another browser connected
      (only one browser at a time may be connected to an RStudio session)

I noticed that is reticulate::install_miniconda() that creates the directory
"C:\Users\Giuseppe\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none"
where are python.exe, python3.dll, python311.dll, pythonw.exe and the DLLs, include, Lib, libs, etc. subdirectories.

sys.path I get with reticulate

'C:\\Users\\Giuseppe\\AppData\\Local\\r-miniconda\\envs\\r-reticulate'
  'C:\\Users\\Giuseppe\\AppData\\Local\\r-miniconda\\envs\\r-reticulate\\python310.zip'
  'C:\\Users\\Giuseppe\\AppData\\Local\\r-miniconda\\envs\\r-reticulate\\DLLs'
  'C:\\Users\\Giuseppe\\AppData\\Local\\r-miniconda\\envs\\r-reticulate\\lib'
  'C:\\Program Files\\RStudio\\resources\\app\\bin'
  'C:\\Users\\Giuseppe\\AppData\\Local\\r-miniconda\\envs\\r-reticulate'
  'C:\\Users\\Giuseppe\\AppData\\Local\\r-miniconda\\envs\\r-reticulate\\lib\\site-packages']

Thank you for your help.

[t-kalinowski]

R Session Disconnected
This browser was disconnected from the R session because another browser connected
(only one browser at a time may be connected to an RStudio session) 

I believe message only appears if you are using RStudio server, and only if you are connecting to an instance on Linux.

Can you confirm if that's true, or are you using the desktop version of RStudio?

 Warning message:
   The request to `use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe")`
   will be ignored because the environment variable RETICULATE_PYTHON is set to "managed"

Do you know where use_python is being called from? Do you perhaps have something in a local .Rprofile?

It looks like the RETICULATE_PYTHON="managed" is working, since a python and tensorflow installation was automatically available. However, some other (non-standard) system configuration is conflicting.

[polizzi4757]

Thank you very much for the prompt answer.
I am using the desktop version of RStudio (RStudio 2024.09.1 Build 394 "Cranberry Hibiscus" Release (a1fe401fc08c232d470278d1bc362d05d79753d9, 2024-11-03) for windows.
I am also suprised to see a call to use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe").
I have an empty .Rprofile and I couldn't find others looking in cmd with dir C:\ /s /b | findstr ".Rprofile" and dir D:\ /s /b | findstr ".Rprofile". But that shouldn't be too bad because it is ignored thanks to the environment variable RETICULATE_PYTHON set to "managed".
Yes, the RETICULATE_PYTHON="managed" is working.
The Error message of the 5th try does give you an idea of a possible conflicting system configuration ?
I am not exaggerating when I say that I have been looking for days for conflicting system configurations. I would hate to give up the chance to use keras and tensorflow. Please let me know which could be the cause of "R Session Disconnected".
Thank you very much.

[t-kalinowski]

Are you using the latest version of RStudio?

[polizzi4757]

I installed the newest version for window RStudio 2024.12.1 Build 563, but sadly I get the same "R Session Disconnected" problem.

[polizzi4757]

Hello, Tomasz. I have some news.

The call to use_python("C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-reticulate/python.exe") that produced the warning was caused because in Tools → Global Options → Python I select a Python interpreter. So this has been cleared.

Out of desperation, because I couldn't get rid of the Error "R Session Disconnected", I opened R.exe and tried

    library(reticulate)
    reticulate::py_config()

and I did not get any Error. So I wen back in Rstudio and I tryed

   Sys.unsetenv("RETICULATE_PYTHON")
   Sys.setenv(RETICULATE_PYTHON = "C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-tensorflow/python.exe")
   library(reticulate)
   reticulate::py_config()

and I did not get any Error. Because I couldn't believe my eyes, I tryed again

Sys.setenv(RETICULATE_PYTHON = "managed")
library(reticulate)
reticulate::py_config()

and boom I got again "R Session Disconnected".
I confess, I tryed back and forth many times because I wanted to be sure and I got always the same result.
This means, I believe, that my installation did not follow all the right steps and somewhere along the process I made a couple of mistakes.
My reference, to guide the installation, is what you find in different web sites for reticulate, tensorflow and keras3, but I noticed that many things are given for granted, and for newbie like me that can derail the installation.
I give you three relevant exemples of what I said.
First, it is given for granted that to set to "managed" the environment variable RETICULATE_PYTHON can make your life a lot easier.
Second, it should be self-evident that you have better to set the environment variable PYTHONPATH = ~/library/reticulate/python, otherwise you get an error if e.g. you run reticulate::py_config() like "Error in py_module_import(module, convert = convert) :
ModuleNotFoundError: No module named 'rpytools'".
Third, the functiion used for the installation are always used without any of their arguments, but I think that some of them should be considered also for a quick start installation.

I said all this to ask you please to give detailed instructions on how to install properly and thoroughly the reticulate, Miniconda, tensorflow, keras, R tensorflow and keras3, including all needed variables and options.
I am available to help at least to try out how such guide works in Rstudio Desktop installed on Windows 10.
I know I am asking a lot and that is not something that can be done right away, but it will make accesible these instruments to newbi like me. Thank you very much. Have a nice weekend.

[t-kalinowski]

Thanks for reporting, this is certainly frustrating.

Second, it should be self-evident that you have better to set the environment variable PYTHONPATH = ~/library/reticulate/python, otherwise you get an error if e.g. you run reticulate::py_config() like "Error in py_module_import(module, convert = convert) :
ModuleNotFoundError: No module named 'rpytools'".

This is not the case. You should not need to set PYTHONPATH. The fact that you do is evidence of some other issue.

If you run RETICULATE_PYTHON="managed" in R.exe directly (outside of rstudio), does that work?

[polizzi4757]

If you run in R.exe

Sys.setenv(RETICULATE_PYTHON = "managed")
library(reticulate)
reticulate::py_config()

you do not get error,

But what is strange is that, after you set in R.exe RETICULATE_PYTHON = "managed", the value "managed" remains persistent i.e. if you

Sys.unsetenv("RETICULATE_PYTHON")
# or
Sys.setenv(RETICULATE_PYTHON = "C:/Users/Giuseppe/AppData/Local/r-miniconda/envs/r-tensorflow/python.exe")
# or
Sys.setenv(RETICULATE_PYTHON = "")

then after you restart R in RStudio, RETICULATE_PYTHON is still = "managed". I tried many times. But maybe does not make sense what I just tried.