Current issues with the dev version | Best way to support you bringing this forward?

[codiflow]

Hey @PatWie, I really appreciate your work and now the builds from the dev channel are working. 🥳

But there's still some work to do until the firebase storage is being replaced completely.

One thing that still seems to be missing is the storage of blobs (images and other uploads). They don't use firestore.googleapis.com but firebasestorage.googleapis.com and those requests are not being replaced currently.

I'm a DevOps guy and not a developer but I really would like to help you with bringing this forward. What is the best way to help you with this?

Thanks 😎
Christian

[codiflow]

Now as I was playing around with your repo a bit these are my results:

My setup:

• I was replacing locahost:3002 in main.go with a publicly resolvable hostname which is SSL terminated by nginx
• I replaced "ssl=!0", "ssl=0" with "ssl=!0", "ssl=1" and "ssl:!0", "ssl:0" with "ssl:!0", "ssl:1" to be able to use HTTPS
• I also set up a second setup using all the default values to cross-check the functionality

---

My findings:

1. ✅ Collaboration is working out of the box without issues.
2. ❌ The backend storage is not working (correctly) (neither in my SSL terminated setup nor in the default localhost setup). So you will always get the following error message after starting a collaboration session:

Couldn't save to the backend database. If problems persist, you should save your file locally to ensure you don't lose your work.

3. ✅ Apart from that requests like the following are getting through:

http://localhost:3002/v1/projects/excalidraw-room-persistence/databases/(default)/documents:batchGet

4. 💡 Using encryption functions in the browser is only possible with SSL encrypted connections. You will get errors in the browser console if that's not the case.

5. ❌ I recognized that the following initial request is failing in both variants (SSL terminated and localhost). This is one of the requests where firebase.googleapis.com was replaced by localhost:3002:

http://localhost:3002/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fexcalidraw-room-persistence%2Fdatabases%2F(default)&VER=8&RID=40837&CVER=22&X-HTTP-Session-Id=gsessionid&%24httpHeaders=X-Goog-Api-Client%3Agl-js%2F%20fire%2F8.3.3%0D%0AX-Firebase-GMPID%3A1%3A654800341332%3Aweb%3A4a692de832b55bd57ce0c1%0D%0AContent-Type%3Atext%2Fplain%0D%0A&zx=66dvi15g5r94&t=1

The chi middleware responds with error 404 here.

6. ❌ The firebase storage firebasestorage.googleapis.com (which is being used for saving blobs like images) is not being hot-patched yet. You can try that on your own by uploading an image in Excalidraw and watch the browser logs (you have to be in a collaboration session during this test otherwise the image is only saved in the browsers storage).

This is an example request being sent while uploading a file:

OPTIONS https://firebasestorage.googleapis.com/v0/b//o?name=files/rooms/30f11559ba40ec4fc013/c48cb6746fe04d40064a3767d3033fec6d6777c9

POST https://firebasestorage.googleapis.com/v0/b//o?name=files%2Frooms%2F30f11559ba40ec4fc013%2Fc48cb6746fe04d40064a3767d3033fec6d6777c9

If I can be of any help with any of these issues please let me know 😎

[PatWie]

I currently lack a clear understanding of which stubs or functionality we need to re-implement from the Firebase API. This could potentially be a significant time-sink. I still prefer not to modify any frontend code and instead focus on providing correct request/response handling for all fetches. My goal is to eventually enable easy self-hosting on AWS Fargate. However, I must admit, I am not super keen in extensive programming during my spare time at the moment.

The backend storage issue, i don't get. It works for me with link sharing using S3. Interestingly, I didn't know that one can embed arbitrary images 🙈. That's news to me.

[codiflow]

I still prefer not to modify any frontend code and instead focus on providing correct request/response handling for all fetches.

Totally agree with that. Unfortunately I never used Firebase before so I can't provide any help either. Maybe I get a better understanding while diving deeper into the code.

What I can say is that there is a working (but outdated) fork out there where the upstream code was modified so a httpStorage backend can be used instead of Firebase. This is the repository: https://github.com/squatica/excalidraw

My goal is to eventually enable easy self-hosting on AWS Fargate. However, I must admit, I am not super keen in extensive programming during my spare time at the moment.

OK I won't use any Amazon services so this is where I'm out 😉 Anyway – I'm sure your approach is so great that we can evolve it so it's becoming an alternative to the current (code modifying) forks. Maybe I can find some other Go contributors who can help here 🙏

The backend storage issue, i don't get. It works for me with link sharing using S3.

Could you maybe try if the backend also works without S3? I'm using an sqlite db at the moment and will try it with the directory storage today.

Interestingly, I didn't know that one can embed arbitrary images 🙈. That's news to me.

Yes, that's one of the greatest features apart from already being a great piece of software at all 😎

[nhatsangvn]

Hey @PatWie ,

This is great work, I really appreciated.

One issue that I got is even I set (before starting the ./excaldraw-complete)

export STORAGE_TYPE=sqlite
export DATA_SOURCE_NAME=/tmp/test.sqlite

the app seems doesn't use it. Although I can see /tmp/test.sqlite but when I draw some charts, that files doesn't change in size. Anyway that I can verify if the app actually use my sqlite db? (1)

Also is the the popup

relating to my STORAGE_TYPE above? (2)

[PatWie]

I added a commit with logging (and updated README.md). I verified S3, filesystem, memory and sqlite storage works here on Linux. Can you provide the logging output and OS info?

[nhatsangvn]

Nevermind, I understand this feature. It only works if we use Save to > Sharable link.
Thank you @PatWie , I just re-tested it but the issue is still happened on my end.
My OS is Debian 12.
Log output:
~~ ~~root@snn-buil:/app# STORAGE_TYPE=sqlite DATA_SOURCE_NAME=test.db go run main.go --loglevel debug --listen=":3002"~~ ~~INFO[0000] Use storage dataSourceName=test.db storageType=sqlite~~ ~~INFO[0000] starting server addr=":3002"~~ ~~DEBU[0000] Server is running in the background~~ ~~2024/06/10 17:34:20 "GET http://mydomain/socket.io/?EIO=4&transport=websocket HTTP/1.1" from 127.0.0.1:33234 - 000 0B in 410.594µs~~ ~~init room %!(EXTRA socket.Room=iuEGYVNJd9S8HwAAAAAAAAAB)~~ ~~Socket iuEGYVNJd9S8HwAAAAAAAAAB has joined aa5a032c1897709ca992~~ ~~ room %!(EXTRA socket.Room=aa5a032c1897709ca992, string= has users , []socket.SocketId=[iuEGYVNJd9S8HwAAAAAAAAAB])~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~...~~ ~~

~~This is the `.env.production` that I used to build the frontend:~~
~~```~~
~~VITE_APP_BACKEND_V2_GET_URL=https://mydomain/api/v2/~~
~~VITE_APP_BACKEND_V2_POST_URL=https://mydomain/api/v2/post/~~

VITE_APP_LIBRARY_URL=https://libraries.excalidraw.com
VITE_APP_LIBRARY_BACKEND=https://mydomain/libraries
VITE_APP_PLUS_LP=https://mydomain/plus/
VITE_APP_PLUS_APP=https://mydomain/app/

~~VITE_APP_AI_BACKEND=https://mydomain/ai/~~

VITE_APP_WS_SERVER_URL=https://mydomain

~~VITE_APP_FIREBASE_CONFIG='{"apiKey":"xxx-yyy","authDomain":"test-note.firebaseapp.com","databaseURL":"https://test-note.firebaseio.com","projectId":"test-note","storageBucket":"test-note.appspot.com","messagingSenderId":"201312225267","appId":"zzz"}'~~

VITE_APP_DISABLE_TRACKING=yes
```

~~When I started the server with~~
~~```~~
~~STORAGE_TYPE=sqlite DATA_SOURCE_NAME=test.db go run main.go --loglevel debug --listen=":3002"~~
~~```~~
~~I can see the `test.db` is there, but no new content is written.~~
~~<img width="1020" alt="image" src="https://github.com/PatWie/excalidraw-complete/assets/20770118/bf37c83c-1c90-4c1e-beec-a8f2eae4fbad">~~

After login to my domain (the app behind the reverse_proxy), open a live-collab, wait a few seconds and this issue happened.

~~I also notice that this request happened when I inspect the network:~~
~~<img width="1431" alt="image" src="https://github.com/PatWie/excalidraw-complete/assets/20770118/cbafd927-4c7c-4b20-b92e-ac1eaf7bf575">~~

May I know how did you test for the STORAGE_TYPE? Did you go to UI and hover around or which method?

[PatWie]

The issue in your screenshot is likely due to the required SSL for Firebase in the front-end implementation. One workaround is running Excalidraw locally using a reverse-proxy setup with Caddy. Caddy can automatically generate valid, self-signed SSL certificates and install them in the browser

[nhatsangvn]

Hi @PatWie , one new issue that I observed in the Room Server is:

when I try to open a colab room, sometimes the people in the rooms keep increasing although only 2 of my browsers are active.

Basically when I draw for sometimes (10s) and the number of people increases.

[PatWie]

This is unfortunately a known issue without a known root-cause:
#3