Migrate to Fly and get READMEs working again #164

simonw · 2025-05-14T04:33:16Z

I'm using this PR to document some hacks.

Finish moving hosting to Fly #163

simonw · 2025-05-14T04:34:01Z

build_directory.py

+    # fix_repo_names = [
+    #     row[0]
+    #     for row in db.execute(
+    #         """
+    #     select full_name from repos
+    #     where readme_html like '%camo.githubusercontent.com%'
+    #     and readme not like '%camo.githubusercontent.com%'
+    #     limit 3
+    # """
+    #     ).fetchall()
+    # ]
+    # print("Fixing HTML for {}".format(fix_repo_names))
+    # repos_to_fetch.extend(fix_repo_names)


This was the worst mess: because I lost the old content.db I had to build from scratch, but somewhere our build scripts had depended on the previous DB having those readme and readme_html columns - without those I couldn't bootstrap a new database.

Need to figure out how to rebuild those, while not getting busted by GitHub's rate limit if we need to get READMEs for hundreds of repos.

simonw · 2025-05-14T04:35:23Z

index_tutorials.py

+    db["tutorials"].create({
+        "path": str,
+        "title": str,
+        "body": str,
+    }, pk="path")


Had to explicitly create this table here, because otherwise when Datasette tried to render the /tutorials page this template fragment would throw an error:

datasette.io/templates/_more_tutorials.html

Lines 2 to 7 in a705655

<h3>More tutorials</h3>

<ul>

{% for row in sql("select path, title from tutorials where path != ?", [path], database="content") %}

<li><a href="{{ row.path }}">{{ row.title }}</a></li>

{% endfor %}

</ul>

simonw · 2025-05-14T04:35:49Z

scripts/build.sh

+sqlite-utils drop-table content.db plugin_repos --ignore
+sqlite-utils drop-table content.db tool_repos --ignore


Added --ignore because otherwise we would be dropping a table that didn't exist yet in our fresh content.db database.

simonw · 2025-05-14T04:36:08Z

templates/dogsheep-beta.yml

+                tools.description as search_1, -- || ' ' || coalesce(repos.readme, '') as search_1,
                1 as is_public
            from tools join repos on tools.name = repos.name
        display_sql: |-
            select
              highlight(
-                tools.description || ' ' || coalesce(repos.readme_html, ''), :q
+                tools.description, -- || ' ' || coalesce(repos.readme_html, ''),
+                :q


More hacks to deal with the missing readme and readme_html columns.

#163 (comment)

simonw · 2025-05-16T14:25:47Z

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.22/x64/bin/dogsheep-beta", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/dogsheep_beta/cli.py", line 36, in index
    run_indexer(
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/dogsheep_beta/utils.py", line 48, in run_indexer
    columns = derive_columns(other_db, sql)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/dogsheep_beta/utils.py", line 65, in derive_columns
    cursor = db.conn.execute(sql + " limit 0")
sqlite3.OperationalError: no such table: til

simonw · 2025-05-16T14:29:08Z

Getting closer...

    [n async for n in self.root_render_func(ctx)]  # type: ignore
  File "/home/runner/work/datasette.io/datasette.io/templates/pages/plugins/{name}.html", line 17, in top-level template code
    {% set plugin = plugins[0] %}
  File "/home/runner/work/datasette.io/datasette.io/templates/page_base.html", line 9, in top-level template code
    {% block extra_head %}{% endblock %}
  File "/home/runner/work/datasette.io/datasette.io/templates/pages/plugins/{name}.html", line 22, in block 'extra_head'
    {% if plugin.usesCustomOpenGraphImage %}
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/jinja2/environment.py", line 490, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: list object has no element 0
Test failed to /plugins/datasette-dashboards

simonw · 2025-05-16T14:38:47Z

  File "/home/runner/work/datasette.io/datasette.io/build_stats.py", line 32, in <module>
    cli()
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/runner/work/datasette.io/datasette.io/build_stats.py", line 17, in cli
    stats = json.load(stats_file)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

simonw · 2025-05-16T14:39:38Z

Urgh I bet that was a rate limit too.

simonw · 2025-05-16T14:43:37Z

  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/datasette/database.py", line 211, in in_thread
    return fn(conn)
  File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/site-packages/datasette/database.py", line 237, in sql_operation_in_thread
    cursor.execute(sql, params if params is not None else {})
sqlite3.OperationalError: near ":q": syntax error
Test failed to /-/beta?q=datasette

simonw · 2025-05-16T15:03:51Z

Oh the problem is:

ERROR: conn=<sqlite3.Connection object at 0x7fecbc9a73f0>, sql = "select\n highlight(\n plugins.description || ' ' || coalesce(repos.readme_html, ''), :q\n :q\n ) as snippet,\n usesCustomOpenGraphImage,\n openGraphImageUrl\nfrom\n plugins join repos on plugins.name = repos.name\nwhere\n plugins.name = :key", params = {'key': 'datasette-visible-internal-db', 'q': 'datasette'}: near ":q": syntax error

#164 (comment)

simonw · 2025-05-16T15:13:48Z

Publishing to Fly requires flyctl to be installed and configured

Because it's in a different location:

Run '/home/runner/.fly/bin/flyctl --help' to get started

#164 (comment)

simonw · 2025-05-16T15:22:58Z

Looks promising...

simonw · 2025-05-16T15:24:39Z

https://datasette.io/plugins/datasette-chronicle is now showing the full README!

simonw added 2 commits May 14, 2025 00:14

Some fixes to build.sh

7ef65cb

Refs #163 (comment)

Botched fixes to deploy to Fly, refs #163

4557dff

simonw commented May 14, 2025

View reviewed changes

simonw mentioned this pull request May 14, 2025

Finish moving hosting to Fly #163

Closed

simonw added 4 commits May 14, 2025 00:42

Temporary fix for broken plugin/tool pages

34afedb

#163 (comment)

Ran black

55445ba

Load readmes from new stashed-readmes repo, refs #165

fb43ba6

Merge branch 'main' into fly

ff5276f

simonw marked this pull request as ready for review May 16, 2025 14:13

simonw added 4 commits May 16, 2025 10:20

if_not_exists=True on tutorial creation

a21d7dd

index the new readme_html stuff

28c3ce8

More if_not_exists=True

2d85438

Oops that should not hawe been there

d5197db

simonw changed the title ~~Notes on migration to Fly~~ Migrate to Fly and get READMEs working again May 16, 2025

Temporarily stop indexing tils

8e8986c

Do not test datasette-dashboards page

5e1a815

Fail loudly if package-stats/main/stats.json is not 200

09ff562

Temporary hack to exclude tils.db from dogsheep search index

54621f2

simonw added 2 commits May 16, 2025 11:04

Fix SQL syntax error

ec1c9b0

#164 (comment)

Install datasette-publish-fly

0b22a43

simonw added 2 commits May 16, 2025 11:14

Alias for flyctl

6098357

#164 (comment)

Add flyctl to PATH

906a8bc

simonw merged commit 2bc8a69 into main May 16, 2025
1 check passed

simonw deleted the fly branch May 16, 2025 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Migrate to Fly and get READMEs working again #164

Migrate to Fly and get READMEs working again #164

Uh oh!

simonw commented May 14, 2025

Uh oh!

simonw May 14, 2025

Uh oh!

simonw May 14, 2025

Uh oh!

simonw May 14, 2025

Uh oh!

simonw May 14, 2025

Uh oh!

simonw May 14, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

Uh oh!

Uh oh!

	<h3>More tutorials</h3>
	<ul>
	{% for row in sql("select path, title from tutorials where path != ?", [path], database="content") %}
	<li><a href="{{ row.path }}">{{ row.title }}</a></li>
	{% endfor %}
	</ul>

		sqlite-utils drop-table content.db plugin_repos --ignore
		sqlite-utils drop-table content.db tool_repos --ignore

Uh oh!

Migrate to Fly and get READMEs working again #164

Migrate to Fly and get READMEs working again #164

Uh oh!

Conversation

simonw commented May 14, 2025

Uh oh!

simonw May 14, 2025

Choose a reason for hiding this comment

Uh oh!

simonw May 14, 2025

Choose a reason for hiding this comment

Uh oh!

simonw May 14, 2025

Choose a reason for hiding this comment

Uh oh!

simonw May 14, 2025

Choose a reason for hiding this comment

Uh oh!

simonw May 14, 2025

Choose a reason for hiding this comment

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

simonw commented May 16, 2025

Uh oh!

Uh oh!

Uh oh!