use SQLITE_DATABASE resource type, to permit larger file uploads

[jordanverasamy]

WHY are these changes introduced?

Fixes https://github.com/Shopify/workflows-operations/issues/3359

WHAT is this pull request doing?

This PR updates the SQLite file upload functionality to:

1. Use the SQLITE_DATABASE resource type instead of FILE for staged uploads
2. Increase the maximum file size limit from 20MB to 300MB
3. Add API version parameter to the staged upload request (using 'unstable')
4. Fix the resource URL construction by appending the key parameter value
5. TEMPORARILY Add SQLite-specific types until SQLITE_DATBASE resource type visibility changes are in place

How to test your changes?

npm shopify store copy --from-file file_bigger_than_20_mb_smaller_than_300mb.sqlite --to-store target_store
with an sqlite file

Measuring impact

How do we know this change was effective? Please choose one:

• n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
• Existing analytics will cater for this addition
• PR includes analytics changes to measure impact

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes

[jordanverasamy]

• use SQLITE_DATABASE resource type, to permit larger file uploads #6098 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Coverage report

St.❔	Category	Percentage	Covered / Total
🟡	Statements	78.25%	13040/16665
🟡	Branches	72.42%	6348/8765
🟡	Functions	78.2%	3375/4316
🟡	Lines	78.66%	12349/15699

Test suite run success

3037 tests passing in 1318 suites.

Report generated by 🧪jest coverage report action from 2468e7c

[odacrem]

?  Import data from large.sqlite to pablo-dev-store-10.myshopify.com?
✔  Yes, import

╭─ info ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                           │
│  Import Operation                                                                                                                                                                         │
│                                                                                                                                                                                           │
│  From: large.sqlite                                                                                                                                                                       │
│  To:   pablo-dev-store-10.myshopify.com                                                                                                                                                   │
│                                                                                                                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


╭─ error ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                           │
│  Operation failed                                                                                                                                                                         │
│                                                                                                                                                                                           │
│                                                                                                                                                                                           │
│  The Admin GraphQL API responded unsuccessfully with errors:                                                                                                                              │
│                                                                                                                                                                                           │
│  [                                                                                                                                                                                        │
│    {                                                                                                                                                                                      │
│      "message": "Variable $input of type [StagedUploadInput!]! was provided invalid value for 0.resource (Expected \"SQLITE_DATABASE\" to be one of: COLLECTION_IMAGE, FILE, IMAGE,       │
│  MODEL_3D, PRODUCT_IMAGE, SHOP_IMAGE, VIDEO, BULK_MUTATION_VARIABLES, RETURN_LABEL, URL_REDIRECT_IMPORT)",                                                                                │
│      "locations": [                                                                                                                                                                       │
│        {                                                                                                                                                                                  │
│          "line": 1,                                                                                                                                                                       │
│          "column": 30                                                                                                                                                                     │
│        }                                                                                                                                                                                  │
│      ],                                                                                                                                                                                   │
│      "extensions": {                                                                                                                                                                      │
│        "value": [                                                                                                                                                                         │
│          {                                                                                                                                                                                │
│            "resource": "SQLITE_DATABASE",                                                                                                                                                 │
│            "filename": "database.sqlite",                                                                                                                                                 │
│            "mimeType": "application/x-sqlite3",                                                                                                                                           │
│            "httpMethod": "POST",                                                                                                                                                          │
│            "fileSize": "24006656"                                                                                                                                                         │
│          }                                                                                                                                                                                │
│        ],                                                                                                                                                                                 │
│        "problems": [                                                                                                                                                                      │
│          {                                                                                                                                                                                │
│            "path": [                                                                                                                                                                      │
│              0,                                                                                                                                                                           │
│              "resource"                                                                                                                                                                   │
│            ],                                                                                                                                                                             │
│            "explanation": "Expected \"SQLITE_DATABASE\" to be one of: COLLECTION_IMAGE, FILE, IMAGE, MODEL_3D, PRODUCT_IMAGE, SHOP_IMAGE, VIDEO, BULK_MUTATION_VARIABLES, RETURN_LABEL,   │
│  URL_REDIRECT_IMPORT"                                                                                                                                                                     │
│          }                                                                                                                                                                                │
│        ]                                                                                                                                                                                  │
│      }                                                                                                                                                                                    │
│    }                                                                                                                                                                                      │
│  ]                                                                                                                                                                                        │
│                                                                                                                                                                                           │
│  Request ID: 198e8bea-0669-4eb9-8604-36a5604c8e6c-1752292381                                                                                                                              │
│                                                                                                                                                                                           │
│                                                                                                                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Sadly there is more to be done here methinks; when I run this command I get the above error.

[jordanverasamy]

Sadly you're right :( I have no idea why though, I was pretty convinced this should work.

It should just require bulk_data_transfer_public to be visible:
https://github.com/shop/world/blob/main/areas/core/shopify/components/platform/admin/app/models/graph_api/admin/upload_resource.rb#L181-L188

And that is manually set to true for the CLI app: https://experiments.shopify.com/flags/bulk_data_transfer_public

@rzaharenkov any idea why this isn't visible to the CLI? 🤔

[jordanverasamy]

ahh looks like this explains it! https://github.com/Shopify/workflows-operations/issues/3359#issuecomment-3070122460

[github-actions]

We detected some changes at packages/*/src and there are no updates in the .changeset.
If the changes are user-facing, run pnpm changeset add to track your changes and include them in the next release CHANGELOG.

Caution

DO NOT create changesets for features which you do not wish to be included in the public changelog of the next CLI release.