Merge #28

28: Add Backfilled data script r=alallema a=alallema # Pull Request ## What does this PR do? Add a script to back-filled data from Firestore to MeiliSearch to give the possibility to import existing documents or reindex them if you already have documents in a Firestore Collection when you enable the MeiliSearch extension. Check [an example from google extension](https://github.com/firebase/extensions/tree/master/firestore-bigquery-export#backfill-your-bigquery-dataset) #15 ## PR checklist - [x] Handles different possibilities to launch the script - [x] Command line with parameter - [x] interactively - [x] Add the possibility to import documents by batch - [x] Add the possibility to import documents from sub-collections - [x] Write a guide **Notes:** - The ability to run the script with a configuration file is not part of the MVP and may be implemented later #29 Co-authored-by: alallema <[email protected]> Co-authored-by: Amélie <[email protected]>
meilisearch · Mar 17, 2022 · e4d139b · e4d139b
2 parents 6f9b965 + 5273516
commit e4d139b
Show file tree

Hide file tree

Showing 25 changed files with 6,141 additions and 17,808 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -20,23 +20,23 @@ jobs:
         with:
           node-version: 14
       - name: Install dependencies
-        run: npm run install:functions
+        run: yarn install:functions
       - name: Run style check
-        run: npm run lint
+        run: yarn lint
 
   integration-tests:
+    name: integration-tests
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        node: ["12", "14"]
-    name: integration-tests node.js_${{ matrix.node }}_test
+        node: ["14", "16", "17"]
     steps:
       - uses: actions/checkout@v2
       - name: Setup node
         uses: actions/setup-node@v1
         with:
           node-version: ${{ matrix.node }}
       - name: Install dependencies
-        run: npm run install:functions
+        run: yarn install:functions
       - name: Run tests with coverage
-        run: npm run test-coverage
+        run: yarn test:coverage
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -34,13 +34,17 @@ First of all, thank you for contributing to Meilisearch! The goal of this docume
 
 To run this project, you will need:
 
-- Node >= 12
+- Node >= 14 && node <= 17
 - Npm >= v7
 - A google account
 - Latest version of `firebase-tools` the Firebase CLI:
-``` bash
-npm install -g firebase-tools
-```
+  ``` bash
+  yarn global add firebase-tools
+  ```
+  Add the directory for the commands of the packages installed globally in yarn, to access of firebase binary:
+  ``` bash
+  export PATH="$(yarn global bin):$PATH"
+  ```
 
 ### Setup <!-- omit in toc -->
 
@@ -54,21 +58,25 @@ firebase --open-sesame extdev
 ```
 Install dependencies:
 ``` bash
-npm run install:functions
+yarn install:functions
 ```
 Build the project:
 ``` bash
-npm run build
+yarn build
 ```
 Launch Meilisearch instance:
 ``` bash
-docker pull getmeili/meilisearch:latest # Fetch the latest version of Meilisearch image from Docker Hub
-docker run -p 7700:7700 getmeili/meilisearch:latest ./meilisearch --master-key=masterKey --no-analytics=true
+curl -L https://install.meilisearch.com | sh # download Meilisearch
+./meilisearch --master-key=masterKey --no-analytics # run Meilisearch
 ```
 Launch emulator:
 ``` bash
 firebase ext:dev:emulators:start --test-params=test-params-example.env --import=dataset --project=name-of-the-project
 ```
+or just
+``` bash
+yarn emulator
+```
 The emulator runs with environment parameters found in `test-params-example.env` and with a provided dataset found in `/dataset`.
 
 Once it is running, open the emulator in a browser at the following address: http://localhost:4000
@@ -80,20 +88,62 @@ NB: If you want to change your Meilisearch credentials or the plugins options yo
 Each PR should pass the tests and the linter to be accepted.
 
 ```bash
+curl -L https://install.meilisearch.com | sh # download Meilisearch
+./meilisearch --master-key=masterKey --no-analytics # run Meilisearch
+
 # Tests
-npm run test
+yarn test
 
 # Tests in watch mode
-npm run test:watch
+yarn test:watch
 
 # Linter
-npm run lint
+yarn lint
 
 # Linter with fixing
-npm run lint:fix
+yarn lint:fix
 
 # Build the project
-npm run build
+yarn build
+```
+
+### Run the backfilled-data script
+
+Run the import script using [`npx` (the Node Package Runner)](https://www.npmjs.com/package/npx).
+- Set up credentials. The import script uses the application's default credentials to communicate with Firebase. Please follow the instructions [generate a private key file for your service account](https://firebase.google.com/docs/admin/setup#initialize-sdk).
+
+- Run the import script interactively using `npx` and run ONE of the following commands:
+  - Run interactively:
+    ```bash
+    npx firestore-meilisearch
+    ```
+
+  - Or run non-interactively with paramaters:
+    ```bash
+    npx firestore-meilisearch \
+      --project <project_id> \
+      --source-collection-path <collection_name> \
+      --index <index_uid> \
+      --batch-size <100/default=300> \
+      --host <host_address> \
+      --api-key <api_key> \
+      --non-interactive
+    ```
+    **Note**: The `--batch-size` and `--query-collection-group` arguments are optional. To see its usage, run the above command with `--help`.
+
+- Run the project for development:
+Launch Meilisearch instance:
+``` bash
+curl -L https://install.meilisearch.com | sh # download Meilisearch
+./meilisearch --master-key=masterKey --no-analytics # run Meilisearch
+```
+Launch the watcher on the project:
+``` bash
+yarn watch
+```
+Launch the watcher on the script. You have to modify the informations of the playground script by your own parameters inside the `package.json` file:
+``` bash
+yarn playground
 ```
 
 ## Git Guidelines
@@ -139,7 +189,26 @@ _[Read more about this](https://github.com/meilisearch/integration-guides/blob/m
 
 ### How to Publish the Release <!-- omit in toc -->
 
-WIP
+⚠️ Before doing anything, make sure you got through the guide about [Releasing an Integration](https://github.com/meilisearch/integration-guides/blob/main/resources/integration-release.md).
+
+1. Make a PR modifying the files [`package.json`](/package.json), [`package.json` in functions directory](/functions/package.json) and the [`extension.yaml`](/extension.yaml) with the right version.
+
+```yaml
+"version": "X.X.X",
+```
+
+2. Test the extension  by installing it on Firestore database:
+```bash
+firebase ext:install . --project=meilisearch-e2fe1
+```
+
+3. Publish the extension by run the following command in the root of the extension directory:
+```bash
+firebase ext:dev:publish meilisearch/firestore-meilisearch --project=meilisearch-e2fe1
+```
+**Note**: `meilisearch` is the publisher id and `meilisearch-e2fe1` is the publish project id for this extension.
+
+Once the changes are merged on `main`, you can publish the current draft release via the [GitHub interface](https://github.com/meilisearch/meilisearch-go/releases): on this page, click on `Edit` (related to the draft release) > update the description (be sure you apply [these recommandations](https://github.com/meilisearch/integration-guides/blob/main/resources/integration-release.md#writting-the-release-description)) > when you are ready, click on `Publish release`.
 
 <hr>
 

diff --git a/extension.yaml b/extension.yaml
@@ -1,4 +1,4 @@
-# Copyright 2021 Meilisearch
+# Copyright 2022 Meilisearch
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,7 +13,7 @@
 # limitations under the License.
 
 # Identifier for your extension
-name: firestore-meilisearch-search
+name: firestore-meilisearch
 
 # Follow semver versioning
 version: 0.1.0
@@ -22,7 +22,7 @@ version: 0.1.0
 specVersion: v1beta
 
 # Friendly display name for your extension (~3-5 words)
-displayName: Search in Firestore with MeiliSearch
+displayName: Search in your Firestore content with Meilisearch
 
 # Brief description of the task your extension performs (~1 sentence)
 description: >-
@@ -42,7 +42,7 @@ author:
 
 # Specify whether a paid-tier billing plan is required to use your extension.
 # Learn more in the docs: https://firebase.google.com/docs/extensions/alpha/ref-extension-yaml#billing-required-field
-billingRequired: false
+billingRequired: true
 
 # In an `apis` field, list any Google APIs (like Cloud Translation, BigQuery, etc.)
 # required for your extension to operate.
@@ -118,7 +118,6 @@ params:
         value: southamerica-east1
       - label: Sydney (australia-southeast1)
         value: australia-southeast1
-    default: europe-west1
     required: true
     immutable: true
 
@@ -132,31 +131,31 @@ params:
     validationErrorMessage: Must be a valid Cloud Firestore Collection.
     required: true
 
-  - param: FIELDS_TO_INDEX
+  - param: MEILISEARCH_FIELDS_TO_INDEX
     label: Fields to index in Meilisearch
     description: >-
       What are the names of the fields you want to be indexed in Meilisearch?
       Create a comma-separated list of the fields name, or leave blank to take all fields.
       The id field is always indexed even when omitted in the list.
+    default: ''
     example: title,overview,release_date
     validationRegex: "^[^,]?[a-zA-Z-_0-9,]*[^,]$"
     validationErrorMessage:
       Fields must be given through a comma-separated list.
-    default: ''
     required: false
 
-  - param: SEARCHABLE_FIELDS
+  - param: MEILISEARCH_SEARCHABLE_FIELDS
     label: Fields in which to search. - Optional
     description: >-
       What are the names of the fields you want to make searchable in Meilisearch?
       This features is optional. See [the documentation about it](https://docs.meilisearch.com/reference/features/field_properties.html#searchable-fields)
       for more information.
       Create a comma-separated list of fields, or leave blank to include all fields.
+    default: ''
     example: title,overview,release_date
     validationRegex: "^[^,]?[a-zA-Z-_0-9,]*[^,]$"
     validationErrorMessage:
       Fields must be given through a comma-separated list.
-    default: ''
     required: false
 
   - param: MEILISEARCH_INDEX_NAME
@@ -175,6 +174,7 @@ params:
     label: Meilisearch host
     description: >-
       What is the url of the host of your Meilisearch database?
+      Make sure your url starts with `http://` or `https://`
     type: string
     example: 'http://127.0.0.1:7700'
     required: true

diff --git a/functions/__tests__/adapter.test.ts b/functions/__tests__/adapter.test.ts
@@ -18,12 +18,28 @@ describe('extensions process', () => {
     test('adaptDocument with fields not set', () => {
       mockedGetFieldsToIndex.mockReturnValueOnce([])
       const snapshot = firebaseMock.firestore.makeDocumentSnapshot(
-        defaultDocument,
-        'docs/1'
+        defaultDocument.document,
+        `docs/${defaultDocument.id}`
       )
-      expect(adaptDocument(defaultDocument.id, snapshot)).toStrictEqual(
-        defaultDocument
+
+      expect(adaptDocument(defaultDocument.id, snapshot)).toStrictEqual({
+        _firestore_id: defaultDocument.id,
+        ...defaultDocument.document,
+      })
+    })
+
+    test('adaptDocument with fields not set and with id field in document', () => {
+      mockedGetFieldsToIndex.mockReturnValueOnce([])
+      const snapshot = firebaseMock.firestore.makeDocumentSnapshot(
+        { id: '12345', ...defaultDocument.document },
+        `docs/${defaultDocument.id}`
       )
+
+      expect(adaptDocument(defaultDocument.id, snapshot)).toStrictEqual({
+        _firestore_id: defaultDocument.id,
+        id: '12345',
+        ...defaultDocument.document,
+      })
     })
 
     test('adaptDocument with fields set', () => {
@@ -33,14 +49,15 @@ describe('extensions process', () => {
         'release_date',
       ])
       const snapshot = firebaseMock.firestore.makeDocumentSnapshot(
-        defaultDocument,
-        'docs/1'
+        defaultDocument.document,
+        `docs/${defaultDocument.id}`
       )
+
       expect(adaptDocument(defaultDocument.id, snapshot)).toStrictEqual({
-        id: defaultDocument.id,
-        title: defaultDocument.title,
-        overview: defaultDocument.overview,
-        release_date: defaultDocument.release_date,
+        _firestore_id: defaultDocument.id,
+        title: defaultDocument.document.title,
+        overview: defaultDocument.document.overview,
+        release_date: defaultDocument.document.release_date,
       })
     })
   })
@@ -54,6 +71,7 @@ describe('extensions process', () => {
     })
     test('adaptValues a geo point value', () => {
       const geoPoint = new firestore.GeoPoint(48.866667, 2.333333)
+
       expect(adaptValues('_geo', geoPoint)).toStrictEqual([
         '_geo',
         {
@@ -67,6 +85,7 @@ describe('extensions process', () => {
     })
     test('adaptValues a wrong geo point value', () => {
       const geoPoint = new firestore.GeoPoint(48.866667, 2.333333)
+
       expect(adaptValues('wrong_geo', geoPoint)).toStrictEqual([
         'wrong_geo',
         geoPoint,