Dynamic Type selection & customization on the frontend

[pavish]

The PR #926 implements the type configuration view for some basic Mathesar types. As discussed in #849 and several other discussions, we intend to make our frontend (and any general-purpose client) type-agnostic eventually.

Inorder to achieve that goal, we need to be able to construct a dynamic UI based on information stored on the db, by using a declarative format.
i. Type selection and customization
ii. Formatting and displaying values in table cell
iii. Inputting values in table cell
iv. Filtering options
v. Grouping options
vi. Default value selection
vii. Data source and formula selection

These are just on top of my head and we may have several other places where this would be required. Essentially any area of the UI that is rendered based on column type would need to be constructed dynamically.

The PR #926 lays the groundwork for all of these usecases while implementing a portion of i.

This discussion is only focused on database options in (i). Type selection and customization i.e this UI:

Reference: Figma prototype

It is split into several sections to facilitate focused conversations.

1. Declarative format to construct the type selection pane
2. Problems we need to handle based on (1)
3. Dealing with type aliases
4. Display options

[pavish]

1. Declarative format to construct the type selection pane

This is the current format, which is implemented in #926.

The configuration for Text Mathesar type:

{
  identifier: 'Text',
  defaultDbType: 'VARCHAR',
  typeSwitchOptions: {
    database: {
      allowDefault: true,
      configuration: {
        form: {
          variables: {
            restrictFieldSize: {
              type: 'boolean',
              default: false,
            },
            fieldSizeLimit: {
              type: 'integer',
              default: 255,
            },
          },
          layout: {
            orientation: 'vertical',
            elements: [
              {
                type: 'input',
                variable: 'restrictFieldSize',
                label: 'Restrict Field Size',
              },
              {
                type: 'if',
                variable: 'restrictFieldSize',
                condition: 'eq',
                value: true,
                elements: [
                  {
                    type: 'input',
                    variable: 'fieldSizeLimit',
                    label: 'Field Size Limit',
                  },
                ],
              },
            ],
          },
        },
        determinationRules: [
          {
            resolve: 'CHAR',
            rule: {
              combination: 'and',
              terms: [
                {
                  id: 'restrictFieldSize',
                  op: 'eq',
                  value: true,
                },
                {
                  id: 'fieldSizeLimit',
                  op: 'lte',
                  value: 255,
                },
                {
                  id: 'fieldSizeLimit',
                  op: 'neq',
                  value: null,
                },
              ],
            },
          },
          {
            resolve: 'VARCHAR',
            rule: {
              combination: 'and',
              terms: [
                {
                  id: 'restrictFieldSize',
                  op: 'eq',
                  value: true,
                },
                {
                  combination: 'or',
                  terms: [
                    {
                      id: 'fieldSizeLimit',
                      op: 'gt',
                      value: 255,
                    },
                    {
                      id: 'fieldSizeLimit',
                      op: 'eq',
                      value: null,
                    },
                  ],
                },
              ],
            },
          },
          {
            resolve: 'TEXT',
            rule: {
              id: 'restrictFieldSize',
              op: 'eq',
              value: false,
            },
          },
        ],
        ruleReversalValues: {
          CHAR: {
            restrictFieldSize: true,
            fieldSizeLimit: 255,
          },
          VARCHAR: {
            restrictFieldSize: true,
            fieldSizeLimit: null,
          },
          TEXT: {
            restrictFieldSize: false,
            fieldSizeLimit: null,
          },
        },
      },
    },
  },
};

Please also refer the configuration for number, which is a bit more sophisticated: https://github.com/centerofci/mathesar/pull/926/files#diff-1a8cf2364d3f861e74c2e0cf0df43feb6be483ff9830ef16cba6e7c5887120d6

Description:

• defaultDbType refers to the database type that is selected by default when the user selects the Mathesar type.
• typeSwitchOptions contains all the configurations for rendering everything on the type selection dropdown.
  • database contains the database tab within the type selection dropdown. Similar to this, we'd have the display options.
    • form refers to the form that is rendered on the dropdown. This configuration is used internally by the form-builder component, so the configuration is more general-purpose than mathesar-specific.
      • variables only hold data integrity specific configurations.
      • layout holds all the presentational aspects.
    • determinationRules are rules that are executed when user modifies the form. These rules help identify the database type based on user input.
    • ruleReversalValues are used to fill the values in the form based on the selected database type. This is supposed to be a temporary fix. Please see #Problem 2 in (2) for more details on this.

Apart from these, this will extend to support input, icon and other configurations.

What I expect out of this thread:

• Does this structure make sense from a backend-perspective?
  • Eventually, this would be a response from the backend (it would be a client specific api).
  • I'm open to modifying the format as long as all the above information can be provided in some way.
• How can we improve names of the properties? I'm terrible at naming things and open to suggestions on this.
• Is there anything we could do to make this simpler?

[kgodey]

I confirmed with Pavish on Matrix that this topic does not need to be resolved until after the alpha release. I'm going to defer reviewing the format until we handle #661.

[pavish]

[RESOLVED] 2. Problems we need to handle based on (1)

Problem 1:

• Currently, we allow the user to enter direct values in the type selection dropdown. Eg., Field size limit in Text mathesar type and Decimal places and Max digits in Number mathesar type.
• We select the db type based on these configurations, as you can see in (1).
• The states for these inputs have to be persisted for when the user opens the type selection dropdown again, across sessions and browsers.
• Also, we may want to actually reflect this on the database. eg., VARCHAR(3000).

Problem 2:

• The UI needs to be able to guess input values based on selected database type.
• Eg., for the Number mathesar type, the UI has to guess values for Number Type, Integer Data Size, Decimal places eg., based on the selected db type.
• In cases where the inputs allow direct values, there is no definite one-to-one link. So, it's impossible to guess values like Decimal places and Max digits.
• It would hard to guess values by reversing the determinationRules structure. The simplest workaround was to introduce ruleReversalValues, which I am not fond of but anything else would make this more complicated.

Solutions that resolve both problems 1 and 2:

Solution A:

• We need an API on the backend to store and return user input values.
• There are some inputs which need not essentially be stored on the db like Restrict Field Size but it would make it a lot easier for the frontend if we did.
• We would have to POST AND GET the values for anything within typeSwitchOptions.database.form.variables.
• Example response for text type:

  {
    restrictFieldSize: true,
    fieldSizeLimit: 3000,
  }

• We have to store display_options on the backend either way, so storing these additionally makes sense.
• Since we are storing these on the backend, it also makes more sense to calculate the db type from the configuration on the backend itself.
• This will need some UX reconsiderations to show loading states and error states within the dropdown.

Solution B:

• Update the UX to remove direct inputs with text boxes, and simplify rules to only perform value matching.
• For eg., Instead of a text field saying Field size limit, we'd have a Select menu with options "Small text" and "Large text".
• The determinationRules would not be complex. It would only hold the values to match. This removes the need for ruleReversalValues.
  eg.,

    determinationRules: [
      {
        resolve: 'CHAR',
        rule: {
          restrictFieldSize: true,
          fieldSize: 'small'
        },
      },
      {
        resolve: 'VARCHAR',
        rule: {
          restrictFieldSize: true,
          fieldSize: 'large'
        },
      },
      {
        resolve: 'TEXT',
        rule: {
          restrictFieldSize: false,
        },
      },
    ],

• The server would not have to store the values for user inputs.
• The drawback is that the UX for type selection can never have direct text inputs.

What I expect out of this thread:

• Which of these solutions make sense?
• Do you have any additional suggestions to handle these problems?

[kgodey]

In cases where the inputs allow direct values, there is no definite one-to-one link. So, it's impossible to guess values like Decimal places and Max digits.

I'm not sure what you mean by "there is no definite one-to-one link". These options map to the precision and scale type options related to the DB type. Also, they are optional, so it's fine to leave them blank unless precision and/or scale are specified.

[pavish]

Ah. My bad, I just found out the type_options property is present in columns. I didn't know this existed.

Thanks. I think that makes this particular comment void.

[kgodey]

Everything under "Database Options" in the UI should map to something in the API, that was part of the design requirements and API specs.

[pavish]

3. Dealing with type aliases

• Currently in Number, we have NUMERIC and DECIMAL which are both aliases of each other (as far as I understand).
• This leads to the configuration views and criteria of both being the same.
• How would the frontend know which one of these to display and send in the column update request?

Solution A:

• Update the format in (1) to support aliases.

Solution B:

• Update the /types endpoint to provide alias information.

What I expect out of this thread:

• Which of these solutions make sense?
• Any further suggestions.

[kgodey]

I think solution B makes sense here. @dmos62 is revamping the types APIs as part of the filtering work anyway.

For existing columns, the frontend should show whichever DB type the API returns. When the user is configuring a new column, the frontend should use the DB type that's the "default" and not the alias. (e.g. if NUMERIC is the default and DECIMAL is the alias, we should use NUMERIC).

[pavish]

@kgodey Thanks, I believe this is the only action item to implement pre-alpha.

@dmos62 Can we expect this to be included as part of your types endpoint revamp?

Also, Please tag the frontend team for review when there are changes in existing endpoints, so that we are upto date with changes.

[dmos62]

@pavish I'll tag you and Sean when proposing API changes.

I opened a ticket to track adding alias information to the db_types endpoint: #1037

[mathemancer]

I'll get more in-depth in other places, but I think showing both NUMERIC and DECIMAL to the front end should be considered a bug. There's no compelling reason I'm aware of to have the front end know about or deal with aliases at all.

[pavish]

4. Display options

• We have display_options on the backend but it cannot be used by clients to construct a UI.
• For eg., We have this for boolean:

  "display_options": {
    "options": [
      {
        "name": "input",
        "type": "string",
        "enum": [
          "dropdown",
          "checkbox"
        ]
      },
      {
        "name": "custom_labels",
        "type": "object",
        "items": [
          {
              "name": "TRUE",
              "type": "string"
          },
          {
              "name": "FALSE",
              "type": "string"
          }
        ]
      }
    ]
  }

• There is no indication that custom_labels is something we need to only display when the value for input is dropdown.
• The type within each option refers to the type of value sent in request, however there is no indication on the type of input required to be shown to the user.
• This format will need everything to be hardcoded on the frontend, making the need for display_options mostly void.

What I expect out of this thread:

• Can we update this to provide a structure similar to typeSwitchOptions.database.form in (1)?
• To what endpoint does the client send the user-entered values of display_options? Does one exist?
• Any ideas or suggestions

[kgodey]

This format will need everything to be hardcoded on the frontend, making the need for display_options mostly void.

The current goal display_options is to store column configuration, not to build a UI. The only fields on there are fields that the frontend should set.

Can we update this to provide a structure similar to typeSwitchOptions.database.form in (1)?

I think we should defer anything related to using the endpoint to construct a UI until #661.

To what endpoint does the client send the user-entered values of display_options? Does one exist?

This should be the same endpoint - you should send user-entered values to the display_options key in the columns API. If this is not working for anything, please open a bug. Please note that backend APIs for all data types are not yet complete.