Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Dictionary Version 2 #24

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
78aefc8
DataDictionary Frontend Skeleton
dws4 May 17, 2024
c5981ab
Sorted by Recent Update and Duplicate Entries
dws4 May 24, 2024
4bef40e
task/DataDictionaryFrontend: Adding Test of spacing for subcomponents
dws4 May 28, 2024
6bd112c
Testing structure for table data
dws4 May 29, 2024
3d80a10
DataDictionaryFrontend: Adding subcols and styling
dws4 May 30, 2024
a809dc2
DataDictionaryFrontend: New Button Method per dup col
dws4 May 30, 2024
4e820ce
DataDictionaryFrontend: New Button Method per dup col Updated
dws4 May 30, 2024
e4b1795
DataDictionaryFrontend: Fixing and Adjusting subtables and simplifyin…
dws4 May 31, 2024
d5b89b2
Fixing Styling
dws4 Jun 4, 2024
d1a054e
Fixing to filter by lastUpdated
dws4 Jun 11, 2024
0fc9f5f
Removing unecessary files
dws4 Jun 11, 2024
20037d9
Finishing Refactor of Vue Component, Renaming, and Optimizing
dws4 Jun 11, 2024
41c962e
Fixing Issues with keys and other files optimized with git
dws4 Jun 27, 2024
769fcb8
Checkpoint for Expansion on Buttons
dws4 Jun 28, 2024
f9c1dbc
Fixed Edge Case, Need to fix Button Styling and Banner
dws4 Jul 5, 2024
6f74bb8
Fixed Buttons on Expand and Search
dws4 Jul 8, 2024
8b4cf54
Fixing the CSS and Adding Banner
dws4 Jul 8, 2024
f513c11
Fixing the CSS and Adding Banner 2
dws4 Jul 8, 2024
ec70fb5
Constructing deployment jar Phase 1
dws4 Jul 16, 2024
44c144e
got the controller working
jwomeara Jul 18, 2024
63f03b3
Data Dictionary REST Controller Optimization and Refactor (V1 & V2)
dws4 Jul 19, 2024
fbd925f
Adding env as endpoint
dws4 Jul 30, 2024
f8e650a
Adding Configuration Changes and README for Data Dictionary V2
dws4 Aug 8, 2024
039a25d
task/DataDictionaryFrontend: Frontend Overhaul and Fixing Column Issues
dws4 Sep 6, 2024
ceeba9c
DictionaryFrontend: Adding Collapse Functionality Fix and Copy/Paste
dws4 Sep 23, 2024
ea8ddbf
task/DictionaryFrontend: Editing Rest Controllers and Formats with Lo…
dws4 Oct 3, 2024
2ab1cea
task/DataDictionary: Fixing banner and img issues and separating logic
dsheehan2 Oct 4, 2024
c77263e
task/DataDictionaryFrontend: Finishing Final Touches and Minor Bug Fixes
dsheehan2 Oct 9, 2024
9b05683
task/DictionaryFrontend: Updated to be able to run in DEV mode after …
dsheehan2 Oct 10, 2024
8973a4c
task/DataDictionaryFrontend: Modernizing and Fixing README
dsheehan2 Oct 10, 2024
08784b0
Update service/src/main/frontend/src/functions/formatters.ts
dsheehan2 Oct 24, 2024
7375169
Apply suggestions from code review
dsheehan2 Oct 24, 2024
44b273d
Update service/src/main/frontend/tsconfig.json
dsheehan2 Oct 24, 2024
ffe8e83
Update formatters.ts
dsheehan2 Oct 28, 2024
d553d02
Update README with instructions for pairing Dictionary with the Autho…
dsheehan2 Oct 29, 2024
59fe9cf
Fixing API endpoint mixup
dsheehan2 Oct 30, 2024
8191814
Updating docker compose in README
dsheehan2 Oct 30, 2024
a7e8a25
Fixing variables for more clarity in formatters
dsheehan2 Nov 7, 2024
598abdb
Fixing Merge Issues with README changes
dsheehan2 Nov 7, 2024
9bacc5e
Fixing variable declarations
dsheehan2 Nov 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
target/
logs/
src/main/resources/static
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved

.idea/
*.iml
Expand Down
76 changes: 76 additions & 0 deletions service/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@
<url>https://github.com/NationalSecurityAgency/datawave-dictionary-service</url>
</scm>
<properties>
<node.downloadRoot>https://nodejs.org/dist/</node.downloadRoot>
<node.npmRegistryUrl>https://registry.npmjs.org/</node.npmRegistryUrl>
<node.version>v18.0.0</node.version>
<npm.downloadRoot>https://registry.npmjs.org/npm/-/</npm.downloadRoot>
<npm.version>9.2.0</npm.version>
<start-class>datawave.microservice.dictionary.DictionaryService</start-class>
<version.accumulo>2.1.1</version.accumulo>
<version.curator>5.2.0</version.curator>
Expand Down Expand Up @@ -232,9 +237,80 @@
</repositories>
<build>
<plugins>
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>1.12.1</version>
<executions>
<execution>
<id>install node and npm</id>
<goals>
<goal>install-node-and-npm</goal>
</goals>
<configuration>
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved
<nodeDownloadRoot>${node.downloadRoot}</nodeDownloadRoot>
<npmDownloadRoot>${npm.downloadRoot}</npmDownloadRoot>
<nodeVersion>${node.version}</nodeVersion>
<npmVersion>${npm.version}</npmVersion>
</configuration>
</execution>
<execution>
<id>npm config set registry</id>
<phase>generate-resources</phase>
<goals>
<goal>npm</goal>
</goals>
<configuration>
<arguments>config set registry=${node.npmRegistryUrl}</arguments>
</configuration>
</execution>
<execution>
<id>npm install</id>
<goals>
<goal>npm</goal>
</goals>
<configuration>
<arguments>install</arguments>
</configuration>
</execution>
<execution>
<id>npm run build</id>
<goals>
<goal>npm</goal>
</goals>
<configuration>
<arguments>run build</arguments>
</configuration>
</execution>
</executions>
<configuration>
<workingDirectory>src/main/frontend</workingDirectory>
</configuration>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<executions>
<execution>
<id>copy frontend content</id>
<phase>generate-resources</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>target/classes/static/data/v2</outputDirectory>
<overwrite>true</overwrite>
<resources>
<resource>
<directory>src/main/frontend/dist/spa</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
Expand Down
9 changes: 9 additions & 0 deletions service/src/main/frontend/.editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
root = true

[*]
charset = utf-8
indent_style = space
indent_size = 2
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
7 changes: 7 additions & 0 deletions service/src/main/frontend/.eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/dist
/src-capacitor
/src-cordova
/.quasar
/node_modules
.eslintrc.cjs
/quasar.config.*.temporary.compiled*
94 changes: 94 additions & 0 deletions service/src/main/frontend/.eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
module.exports = {
// https://eslint.org/docs/user-guide/configuring#configuration-cascading-and-hierarchy
// This option interrupts the configuration hierarchy at this file
// Remove this if you have an higher level ESLint config file (it usually happens into a monorepos)
root: true,

// https://eslint.vuejs.org/user-guide/#how-to-use-a-custom-parser
// Must use parserOptions instead of "parser" to allow vue-eslint-parser to keep working
// `parser: 'vue-eslint-parser'` is already included with any 'plugin:vue/**' config and should be omitted
parserOptions: {
parser: require.resolve('@typescript-eslint/parser'),
extraFileExtensions: ['.vue'],
},

env: {
browser: true,
es2021: true,
node: true,
},

// Rules order is important, please avoid shuffling them
extends: [
// Base ESLint recommended rules
// 'eslint:recommended',

// https://github.com/typescript-eslint/typescript-eslint/tree/master/packages/eslint-plugin#usage
// ESLint typescript rules
'plugin:@typescript-eslint/recommended',

// Uncomment any of the lines below to choose desired strictness,
// but leave only one uncommented!
// See https://eslint.vuejs.org/rules/#available-rules
'plugin:vue/vue3-essential', // Priority A: Essential (Error Prevention)
// 'plugin:vue/vue3-strongly-recommended', // Priority B: Strongly Recommended (Improving Readability)
// 'plugin:vue/vue3-recommended', // Priority C: Recommended (Minimizing Arbitrary Choices and Cognitive Overhead)

// https://github.com/prettier/eslint-config-prettier#installation
// usage with Prettier, provided by 'eslint-config-prettier'.
'prettier',
],

plugins: [
// required to apply rules which need type information
'@typescript-eslint',

// https://eslint.vuejs.org/user-guide/#why-doesn-t-it-work-on-vue-files
// required to lint *.vue files
'vue',

// https://github.com/typescript-eslint/typescript-eslint/issues/389#issuecomment-509292674
// Prettier has not been included as plugin to avoid performance impact
// add it as an extension for your IDE
],

globals: {
ga: 'readonly', // Google Analytics
cordova: 'readonly',
__statics: 'readonly',
__QUASAR_SSR__: 'readonly',
__QUASAR_SSR_SERVER__: 'readonly',
__QUASAR_SSR_CLIENT__: 'readonly',
__QUASAR_SSR_PWA__: 'readonly',
process: 'readonly',
Capacitor: 'readonly',
chrome: 'readonly',
},

// add your custom rules here
rules: {
// Allows different null assertions and types to be utilized
'@typescript-eslint/no-unused-vars': 'off',
'@typescript-eslint/no-non-null-assertion': 'off',

'prefer-promise-reject-errors': 'off',

quotes: ['warn', 'single', { avoidEscape: true }],

// this rule, if on, would require explicit return type on the `render` function
'@typescript-eslint/explicit-function-return-type': 'off',

// allows to define an 'any' component
'@typescript-eslint/no-explicit-any': 'off',

// in plain CommonJS modules, you can't use `import foo = require('foo')` to pass this rule, so it has to be disabled
'@typescript-eslint/no-var-requires': 'off',

// The core 'no-unused-vars' rules (in the eslint:recommended ruleset)
// does not work with type definitions
'no-unused-vars': 'off',

// allow debugger during development only
'no-debugger': process.env.NODE_ENV === 'production' ? 'error' : 'off',
},
};
41 changes: 41 additions & 0 deletions service/src/main/frontend/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
.DS_Store
.thumbs.db
node
node_modules

# Quasar core related directories
.quasar
/dist
/quasar.config.*.temporary.compiled*

# Cordova related directories and files
/src-cordova/node_modules
/src-cordova/platforms
/src-cordova/plugins
/src-cordova/www

# Capacitor related directories and files
/src-capacitor/www
/src-capacitor/node_modules

# BEX related directories and files
/src-bex/www
/src-bex/js/core

# Log files
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Editor directories and files
.idea
*.suo
*.ntvs*
*.njsproj
*.sln

# .env file (This is optional, uncomment below)
# .env

# Exclude package config files
/package-lock.json
4 changes: 4 additions & 0 deletions service/src/main/frontend/.npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
shamefully-hoist=true
strict-peer-dependencies=false
resolution-mode=highest
registry=https://registry.npmjs.org/
4 changes: 4 additions & 0 deletions service/src/main/frontend/.prettierrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"singleQuote": true,
"semi": true
}
87 changes: 87 additions & 0 deletions service/src/main/frontend/README.md
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# <p style="margin-bottom: 0px">Datawave Data Dictionary Version 2 <img src="./public/favicon.ico" style="width: 2em; height: 2em; vertical-align: middle;"></p>

### Project Overview
Data Dictionary Version 2 is a new microservice designed to enhance the original Data Dictionary. It leverages an updated tech stack to incorporate features that facilitate data analysis and provide quick query functions. Below are detailed instructions on how to install, run, and configure this project. This README will guide you through the process of starting and running the project.

### Tech Stack Overview
The tech stack for this project includes **Vue.js (v3.0)** with **Vite** for building the frontend and enabling hot module replacement. **Quasar (v2.8)** is used for constructing the main table and handling data insertion. The backend is powered by **Java 11** with **Spring Boot** for serving data through RESTful methods, while **Node.js (v16+)** and **NPM** manage packages and dependencies. Lastly, **Apache Accumulo** provides data storage for visualization.

## Running the Application
This section provides the installation details and outlines the steps to run Data Dictionary V2 in production-ready mode. Additionally, it includes instructions for making rapid frontend changes using Quasar's Dev Mode with Vue. While both modes can be operated independently, it is strongly recommended to conduct thorough testing in production mode, as it closely simulates the deployment environment.

### Instructions for Production-Ready Container Mode
1. To run the application in production mode, first ensure Java 11 and Maven is installed. Next, clone the [Datawave Repository](https://github.com/NationalSecurityAgency/datawave) into your preferred directory, then navigate into it by running the following command(s):

```
git clone [email protected]:NationalSecurityAgency/datawave.git
cd /path/to/your/directory/datawave
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved
```

2. Make sure you are on the integration branch for Datawave, and on the main branch for each of the microservice submodules, as the Dictionary service depends on the Configuration and Authorization microservices. Then run the following to build the project. Ensure that the cache is disabled in the maven command:

```
# First checkout the main branch for each submodule:

git submodule foreach 'git checkout main || :'

# Then build the datawave project and produce images for each microservice:

mvn -Pcompose -Dmicroservice-docker -Dquickstart-docker -Ddeploy -Dtar -Ddist clean install -DskipTests -Dmaven.build.cache.enabled=false

# Spin up all the containers for the microservice after full datawave build:

cd /datawave/docker
docker compose --profile dictionary up -d
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved
```

3. Now if you go to https://localhost:8643/dictionary/data/v2/#/ you should be able to see the Data Dictionary Version 2 and if you go to https://localhost:8643/dictionary/data/v1/ you will be able to see the predecessor. If you can't reach it due to client certificate errors please download the [p12 certificate](https://github.com/NationalSecurityAgency/datawave-spring-boot-starter/raw/refs/heads/main/src/main/resources/testUser.p12). Once downloaded, on your browser (this is assuming you are using Chrome but should be similar with other browsers): go to _Privacy and Secrurity_, then select _Security_, then scroll down to _Manage Certificates_, under the _Your Certificates_ tab, click import and select the p12 file you downloaded. The password should be "ChangeIt", and now you are paired with the authorization service. It is highly recommended you change this certificate for yourself, more information to do this can be found at this [README](https://github.com/NationalSecurityAgency/datawave-microservices-root/blob/main/README.md#getting-started) for the Datawave External Services.

##### To Make a Quick Change
1. If you would like to make a quick change either to the backend or frontend of the service, run the following:

```
# First go to the docker directory and take down the containers:

cd /datawave/docker
docker compose down

# Enter only the Dictionary Service Directory and rebuild the jar and image:

cd /datawave/microservices/services/dictionary/

mvn -Pcompose -Dmicroservice-docker -Dquickstart-docker -Ddeploy -Dtar -Ddist clean install -DskipTests -Dmaven.build.cache.enabled=false

# Go back to the docker directory and spin up the containers again.
cd /datawave/docker
docker compose up -d
```
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved
2. Then you should be able to hit this endpoint again: https://localhost:8643/dictionary/data/v2/#/

### Instructions for Quasar Dev Mode (Uses Vite.js)
1. To run this application in DEV mode, first follow the steps above to build and run the containers. This process only needs to be completed once to retrieve the necessary endpoints.
2. Next, in order to run in DEV mode, you will first need to install [Node.js and NPM](https://nodejs.org/en/download/package-manager/). Ensure you installed them by running `npm -v` and `node -v` to check their versions.
3. Then you will need to globally install the the Quasar/CLI package. This can be done using npm by running:

```
npm i -g @quasar/cli
```
4. Once installed, you can make quick changes in Quasar's Dev Mode. Open your code editor (such as VSCode) and ensure you have the official Vue and Vite extensions installed. Then, open the Datawave project directory in your code editor. Navigate to the main repository of the Data Dictionary service in your code editor and in your terminal which is:

```
cd /datawave/microservices/services/dictionary/service/src/main/frontend
```
5. Now, make a change to the `DataDictionary.vue` file (where most of the frontend components are housed) or any TypeScript file, such as logging a message to the terminal or modifying the color of a button. While still in this directory, execute the following command:

```
quasar dev
```
*For more information, please refer to [Quasar's documentation](https://quasar.dev/start/quasar-cli#running-without-the-global-quasar-cli) on running in development mode.*
jsduncan2 marked this conversation as resolved.
Show resolved Hide resolved

## Configuration Information
- `package.json`: This file defines the project's build process and manage its dependencies through NPM. They include crucial information such as project metadata, versioning, icons, and scripts for running the project in Quasar DEV mode. For instance, you will be able to see that this project uses Bootstrap Icons in order to populate some of buttons.

- `quasar.config.js`: This configuration file governs the behavior of Quasar, managing essential plugins and modules for its operation. It also integrates settings for Vite, to allow for DEV mode. Additionally, this file defines routes for specific API endpoints, such as the *banner/* endpoint, along with [other data endpoints](https://github.com/NationalSecurityAgency/datawave-dictionary-service/blob/9b0568347e360f32392d4feee662d2ddf9eacd17/README.md) used to populate rows and columns. Furthermore, it specifies routing configurations to streamline navigation within the application.

*Please note that `package-lock.json` is not required for this project and may be removed.*

***
21 changes: 21 additions & 0 deletions service/src/main/frontend/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<!DOCTYPE html>
<html>
<head>
<title><%= productName %></title>
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved

<meta charset="utf-8">
<meta name="description" content="<%= productDescription %>">
<meta name="format-detection" content="telephone=no">
<meta name="msapplication-tap-highlight" content="no">
<meta name="viewport" content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width<% if (ctx.mode.cordova || ctx.mode.capacitor) { %>, viewport-fit=cover<% } %>">

<link rel="icon" type="image/png" sizes="128x128" href="icons/favicon-128x128.png">
dsheehan2 marked this conversation as resolved.
Show resolved Hide resolved
<link rel="icon" type="image/png" sizes="96x96" href="icons/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="32x32" href="icons/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="icons/favicon-16x16.png">
<link rel="icon" type="image/ico" href="favicon.ico">
</head>
<body>
<!-- quasar:entry-point -->
</body>
</html>
Loading