Eden/value matching

[EdenWuyifan]

@roquelopez Here are the new designs made for value matching annotations.

This pull request introduces several improvements and minor fixes across the backend and frontend codebases. The most significant changes include updates to ontology resource paths to use a new JSON file, UI enhancements for better usability and clarity, and minor code cleanups. Below are the most important changes grouped by theme:

Ontology Resource Path Updates

• Changed references to ontology resource files from gdc_ontology_flat.json to gdc_ontology.json in api/index.py, api/utils.py, and removed redundant path definitions from api/tools/task_tools.py. This ensures consistency and uses the updated ontology data. [1] [2] [3]

Frontend UI Enhancements

• Added a close button to the file upload dialog in fileuploading.tsx for improved user experience.
• Enhanced enum value display in RelevantKnowledgeView.tsx by adding tooltips with definitions and highlighting search matches, improving clarity and interactivity. [1] [2] [3]
• Replaced unused MUI components (Chip, CircularProgress) with Tooltip for cleaner imports.
• Added import for CloseIcon to support the new close button functionality.

Backend Minor Fixes and Cleanups

• Reformatted logging and dictionary assignment code for readability and maintainability in memory.py and matching_task.py. [1] [2]
• Improved docstrings and error logging formatting for task tools in task_tools.py. [1] [2]

Build Script Cleanup

• Removed unused Redis kill scripts from package.json to simplify development and start scripts. [1] [2]

[github-actions]

[black-format] _{reported by reviewdog 🐶}

Suggested change

	f"🧰Tool called: search_explanations with query='{query}', " f"limit={limit}"
	f"🧰Tool called: search_explanations with query='{query}', "
	f"limit={limit}"

[github-actions]

[black-format] _{reported by reviewdog 🐶}

Suggested change

	self.cached_candidates[
	"matchers"
	] = self.weight_updater.update_matchers(
	self.cached_candidates["matchers"]
	self.cached_candidates["matchers"] = (
	self.weight_updater.update_matchers(
	self.cached_candidates["matchers"]
	)

[github-actions]

Coverage report

St.❔	Category	Percentage	Covered / Total
🔴	Statements	58.18%	7408/12733
🟡	Branches	60.7%	485/799
🔴	Functions	40.4%	122/302
🔴	Lines	58.18%	7408/12733

Test suite run success

3 tests passing in 3 suites.

Report generated by 🧪jest coverage report action from 4b7c585

[Copilot]

Pull Request Overview

This PR implements an enhanced value matching system with a new custom table interface for value comparisons. The changes replace the MaterialReactTable component with a custom implementation that provides better interaction features and enum support.

• Replaces MaterialReactTable with custom table implementation for better control and performance
• Adds enum popup functionality with search and tooltips for GDC attributes
• Enhances value editing capabilities with inline editing and filtering options

Reviewed Changes

Copilot reviewed 10 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
package.json	Removes Redis kill scripts for development workflow
app/lib/heatmap/heatmap-helper.tsx	Adds enumDef property to GDCAttribute interface
app/dashboard/components/value-comparisons/value-comparison-table.tsx	Complete rewrite from MaterialReactTable to custom table with enum support
app/dashboard/components/fileuploading.tsx	Adds close button to file upload dialog
app/dashboard/components/explanation/RelevantKnowledgeView.tsx	Adds tooltips for enum values using enumDef
api/utils.py	Updates GDC ontology file path
api/tools/task_tools.py	Code formatting improvements and cleanup
api/matching_task.py	Code formatting improvements
api/langchain/memory.py	Code formatting improvements
api/index.py	Updates GDC ontology file path

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

[nitpick] The import statement is too long and imports many hooks. Consider grouping related imports or splitting into multiple lines for better readability.

Suggested change

	import { useMemo, useContext, useEffect, useState, useCallback, useRef, useLayoutEffect } from "react";
	import {
	useMemo,
	useContext,
	useEffect,
	useState,
	useCallback,
	useRef,
	useLayoutEffect
	} from "react";

[Copilot]

Potential runtime error if rows[0] is undefined. The condition !rows.length is checked above, but there's a race condition where rows could be empty between the check and this line.

[Copilot]

Same potential runtime error as above - accessing rows[0] without ensuring rows is not empty could cause a crash.

[Copilot]

[nitpick] Inline CSS styles should be moved to a separate CSS file or styled components for better maintainability and reusability.

[Copilot]

[nitpick] String literals should use consistent quote style. The component uses double quotes elsewhere but single quotes here.

Suggested change

	anchorOrigin={{ vertical: 'top', horizontal: 'center' }}
	transformOrigin={{ vertical: 'bottom', horizontal: 'center' }}
	transformOrigin={{ vertical: "bottom", horizontal: "center" }}

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
api
index.py	577	245	58%	38–39, 106, 114–118, 123–179, 184–240, 249–287, 305–310, 323–328, 378, 411, 430–456, 471, 492–505, 520, 536, 542, 554, 569–585, 600–606, 618–631, 643–654, 666–679, 692–705, 749, 758–768, 775–803, 832, 842, 848, 866, 903–906, 921, 938–941, 1020–1052, 1074, 1081–1083, 1088–1102, 1117–1127, 1137
matching_task.py	660	243	63%	83–85, 113–159, 167–224, 280, 313, 334–336, 395, 399–414, 426, 437–500, 534, 667, 710, 750, 814–825, 866, 886–899, 904–957, 961–963, 977–1015, 1025, 1028–1035, 1069, 1084–1093, 1101–1116, 1119, 1122, 1126, 1133, 1142, 1158–1161, 1185–1201, 1229, 1240, 1267–1273, 1281, 1287, 1340, 1375–1384, 1431, 1444, 1473, 1481–1483, 1500, 1503–1507, 1510
session_manager.py	50	11	78%	25–26, 30–31, 39, 47–50, 53, 56
utils.py	315	121	62%	33, 47, 58–60, 63–65, 88–89, 102–119, 123–124, 128–129, 161–162, 187–189, 206, 209–218, 225–232, 242, 271, 281–296, 300–302, 314–354, 360–366, 407–409, 419–421, 425–427, 435–439, 447–450, 516–520, 536–540
api/candidate_quadrants
candidate_quadrants.py	90	22	76%	130, 136–152, 156–162, 165–168, 172
api/clusterer
column_encoder.py	43	13	70%	51, 55, 83, 92, 101–102, 111, 115, 125–126, 135–136, 144
embedding_clusterer.py	74	19	74%	59–76, 83, 102–110
utils.py	129	99	23%	14–16, 20–22, 27–29, 34–35, 40–46, 50–60, 64–66, 125, 129, 140–172, 177–190, 207–208, 236–307
api/langchain
agent.py	196	45	77%	85, 99–104, 106–109, 273, 281, 284–287, 296, 378–381, 400–403, 430–461, 466–475, 506–511
memory.py	284	80	72%	147–173, 295, 304, 312–313, 378, 380, 382, 545, 565–566, 585, 664–675, 678–689, 692–706, 709–723, 726–739, 767–768, 816–817, 825–826
rag.py	38	38	0%	1–69
api/langgraph
langgraph.py	201	201	0%	2–778
api/matcher
bdikit.py	41	2	95%	52, 113
difflib.py	23	2	91%	34, 48
magneto.py	25	1	96%	28
rapidfuzz_value.py	93	9	90%	49, 99, 106, 126, 148–151, 164–166
utils.py	10	2	80%	26, 49
valentine.py	40	2	95%	68, 79
api/tools
candidate_tools.py	65	15	77%	114–116, 136–138, 184–186, 227–229, 270–272
online_research_tools.py	181	67	63%	31, 61, 76, 85–87, 116–132, 143–159, 170–186, 197–213, 241, 247, 257, 285–287, 302, 330–332, 351, 390–392, 411, 421, 455–457, 461–484
query_tools.py	78	15	81%	132, 160–167, 189–196, 218–219
source_scraper.py	10	1	90%	40
task_tools.py	52	14	73%	21, 50–54, 82, 96–101, 117–123
TOTAL	3425	1267	63%

Tests	Skipped	Failures	Errors	Time
60	0 💤	1 ❌	0 🔥	1m 55s ⏱️