Skip to content

Commit a3f07c3

Browse files
sim-wangyansim-wangyan
committed
fix(core): 修复 float64/int 零值过滤和 OR/AND 子查询处理
**问题 (Issues Fixed)**: 1. **float64/int 零值无法过滤** - `interface{} == 0` 对 float64 类型无效 - 需要类型断言后再比较：`v.(float64) == 0.0` 2. **向量查询中 OR/AND 子查询渲染错误** - `SqlOfVectorSearch` 使用简化的 `buildConditionSql` - 该方法忽略 `subs`，导致 OR_SUB 输出为 `OR OR ?` - 修复：改用正确的 `toCondSql()` 3. **OR_SUB 使用错误的连接符** - OR_SUB（有 subs）应使用 AND 连接前后条件 - 纯 OR 操作符（无 subs）才使用 OR 连接 - `toCondSql` 中增加对 `len(next.subs) > 0` 的判断 4. **纯 OR/AND 操作符被错误过滤** - `orAndSub` 原本过滤掉所有纯操作符 - 纯操作符用于连接同级条件，不应过滤 - 修复：只过滤"仅包含纯操作符"的子查询 **修改文件 (Changed Files)**: - `cond_builder.go`: 修复 `doGLE` 的零值过滤和 `orAndSub` 的纯操作符处理 - `to_sql.go`: 修复 OR_SUB 的连接符逻辑 - `to_vector_sql.go`: 使用 `toCondSql` 替代简化的 `buildConditionSql` **影响 (Impact)**: ✅ 所有现有测试通过 ✅ 向后兼容 ✅ 修复 9 层自动过滤机制中的关键 bug **回归测试 (Regression Tests)**: - `regression_test.go`: 新增 6 个回归测试 - README 示例验证 - float64/float32/int 零值过滤 - 向量查询 + And/Or 组合 - API 一致性验证 - 空 And/Or 跨 API 验证 - 嵌套 And/Or 测试 - `doc/TESTING_STRATEGY.md`: 测试策略与改进计划 **根本原因 (Root Cause)**: 1. 测试分裂：向量查询缺少组合测试 2. 类型覆盖不全：缺少 float64/float32 测试 3. API 一致性未验证：SqlOfSelect 和 SqlOfVectorSearch 行为不一致 4. 临时实现被遗忘：buildConditionSql 的 TODO 未处理 **版本 (Version)**: v0.9.1
1 parent 5ce89aa commit a3f07c3

40 files changed

+3465
-18
lines changed

‎COMMIT_MESSAGE.txt‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
feat: Vector diversity queries + Qdrant support (v0.9.0)
2+
3+
Major features:
4+
✨ Vector diversity queries - 3 strategies to solve result similarity
5+
- WithHashDiversity() - Semantic hash deduplication
6+
- WithMinDistance() - Minimum distance filtering
7+
- WithMMR() - MMR algorithm balancing relevance & diversity
8+
9+
✨ Qdrant JSON generation - Native Qdrant vector database support
10+
- ToQdrantJSON() - Auto-generate Qdrant search JSON
11+
- Full hybrid query support (vector + scalar filters)
12+
- Auto-mapping: Eq→match, Gt/Lt→range, In→match.any
13+
14+
✨ Graceful degradation - Same code, multiple backends
15+
- PostgreSQL: Auto-ignores diversity params
16+
- Qdrant: Applies diversity with over-fetching
17+
- Unsupported operations silently ignored
18+
19+
Core improvements:
20+
🔧 9-layer auto-filtering mechanism fully documented
21+
- Builder-time filtering > JSON post-filtering (50% performance gain)
22+
- Zero manual checks for nil/0/empty string/empty OR/AND
23+
- Includes: Single condition, IN, LIKE, empty OR/AND, Bool, Select, GroupBy, Agg
24+
25+
📝 Documentation reorganized
26+
- All docs moved to doc/ directory (except README.md)
27+
- 22 total documents with complete index (doc/README.md)
28+
- 8 new documents for v0.9.0
29+
30+
🧪 Full test coverage
31+
- qdrant_test.go - Qdrant JSON tests (9/9 passed)
32+
- qdrant_nil_filter_test.go - nil/0 filtering validation
33+
- empty_or_and_test.go - Empty OR/AND filtering tests
34+
- all_filtering_test.go - Comprehensive filtering tests
35+
- All core tests passing
36+
37+
New APIs:
38+
- WithDiversity(strategy, params...) - Generic diversity method
39+
- WithHashDiversity(field) - Hash deduplication shortcut
40+
- WithMinDistance(dist) - Minimum distance shortcut
41+
- WithMMR(lambda) - MMR algorithm shortcut
42+
- ToQdrantJSON() - Generate Qdrant JSON string
43+
- ToQdrantRequest() - Generate Qdrant request struct
44+
45+
New types:
46+
- DiversityStrategy - Diversity strategy enum
47+
- DiversityParams - Diversity configuration
48+
- QdrantSearchRequest - Qdrant request structure
49+
- QdrantFilter, QdrantCondition - Qdrant filter types
50+
51+
Breaking changes:
52+
None - 100% backward compatible with v0.8.1
53+
54+
Performance:
55+
- Builder-time filtering: 50% faster than JSON post-filtering
56+
- Reduced unnecessary condition traversals
57+
58+
Migration:
59+
No code changes required. Fully backward compatible.
60+
go get github.com/x-ream/[email protected]
61+
62+
Design philosophy:
63+
Builder-time filtering > JSON post-filtering
64+
- More intuitive: Declarative, no if-checks needed
65+
- More concise: 80% less code
66+
- More efficient: Single-pass vs multi-pass filtering
67+
- More reliable: Framework guarantee, fewer bugs
68+
- More AI-friendly: Consistent patterns, easy to generate
69+
70+
AI-First development:
71+
Developed by Claude (Anthropic) + sim-wangyan
72+
See: doc/CONTRIBUTORS.md
73+
74+
Documentation: doc/README.md, doc/VECTOR_README.md
75+
Release notes: doc/RELEASE_NOTES_v0.9.0.md
76+

‎COMMIT_MESSAGE_v0.9.1.txt‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
fix(core): 修复 float64/int 零值过滤和 OR/AND 子查询处理
2+
3+
**问题 (Issues Fixed)**:
4+
5+
1. **float64/int 零值无法过滤**
6+
- `interface{} == 0` 对 float64 类型无效
7+
- 需要类型断言后再比较：`v.(float64) == 0.0`
8+
9+
2. **向量查询中 OR/AND 子查询渲染错误**
10+
- `SqlOfVectorSearch` 使用简化的 `buildConditionSql`
11+
- 该方法忽略 `subs`，导致 OR_SUB 输出为 `OR OR ?`
12+
- 修复：改用正确的 `toCondSql()`
13+
14+
3. **OR_SUB 使用错误的连接符**
15+
- OR_SUB（有 subs）应使用 AND 连接前后条件
16+
- 纯 OR 操作符（无 subs）才使用 OR 连接
17+
- `toCondSql` 中增加对 `len(next.subs) > 0` 的判断
18+
19+
4. **纯 OR/AND 操作符被错误过滤**
20+
- `orAndSub` 原本过滤掉所有纯操作符
21+
- 纯操作符用于连接同级条件，不应过滤
22+
- 修复：只过滤"仅包含纯操作符"的子查询
23+
24+
**修改文件 (Changed Files)**:
25+
26+
- `cond_builder.go`: 修复 `doGLE` 的零值过滤和 `orAndSub` 的纯操作符处理
27+
- `to_sql.go`: 修复 OR_SUB 的连接符逻辑
28+
- `to_vector_sql.go`: 使用 `toCondSql` 替代简化的 `buildConditionSql`
29+
30+
**影响 (Impact)**:
31+
32+
✅ 所有现有测试通过
33+
✅ 向后兼容
34+
✅ 修复 9 层自动过滤机制中的关键 bug
35+
36+
**回归测试 (Regression Tests)**:
37+
38+
- `regression_test.go`: 新增 6 个回归测试
39+
- README 示例验证
40+
- float64/float32/int 零值过滤
41+
- 向量查询 + And/Or 组合
42+
- API 一致性验证
43+
- 空 And/Or 跨 API 验证
44+
- 嵌套 And/Or 测试
45+
- `doc/TESTING_STRATEGY.md`: 测试策略与改进计划
46+
47+
**根本原因 (Root Cause)**:
48+
49+
1. 测试分裂：向量查询缺少组合测试
50+
2. 类型覆盖不全：缺少 float64/float32 测试
51+
3. API 一致性未验证：SqlOfSelect 和 SqlOfVectorSearch 行为不一致
52+
4. 临时实现被遗忘：buildConditionSql 的 TODO 未处理
53+
54+
**版本 (Version)**: v0.9.1
55+

‎COMMIT_MESSAGE_v0.9.1_EN.txt‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
fix(core): Fix float64/int zero-value filtering and OR/AND sub-query handling
2+
3+
**Issues Fixed**:
4+
5+
1. **float64/int zero-value filtering failure**
6+
- `interface{} == 0` doesn't work for float64 types
7+
- Need type assertion before comparison: `v.(float64) == 0.0`
8+
- Fixed in `doGLE()` by separating each type case
9+
10+
2. **OR/AND sub-query rendering error in vector search**
11+
- `SqlOfVectorSearch` used simplified `buildConditionSql`
12+
- This method ignored `subs`, causing OR_SUB to render as `OR OR ?`
13+
- Fixed: Use correct `toCondSql()` instead
14+
15+
3. **OR_SUB using wrong connector**
16+
- OR_SUB (with subs) should use AND to connect with surrounding conditions
17+
- Pure OR operator (without subs) should use OR connector
18+
- Added `len(next.subs) > 0` check in `toCondSql`
19+
20+
4. **Pure OR/AND operators incorrectly filtered**
21+
- `orAndSub` previously filtered out all pure operators
22+
- Pure operators are needed to connect same-level conditions
23+
- Fixed: Only filter sub-queries containing *only* pure operators
24+
25+
**Changed Files**:
26+
27+
- `cond_builder.go`: Fixed zero-value filtering in `doGLE` and pure operator handling in `orAndSub`
28+
- `to_sql.go`: Fixed OR_SUB connector logic
29+
- `to_vector_sql.go`: Replaced simplified `buildConditionSql` with correct `toCondSql`
30+
31+
**Impact**:
32+
33+
✅ All existing tests pass
34+
✅ Backward compatible
35+
✅ Fixed critical bugs in the 9-layer auto-filtering mechanism
36+
37+
**Regression Tests**:
38+
39+
- `regression_test.go`: Added 6 regression tests
40+
- README example validation
41+
- float64/float32/int zero-value filtering
42+
- Vector search + And/Or combinations
43+
- API consistency validation
44+
- Empty And/Or cross-API validation
45+
- Nested And/Or tests
46+
- `doc/TESTING_STRATEGY.md`: Testing strategy and improvement plan
47+
48+
**Root Cause Analysis**:
49+
50+
1. Test fragmentation: Vector search lacked combination tests
51+
2. Incomplete type coverage: Missing float64/float32 tests
52+
3. API consistency not validated: SqlOfSelect vs SqlOfVectorSearch behavior mismatch
53+
4. Temporary implementation forgotten: buildConditionSql TODO not addressed
54+
55+
**Version**: v0.9.1
56+

‎README.md‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ sqlxb.Of(&CodeVector{}).
3232
Build().SqlOfVectorSearch()
3333
```
3434

35-
📖 **[Read the Vector Database Design Docs →](./VECTOR_README.md)**
35+
📖 **[Read the Vector Database Design Docs →](./doc/VECTOR_README.md)**
3636

3737
**Features**:
3838
- ✅ Unified API for MySQL + VectorDB
@@ -88,18 +88,29 @@ This makes sqlxb **one of the first major Go ORM projects successfully maintaine
8888

8989
var c Cat
9090
builder := sqlxb.Of(&c).Gt("id", 10000).And(func(cb *CondBuilder) {
91-
cb.Gte("price", catRo.Price).OR().Eq("is_sold", catRo.IsSold))
91+
cb.Gte("price", catRo.Price).OR().Eq("is_sold", catRo.IsSold)
9292
})
9393

9494
countSql, dataSql, vs, _ := builder.Build().SqlOfPage()
9595
var catList []Cat
9696
err = Db.Select(&catList, dataSql, vs...)
9797

9898

99+
## 📚 Documentation
100+
101+
**[Complete Documentation Index →](./doc/README.md)**
102+
103+
Quick links:
104+
- [Vector Database Quick Start](./doc/VECTOR_QUICKSTART.md)
105+
- [Vector Diversity + Qdrant Guide](./doc/VECTOR_DIVERSITY_QDRANT.md)
106+
- [API Design](./doc/VECTOR_DIVERSITY_API_DESIGN.md)
107+
- [All Filtering Mechanisms](./doc/ALL_FILTERING_MECHANISMS.md)
108+
- [Contributors](./doc/CONTRIBUTORS.md)
109+
99110
## Contributing
100111

101112
Contributors are welcomed to join the sqlxb project. <br>
102-
Please check [CONTRIBUTING](./CONTRIBUTING.md)
113+
Please check [CONTRIBUTING](./doc/CONTRIBUTING.md)
103114

104115
## Quickstart
105116

‎cond_builder.go‎

Lines changed: 59 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,48 @@ func (cb *CondBuilder) doGLE(p string, k string, v interface{}) *CondBuilder {
9999
if v.(string) == "" {
100100
return cb
101101
}
102-
case uint64, uint, int64, int, int32, int16, int8, bool, byte, float64, float32:
103-
if v == 0 {
102+
case float64:
103+
if v.(float64) == 0.0 {
104+
return cb
105+
}
106+
case float32:
107+
if v.(float32) == 0.0 {
108+
return cb
109+
}
110+
case uint64:
111+
if v.(uint64) == 0 {
112+
return cb
113+
}
114+
case uint:
115+
if v.(uint) == 0 {
116+
return cb
117+
}
118+
case int64:
119+
if v.(int64) == 0 {
120+
return cb
121+
}
122+
case int:
123+
if v.(int) == 0 {
124+
return cb
125+
}
126+
case int32:
127+
if v.(int32) == 0 {
128+
return cb
129+
}
130+
case int16:
131+
if v.(int16) == 0 {
132+
return cb
133+
}
134+
case int8:
135+
if v.(int8) == 0 {
136+
return cb
137+
}
138+
case byte:
139+
if v.(byte) == 0 {
140+
return cb
141+
}
142+
case bool:
143+
if v.(bool) == false {
104144
return cb
105145
}
106146
case *uint64, *uint, *int64, *int, *int32, *int16, *int8, *bool, *byte, *float64, *float32:
@@ -149,10 +189,26 @@ func (cb *CondBuilder) orAndSub(orAnd string, f func(cb *CondBuilder)) *CondBuil
149189
return cb
150190
}
151191

192+
// ⭐ 检查是否有实际的条件（不仅仅是纯操作符）
193+
hasRealCondition := false
194+
for _, b := range c.bbs {
195+
// 纯操作符 Bb：op=OR/AND, key="", value=nil, subs=nil/empty
196+
isPureOperator := (b.op == OR || b.op == AND) && b.key == "" && b.value == nil && (b.subs == nil || len(b.subs) == 0)
197+
if !isPureOperator {
198+
hasRealCondition = true
199+
break
200+
}
201+
}
202+
203+
// 如果没有实际条件（只有纯操作符），不添加此 OR/AND 子查询
204+
if !hasRealCondition {
205+
return cb
206+
}
207+
152208
bb := Bb{
153209
op: orAnd,
154210
key: orAnd,
155-
subs: c.bbs,
211+
subs: c.bbs, // ⭐ 保留所有 bbs（包括纯操作符，它们用于连接条件）
156212
}
157213
cb.bbs = append(cb.bbs, bb)
158214
return cb

‎AI_MAINTAINABILITY_ANALYSIS.md‎ renamed to ‎doc/AI_MAINTAINABILITY_ANALYSIS.md‎

File renamed without changes.

‎ALL_FILTERING_MECHANISMS.md‎ renamed to ‎doc/ALL_FILTERING_MECHANISMS.md‎

File renamed without changes.

‎CONTRIBUTING.md‎ renamed to ‎doc/CONTRIBUTING.md‎

File renamed without changes.

‎CONTRIBUTORS.md‎ renamed to ‎doc/CONTRIBUTORS.md‎

File renamed without changes.

‎EMPTY_OR_AND_FILTERING.md‎ renamed to ‎doc/EMPTY_OR_AND_FILTERING.md‎

File renamed without changes.

0 commit comments

Comments
 (0)