Skip to content

null_if in column block not working as expected #393

Open
@pskrbasu

Description

@pskrbasu

Tailpipe - v0.3.2
Core plugin - v0.2.2

null_if_data.csv

id,status,value,description
1,active,42,normal value
2,inactive,0,zero value
3,active,-1,negative value
4,active,2,normal value
5,active,999,special value 

config:

format "delimited" "null_if_test" {
  delimiter = ","
}

table "null_if_test" {
  format = format.delimited.null_if_test

  column "tp_timestamp" {
    type = "datetime"
    transform = "now()"
  }

  column "tp_date" {
    type = "date"
    transform = "current_date"
  }

  column "id" {
    type = "integer"
    source = "id"
  }

  column "status" {
    type = "varchar"
    source = "status"
    null_if = "inactive"
  }

  column "value" {
    type = "integer"
    source = "value"
    null_if = 0
  }

  column "description" {
    type = "varchar"
    source = "description"
  }
}

partition "null_if_test" "local" {
  source "file" {
    format = format.delimited.null_if_test
    paths = ["/path/to/custom_logs/"]
    file_layout = "null_if_data.csv"
  }
}

Collect:

tailpipe collect null_if_test --from=2024-05-01

Collecting logs for null_if_test.local from 2024-05-01 

Artifacts:
  Discovered: 1 
  Downloaded: 1 154B
  Extracted:  1

Rows:
  Received: 5
  Enriched: 5
  Saved:    5

Files:
  Compacted: 1 => 1

Completed: 54ms

Query:

tailpipe query
Welcome to Tailpipe v0.3.2
For more information, type .help
> select id, status, value, description from null_if_test order by id
+----+----------+-------+----------------+
| id | status   | value | description    |
+----+----------+-------+----------------+
| 1  | active   | 42    | normal value   |
| 2  | inactive | 0     | zero value     |
| 3  | active   | -1    | negative value |
| 4  | active   | 2     | empty value    |
| 5  | active   | 999   | special value  |
+----+----------+-------+----------------+
> 

expected query output (the 2nd row containing inactive status should return ):

> select id, status, value, description from null_if_test order by id
+----+----------+-------+----------------+
| id | status   | value | description    |
+----+----------+-------+----------------+
| 1  | active   | 42    | normal value   |
| 2  | <null> | <null>     | zero value     |
| 3  | active   | -1    | negative value |
| 4  | active   | 2     | empty value    |
| 5  | active   | 999   | special value  |
+----+----------+-------+----------------+

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions