Skip to content

Using LocalStore unexpected folder popping up in _delta_log #53

Open
@mario-imperato

Description

@mario-imperato

I've taken the examples/local_example_test.go and modified in the attempt to create a delta table and adding some stuff in it. I attach the full code of the sample that is somewhat derived from an HelloWord example in the java standalone lib.
Not sure if I'm missing something. At first sight I get the result but in the _delta_log it shows up an unexpected folder of the form: tests/golden/snapshot-data4/_delta_log/file:/__0x2f____0x2f__Users/marioa.imperato/projects/gits/csimplestring/delta-go/tests/golden/snapshot-data4/_delta_log__0x2f__ (see attached image).

Screenshot 2024-09-05 at 11 30 54

What I'm missing?
Thank you
Mario

Sample code:

package examples

import (
	"fmt"
	"github.com/csimplestring/delta-go/action"
	"github.com/csimplestring/delta-go/iter"
	"github.com/csimplestring/delta-go/op"
	"github.com/csimplestring/delta-go/types"
	"log"
	"path/filepath"
	"testing"
	"time"

	delta "github.com/csimplestring/delta-go"
)

const (
	engineInfo = "local"
)

func TestLocalCreateExample(t *testing.T) {
	path, err := filepath.Abs("../tests/golden/snapshot-data4")
	if err != nil {
		log.Fatal(err)
	}

	path = "file://" + path + "/"

	config := delta.Config{
		StoreType: "file",
	}

	table, err := delta.ForTable(path, config, &delta.SystemClock{})
	if err != nil {
		log.Fatal(err)
	}

	schema := &types.StructType{}
	schema = schema.Add(types.NewStructField("foo", &types.IntegerType{}, false))
	schema = schema.Add(types.NewStructField("bar", &types.IntegerType{}, false))
	schema = schema.Add(types.NewStructField("zip", &types.StringType{}, false))

	metadata := action.DefaultMetadata()
	metadata.SchemaString, err = types.ToJSON(schema)
	if err != nil {
		log.Fatal(err)
	}

	op := op.Operation{
		Name: op.WRITE,
	}

	for i := 0; i < 15; i++ {
		txn, err := table.StartTransaction()
		if err != nil {
			log.Fatal(err)
		}

		if i == 0 {
			err = txn.UpdateMetadata(metadata)
			if err != nil {
				log.Fatal(err)
			}
		}

		now := time.Now()

		addFile := action.AddFile{
			Path:             fmt.Sprintf("%d", i),
			DataChange:       true,
			PartitionValues:  nil,
			Size:             100,
			ModificationTime: now.UnixMilli(),
			Stats:            "",
			Tags:             map[string]string{"someTagKey": "someTagVal"},
		}

		actions := []action.Action{&addFile}
		res, err := txn.Commit(iter.FromSlice[action.Action](actions), &op, engineInfo)
		if err != nil {
			log.Fatal(err)
		}
		t.Log(res)
	}

	s, err := table.Snapshot()
	if err != nil {
		log.Fatal(err)
	}

	version := s.Version()
	log.Println(version)

	files, err := s.AllFiles()
	if err != nil {
		log.Fatal(err)
	}
	for _, f := range files {
		log.Println(f.Path)
	}

	m, err := s.Metadata()
	if err != nil {
		log.Fatal(err)
	}

	schema2, err := m.Schema()
	if err != nil {
		log.Fatal(err)
	}

	for _, f := range schema2.GetFields() {
		log.Println(f)
	}
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions