Initial object versioning support - retention of old generations if v…

…ersion is enabled, and getObject by generation (#60)

* Object supporting updated/created/deleted date, as first step to support versions

* Dockerfile reproducing with stages what drone is doing the CI, so we can reproduce locally. fmt-ed the files to pass the tests

* rebase conflict issue - back to main golang image to run tests

* rebase conflict issue - back to main golang image to run tests

* attr related tests refactor

* creation and modification dates should be generated and returned if not provided as inputs. Generation now also part of the output, following the same policy

* some IaC state/config files we dont want to commit or embed in docker images

* this allows us to create a bucket, for some extra manual tests

* fmt, so docker build now passes

* unifing gitignore. renaming some functions

* storage interface we use in the backend now considering a bucket not just as an string, but an object that can have extra props. Introduced objectVersioning property in that layer, but still not being used

* memory backend storing versionEnabled property as part of a type-struct that will keep the bucket objects and attributes, not just the name as we used to have. Some locks reviewed, as we can benefit from read locking in some functions. While createBucket used to always succeed, we have introduced a failure scenario: bucket already there and with different props than the new one being requested

* versionEnabled bucket prop consumed by fakestorage package, exposed also to the bucket http endpoint for the server model, but only available and persisted for the memory engine

* replacing a test that relies on the order of a listing by one that finds an occurrence in the output

* WIP - developing tests for new  getObject functions that accept specfic generation lookups, so we can implement multiple versions retention and verify that works

* consolidatin backend object CRUD in one, looping for versioned and non versioned buckets

* Introducing in bucket in memory an archive of objects as a second list, that lives in parallel to the active one. bucket in memory centralizing primitives to deal with two lists, considering if there is versioning, while the external functions consume them and deal with locking.

* memory backends doing specific generation lookups, consdiering also the archived object list. fs backend refuses to work with specific generations

* adding removal test in the backend reveals a bug. Fixed.. need to work to refactor/extend tests, also at fakestorage level

* backend tests refactor. Now time to move to fakestorage and consume the versioning support there

* dont see the point of this infra as code - not needed for the development in the end

* that was pointing to IaC we have removed

* minor test related refactors

* reader against generation test, that surfaced a different handler needed to be modified to retrieve specific generations in that scenario

* basic versioning and delete use cases mapped to fakestorage tests too, that again, surfaced some minor gaps. This starts to be potentially shareable

* almost dummy change to retry build - node tests looks flaky. Minor doc update

* if an object gets replaced by a new version, the archived one should have an according deletion date

* bucket listing response was not considering versioning property

* revisting logger usage - removing debugging stuff

* minor fixes after rebase from latest master. Not checking acls in backend tests, as we currently do in master

* this test duplicates too much the previous (versioning with multiple gens not affecting the normal behavior). We will consolidate and improve with the generations listing support

* refactoring listing tests to reduce duplication and cover extra cases - after create on top of versioning or overwritting files should behave the same. That also prepares the ground for listing with versioning

* generations in gcs are microseconds, so aligning the implementation

* Update Dockerfile

version bump

Co-Authored-By: francisco souza <[email protected]>

* pr feedback - promoting a CreateBucketWithOpts and keeping the old as it was, but deprecated

* Update internal/backend/backend_test.go

As suggested, we should mark test helpers to facilitate tracking where the original test is actually failing

Co-Authored-By: francisco souza <[email protected]>

* pr feedback - marking test helpers rather than passing descriptions to track where tets are failing. Keeping a few ones that are relevant as t.Log()

* Update internal/backend/fs.go

pr feedback - errors.New rather than fmt.Errorf

Co-Authored-By: francisco souza <[email protected]>

* Update internal/backend/fs.go

pr feedback - errors.New replacing fmt.Errorf

Co-Authored-By: francisco souza <[email protected]>

* Update fakestorage/response.go

pr feedback - no naked response

Co-Authored-By: francisco souza <[email protected]>

* Update fakestorage/response.go

pr feedback - no naked response

Co-Authored-By: francisco souza <[email protected]>

* PR feedback

* saving time and space reusing mod download from the tester layer, and we dont want to keep golangci-lint cache, as docker is invalidating the whole layer on changes, so its never reused but takes space

.dockerignore

@@ -3,3 +3,4 @@ README.md
 examples
 LICENSE
 .git
+ci/

Dockerfile

@@ -2,15 +2,31 @@
 # Use of this source code is governed by a BSD-style
 # license that can be found in the LICENSE file.
 
-FROM golang:1.13.5-alpine AS builder
-
+FROM golang:1.13.5 AS tester
 WORKDIR /code
-ENV CGO_ENABLED=0
 ADD go.mod go.sum ./
 RUN go mod download
 ADD . ./
+RUN go test -race -vet all -mod readonly ./...
+
+FROM golangci/golangci-lint AS linter
+WORKDIR /code
+COPY --from=tester /go/pkg /go/pkg
+COPY --from=tester /code .
+RUN golangci-lint run --enable-all \
+    -D errcheck -D lll -D dupl -D gochecknoglobals -D unparam \
+    --deadline 5m \
+    ./... \
+    && rm -rf /root/.cache
+
+FROM golang:1.13.5-alpine AS builder
+WORKDIR /code
+ENV CGO_ENABLED=0
+COPY --from=tester /go/pkg /go/pkg
+COPY --from=tester /code .
 RUN go build -o fake-gcs-server
 
 FROM alpine:3.10.3
 COPY --from=builder /code/fake-gcs-server /bin/fake-gcs-server
+RUN /bin/fake-gcs-server -h
 ENTRYPOINT ["/bin/fake-gcs-server"]

fakestorage/bucket.go

@@ -15,8 +15,29 @@ import (
 // require the bucket name will recognize this bucket.
 //
 // If the bucket already exists, this method does nothing.
+//
+// Deprecated: use CreateBucketWithOpts()
 func (s *Server) CreateBucket(name string) {
-	err := s.backend.CreateBucket(name)
+	err := s.backend.CreateBucket(name, false)
+	if err != nil {
+		panic(err)
+	}
+}
+
+// CreateBucketOpts defines the properties of a bucket you can create
+// with CreateBucketWithOpts()
+type CreateBucketOpts struct {
+	Name              string
+	VersioningEnabled bool
+}
+
+// CreateBucketWithOpts creates a bucket inside the server, so any API calls that
+// require the bucket name will recognize this bucket. Use CreateBucketOpts to
+// customize the options for this bucket
+//
+// If the bucket already exists, this method does nothing but panics if props differs
+func (s *Server) CreateBucketWithOpts(opts CreateBucketOpts) {
+	err := s.backend.CreateBucket(opts.Name, opts.VersioningEnabled)
 	if err != nil {
 		panic(err)
 	}
@@ -25,49 +46,55 @@ func (s *Server) CreateBucket(name string) {
 // createBucketByPost handles a POST request to create a bucket
 func (s *Server) createBucketByPost(w http.ResponseWriter, r *http.Request) {
 	// Minimal version of Bucket from google.golang.org/api/storage/v1
+
 	var data struct {
-		Name string
+		Name       string            `json:"name,omitempty"`
+		Versioning *bucketVersioning `json:"versioning,omitempty"`
 	}
 
-	// Read the bucket name from the request body JSON
+	// Read the bucket props from the request body JSON
 	decoder := json.NewDecoder(r.Body)
 	if err := decoder.Decode(&data); err != nil {
 		http.Error(w, err.Error(), http.StatusBadRequest)
 		return
 	}
 	name := data.Name
-
+	versioning := false
+	if data.Versioning != nil {
+		versioning = data.Versioning.Enabled
+	}
 	// Create the named bucket
-	if err := s.backend.CreateBucket(name); err != nil {
+	if err := s.backend.CreateBucket(name, versioning); err != nil {
 		http.Error(w, err.Error(), http.StatusInternalServerError)
 		return
 	}
 
 	// Return the created bucket:
-	resp := newBucketResponse(name)
+	resp := newBucketResponse(name, versioning)
 	json.NewEncoder(w).Encode(resp)
 }
 
 func (s *Server) listBuckets(w http.ResponseWriter, r *http.Request) {
-	bucketNames, err := s.backend.ListBuckets()
+	buckets, err := s.backend.ListBuckets()
 	if err != nil {
 		http.Error(w, err.Error(), http.StatusInternalServerError)
 		return
 	}
-	resp := newListBucketsResponse(bucketNames)
+	resp := newListBucketsResponse(buckets)
 	json.NewEncoder(w).Encode(resp)
 }
 
 func (s *Server) getBucket(w http.ResponseWriter, r *http.Request) {
 	bucketName := mux.Vars(r)["bucketName"]
 	encoder := json.NewEncoder(w)
-	if err := s.backend.GetBucket(bucketName); err != nil {
+	bucket, err := s.backend.GetBucket(bucketName)
+	if err != nil {
 		w.WriteHeader(http.StatusNotFound)
 		err := newErrorResponse(http.StatusNotFound, "Not found", nil)
 		encoder.Encode(err)
 		return
 	}
-	resp := newBucketResponse(bucketName)
+	resp := newBucketResponse(bucket.Name, bucket.VersioningEnabled)
 	w.WriteHeader(http.StatusOK)
 	encoder.Encode(resp)
 }

fakestorage/bucket_test.go

@@ -8,9 +8,9 @@ import (
 	"context"
 	"io/ioutil"
 	"os"
-	"reflect"
 	"testing"
 
+	"cloud.google.com/go/storage"
 	"google.golang.org/api/iterator"
 )
 
@@ -32,40 +32,60 @@ func TestServerClientBucketAttrs(t *testing.T) {
 		if attrs.Name != expectedName {
 			t.Errorf("wrong bucket name returned\nwant %q\ngot  %q", expectedName, attrs.Name)
 		}
+		if attrs.VersioningEnabled != false {
+			t.Errorf("wrong bucket props for %q\nexpecting no versioning by default, got it enabled", expectedName)
+		}
 	})
 }
 
 func TestServerClientBucketAttrsAfterCreateBucket(t *testing.T) {
-	runServersTest(t, nil, func(t *testing.T, server *Server) {
-		const bucketName = "best-bucket-ever"
-		server.CreateBucket(bucketName)
-		client := server.Client()
-		attrs, err := client.Bucket(bucketName).Attrs(context.Background())
-		if err != nil {
-			t.Fatal(err)
-		}
-		if attrs.Name != bucketName {
-			t.Errorf("wrong bucket name returned\nwant %q\ngot  %q", bucketName, attrs.Name)
-		}
-	})
+	for _, versioningEnabled := range []bool{true, false} {
+		versioningEnabled := versioningEnabled
+		runServersTest(t, nil, func(t *testing.T, server *Server) {
+			const bucketName = "best-bucket-ever"
+			server.CreateBucketWithOpts(CreateBucketOpts{Name: bucketName, VersioningEnabled: versioningEnabled})
+			client := server.Client()
+			attrs, err := client.Bucket(bucketName).Attrs(context.Background())
+			if err != nil {
+				t.Fatal(err)
+			}
+			if attrs.Name != bucketName {
+				t.Errorf("wrong bucket name returned\nwant %q\ngot  %q", bucketName, attrs.Name)
+			}
+			if attrs.VersioningEnabled != versioningEnabled {
+				t.Errorf("wrong bucket props for %q:\nwant versioningEnabled: %t\ngot versioningEnabled: %t", bucketName, versioningEnabled, attrs.VersioningEnabled)
+			}
+		})
+	}
 }
 
 func TestServerClientBucketAttrsAfterCreateBucketByPost(t *testing.T) {
-	runServersTest(t, nil, func(t *testing.T, server *Server) {
-		const bucketName = "post-bucket"
-		client := server.Client()
-		bucket := client.Bucket(bucketName)
-		if err := bucket.Create(context.Background(), "whatever", nil); err != nil {
-			t.Fatal(err)
-		}
-		attrs, err := client.Bucket(bucketName).Attrs(context.Background())
-		if err != nil {
-			t.Fatal(err)
-		}
-		if attrs.Name != bucketName {
-			t.Errorf("wrong bucket name returned\nwant %q\ngot  %q", bucketName, attrs.Name)
-		}
-	})
+	for _, versioningEnabled := range []bool{true, false} {
+		versioningEnabled := versioningEnabled
+		runServersTest(t, nil, func(t *testing.T, server *Server) {
+			const bucketName = "post-bucket"
+			client := server.Client()
+			bucket := client.Bucket(bucketName)
+
+			bucketAttrs := storage.BucketAttrs{
+				VersioningEnabled: versioningEnabled,
+			}
+			if err := bucket.Create(context.Background(), "whatever", &bucketAttrs); err != nil {
+				t.Fatal(err)
+			}
+			attrs, err := client.Bucket(bucketName).Attrs(context.Background())
+			if err != nil {
+				t.Fatal(err)
+			}
+			if attrs.Name != bucketName {
+				t.Errorf("wrong bucket name returned\nwant %q\ngot  %q", bucketName, attrs.Name)
+			}
+
+			if attrs.VersioningEnabled != bucketAttrs.VersioningEnabled {
+				t.Errorf("wrong bucket props for %q:\nwant versioningEnabled: %t\ngot versioningEnabled: %t", bucketName, bucketAttrs.VersioningEnabled, attrs.VersioningEnabled)
+			}
+		})
+	}
 }
 
 func TestServerClientBucketAttrsNotFound(t *testing.T) {
@@ -91,18 +111,35 @@ func TestServerClientListBuckets(t *testing.T) {
 
 	runServersTest(t, objs, func(t *testing.T, server *Server) {
 		client := server.Client()
+		const versionedBucketName = "post-bucket-with-versioning"
+		versionedBucketAttrs := storage.BucketAttrs{
+			VersioningEnabled: true,
+		}
+		if err := client.Bucket(versionedBucketName).Create(context.Background(), "whatever", &versionedBucketAttrs); err != nil {
+			t.Fatal(err)
+		}
 		it := client.Buckets(context.Background(), "whatever")
-		var returnedNames []string
+		expectedBuckets := map[string]bool{
+			"other-bucket": false, "some-bucket": false, versionedBucketName: true}
 		b, err := it.Next()
+		numberOfBuckets := 0
 		for ; err == nil; b, err = it.Next() {
-			returnedNames = append(returnedNames, b.Name)
+			numberOfBuckets++
+			versioning, found := expectedBuckets[b.Name]
+			if !found {
+				t.Errorf("unexpected bucket found\nname %s", b.Name)
+				continue
+			}
+			if versioning != b.VersioningEnabled {
+				t.Errorf("unexpected versioning value for %s\nwant %t\ngot  %t", b.Name, versioning, b.VersioningEnabled)
+			}
 		}
 		if err != iterator.Done {
 			t.Fatal(err)
 		}
-		expectedNames := []string{"other-bucket", "some-bucket"}
-		if !reflect.DeepEqual(returnedNames, expectedNames) {
-			t.Errorf("wrong names returned\nwant %#v\ngot  %#v", expectedNames, returnedNames)
+
+		if len(expectedBuckets) != numberOfBuckets {
+			t.Errorf("wrong number of buckets returned\nwant %d\ngot  %d", len(expectedBuckets), numberOfBuckets)
 		}
 	})
 }
@@ -139,7 +176,7 @@ func TestServerClientListObjects(t *testing.T) {
 			objAttrs, err := it.Next()
 			for ; err == nil; objAttrs, err = it.Next() {
 				seenFiles[objAttrs.Name] = struct{}{}
-				t.Logf("Seen file %s", objAttrs.Name)
+				t.Logf("seen file %s", objAttrs.Name)
 			}
 			if len(objects) != len(seenFiles) {
 				t.Errorf("wrong number of files\nwant %d\ngot %d", len(objects), len(seenFiles))