Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(infrastructure): Add support for testing against both Delivery and Compute services. #562

Merged
merged 1 commit into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
18 changes: 13 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,11 @@ GOMAXPROCS ?= 4

NAME := $(notdir $(shell pwd))

# Test Service ID
FASTLY_TEST_SERVICE_ID ?=
# Test Service IDs
FASTLY_TEST_DELIVERY_SERVICE_ID ?=
DEFAULT_FASTLY_TEST_DELIVERY_SERVICE_ID = kKJb5bOFI47uHeBVluGfX1
FASTLY_TEST_COMPUTE_SERVICE_ID ?=
DEFAULT_FASTLY_TEST_COMPUTE_SERVICE_ID = XsjdElScZGjmfCcTwsYRC1
FASTLY_API_KEY ?=
#
# Enables support for tools such as https://github.com/rakyll/gotest
Expand Down Expand Up @@ -60,10 +63,15 @@ test-full: ## Runs the tests with VCR disabled (i.e., makes external calls).
'${GO} test -timeout=60s -parallel=20 ${GOPKGS} ${TESTARGS}'
.PHONY: test-full

fix-fixtures: ## Updates test fixtures with a specified default service ID.
fix-delivery-fixtures: ## Updates test fixtures with a specified default Delivery service ID.
@echo "==> Updating fixtures"
@$(shell pwd)/scripts/fixFixtures.sh ${FASTLY_TEST_SERVICE_ID}
.PHONY: fix-fixtures
@$(shell pwd)/scripts/fixFixtures.sh ${FASTLY_TEST_DELIVERY_SERVICE_ID} ${DEFAULT_FASTLY_TEST_DELIVERY_SERVICE_ID}
.PHONY: fix-delivery-fixtures

fix-compute-fixtures: ## Updates test fixtures with a specified default Compute service ID.
@echo "==> Updating fixtures"
@$(shell pwd)/scripts/fixFixtures.sh ${FASTLY_TEST_COMPUTE_SERVICE_ID} ${DEFAULT_FASTLY_TEST_COMPUTE_SERVICE_ID}
.PHONY: fix-compute-fixtures

check-imports: ## A check which lists improperly-formatted imports, if they exist.
@$(shell pwd)/scripts/check-imports.sh
Expand Down
59 changes: 45 additions & 14 deletions TESTING.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,73 @@
# Testing

Go Fastly uses [go-vcr](https://github.com/dnaeon/go-vcr) to "record" and "replay" API request fixtures to improve the speed and portability of integration tests. The test suite uses a single test service ID for all test fixtures.
Go Fastly uses [go-vcr](https://github.com/dnaeon/go-vcr) to "record"
and "replay" API request fixtures to improve the speed and portability
of integration tests. The test suite uses a pair of service IDs (one
for Delivery, one for Compute) for its test cases.

Contributors without access to the test service can still update the fixtures but with some additional steps required. Below is an example workflow for updating a set of fixture files (where `...` should be replaced with an appropriate value):
Contributors without access to the test services can still update the
fixtures but with some additional steps required. Below is an example
workflow for updating a set of fixture files (where `...` should be
replaced with an appropriate value):

```sh
# Remove all yaml fixture files from the specified directory.
# Remove all YAML fixture files from the specified directory.
#
rm -r fastly/fixtures/.../*

# Run a subsection of the tests.
# Run a subset of the tests.
# This will cause the deleted fixtures to be recreated.
#
# FASTLY_TEST_SERVICE_ID: should correspond to a real service you control.
# FASTLY_API_KEY: should be a real token associated with the Service you control.
#
# FASTLY_TEST_DELIVERY_SERVICE_ID: should correspond to a real Delivery service you control.
# FASTLY_TEST_COMPUTE_SERVICE_ID: should correspond to a real Compute service you control.
# FASTLY_API_KEY: should be a real token associated with the services you control.
# TESTARGS: allows you to use the -run flag of the 'go test' command.
#
make test FASTLY_TEST_SERVICE_ID="..." FASTLY_API_KEY="..." TESTARGS="-run=..."
#
make test FASTLY_TEST_DELIVERY_SERVICE_ID="..." FASTLY_TEST_COMPUTE_SERVICE_ID="..." FASTLY_API_KEY="..." TESTARGS="-run=..."
```

> **NOTE**: to run the tests with go-vcr disabled, set `VCR_DISABLE=1` (`make test-full` does this).

When adding or updating client code and integration tests, contributors should record a new set of fixtures. Before submitting a pull request with new or updated fixtures, we ask that contributors update them to use the default service ID by running `make fix-fixtures` with `FASTLY_TEST_SERVICE_ID` set to the same value used to run your tests.
When adding or updating client code and integration tests,
contributors should record a new set of fixtures. Before submitting a
pull request with new or updated fixtures, we ask that contributors
update them to use the default service IDs by running two commands:

```sh
make fix-delivery-fixtures FASTLY_TEST_DELIVERY_SERVICE_ID=<your Delivery SID>
```

```sh
make fix-fixtures FASTLY_TEST_SERVICE_ID="..."
make fix-compute-fixtures FASTLY_TEST_COMPUTE_SERVICE_ID=<your Compute SID>
```

### Important Test Tips!

There are two important things external contributors need to do when running the tests:

1. Use a 'temporary' token for running the tests (only if regenerating the token fixtures).
1. Use a 'temporary' token for running the tests (only if regenerating
the token fixtures).
2. Redact sensitive information in your fixtures.

You only need to use a temporary token when regenerating the 'token' fixtures. This is because there is a test to validate the _revoking_ of a token using the [`/tokens/self`](https://developer.fastly.com/reference/api/auth/#revoke-token-current) API endpoint, for which running this test (if there are no existing fixtures) will cause the token you provided at your command-line shell to be revoked/expired. So please don't use a token that's also used by a real/running application! Otherwise you'll discover those application may stop working as you've inadvertently caused your token to be revoked.
You only need to use a temporary token when regenerating the 'token'
fixtures. This is because there is a test to validate the _revocation_
of a token using the
[`/tokens/self`](https://developer.fastly.com/reference/api/auth/#revoke-token-current)
API endpoint, for which running this test (if there are no existing
fixtures) will cause the token you provided at your command-line shell
to be revoked/expired. So please don't use a token that's also used by
a real/running application! Otherwise you'll discover those
application mays stop working as you've inadvertently caused your token
to be revoked.

In general, any time you regenerate fixtures you should be sure to redact any sensitive information served back from the API, but specifically there is a test which _creates_ tokens that needs special attention: when regenerating the token fixtures this will require you to enter your actual account credentials (i.e. username/password) into the `token_test.go` file. You'll want to ensure that once the fixtures are created that you redact those values from both the generated fixture as well as the go test file itself. For example...
In general, any time you regenerate fixtures you should be sure to
redact any sensitive information served back from the API, but
specifically there is a test which _creates_ tokens that needs special
attention: when regenerating the token fixtures this will require you
to enter your actual account credentials (i.e. username/password) into
the `token_test.go` file. You'll want to ensure that once the fixtures
are created that you redact those values from both the generated
fixture as well as the go test file itself. For example...

```go
input := &CreateTokenInput{
Expand Down
18 changes: 9 additions & 9 deletions fastly/acl_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ func TestClient_ACLs(t *testing.T) {

fixtureBase := "acls/"

testVersion := createTestVersion(t, fixtureBase+"version", testServiceID)
testVersion := createTestVersion(t, fixtureBase+"version", testDeliveryServiceID)

// Create
var err error
var a *ACL
record(t, fixtureBase+"create", func(c *Client) {
a, err = c.CreateACL(&CreateACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
Name: ToPointer("test_acl"),
})
Expand All @@ -29,7 +29,7 @@ func TestClient_ACLs(t *testing.T) {
var errExpected error
record(t, fixtureBase+"create_expected_error", func(c *Client) {
_, errExpected = c.CreateACL(&CreateACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
})
})
Expand All @@ -41,13 +41,13 @@ func TestClient_ACLs(t *testing.T) {
defer func() {
record(t, fixtureBase+"cleanup", func(c *Client) {
_ = c.DeleteACL(&DeleteACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
Name: "test_acl",
})

_ = c.DeleteACL(&DeleteACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
Name: "new_test_acl",
})
Expand All @@ -62,7 +62,7 @@ func TestClient_ACLs(t *testing.T) {
var as []*ACL
record(t, fixtureBase+"list", func(c *Client) {
as, err = c.ListACLs(&ListACLsInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
})
})
Expand All @@ -77,7 +77,7 @@ func TestClient_ACLs(t *testing.T) {
var na *ACL
record(t, fixtureBase+"get", func(c *Client) {
na, err = c.GetACL(&GetACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
Name: "test_acl",
})
Expand All @@ -93,7 +93,7 @@ func TestClient_ACLs(t *testing.T) {
var ua *ACL
record(t, fixtureBase+"update", func(c *Client) {
ua, err = c.UpdateACL(&UpdateACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
Name: "test_acl",
NewName: ToPointer("new_test_acl"),
Expand All @@ -113,7 +113,7 @@ func TestClient_ACLs(t *testing.T) {
// Delete
record(t, fixtureBase+"delete", func(c *Client) {
err = c.DeleteACL(&DeleteACLInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *testVersion.Number,
Name: "new_test_acl",
})
Expand Down
12 changes: 6 additions & 6 deletions fastly/alerts_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ func TestClient_FastlyAlerts(t *testing.T) {
IntegrationIDs: []string{},
Metric: ToPointer("status_5xx"),
Name: ToPointer("test name"),
ServiceID: ToPointer(testServiceID),
ServiceID: ToPointer(testDeliveryServiceID),
Source: ToPointer("domains"),
}

Expand Down Expand Up @@ -68,7 +68,7 @@ func TestClient_FastlyAlerts(t *testing.T) {
t.Errorf("bad name: %v", ad.Name)
}

if ad.ServiceID != testServiceID {
if ad.ServiceID != testDeliveryServiceID {
t.Errorf("bad service_id: %v", ad.ServiceID)
}

Expand All @@ -91,7 +91,7 @@ func TestClient_FastlyAlerts(t *testing.T) {
Cursor: ToPointer(""),
Limit: ToPointer(10),
Name: ToPointer(ad.Name),
ServiceID: ToPointer(testServiceID),
ServiceID: ToPointer(testDeliveryServiceID),
Sort: ToPointer("name"),
})
})
Expand Down Expand Up @@ -154,7 +154,7 @@ func TestClient_FastlyAlerts(t *testing.T) {
Cursor: ToPointer(""),
DefinitionID: ToPointer(ad.ID),
Limit: ToPointer(10),
ServiceID: ToPointer(testServiceID),
ServiceID: ToPointer(testDeliveryServiceID),
Sort: ToPointer("-start"),
Status: ToPointer(""),
})
Expand All @@ -181,7 +181,7 @@ func TestClient_FastlyPercentAlerts(t *testing.T) {
IntegrationIDs: []string{},
Metric: ToPointer("status_5xx"),
Name: ToPointer("test name"),
ServiceID: ToPointer(testServiceID),
ServiceID: ToPointer(testDeliveryServiceID),
Source: ToPointer("stats"),
}

Expand Down Expand Up @@ -216,7 +216,7 @@ func TestClient_FastlyPercentAlerts(t *testing.T) {
t.Errorf("bad name: %v", ad.Name)
}

if ad.ServiceID != testServiceID {
if ad.ServiceID != testDeliveryServiceID {
t.Errorf("bad service_id: %v", ad.ServiceID)
}

Expand Down
18 changes: 9 additions & 9 deletions fastly/backend_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ func TestClient_Backends(t *testing.T) {
var b *Backend
record(t, "backends/create", func(c *Client) {
b, err = c.CreateBackend(&CreateBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: ToPointer("test-backend"),
Address: ToPointer("integ-test.go-fastly.com"),
Expand All @@ -36,13 +36,13 @@ func TestClient_Backends(t *testing.T) {
defer func() {
record(t, "backends/cleanup", func(c *Client) {
_ = c.DeleteBackend(&DeleteBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "test-backend",
})

_ = c.DeleteBackend(&DeleteBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "new-test-backend",
})
Expand Down Expand Up @@ -78,7 +78,7 @@ func TestClient_Backends(t *testing.T) {
var bs []*Backend
record(t, "backends/list", func(c *Client) {
bs, err = c.ListBackends(&ListBackendsInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
})
})
Expand All @@ -93,7 +93,7 @@ func TestClient_Backends(t *testing.T) {
var nb *Backend
record(t, "backends/get", func(c *Client) {
nb, err = c.GetBackend(&GetBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "test-backend",
})
Expand Down Expand Up @@ -121,7 +121,7 @@ func TestClient_Backends(t *testing.T) {
var ub *Backend
record(t, "backends/update", func(c *Client) {
ub, err = c.UpdateBackend(&UpdateBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "test-backend",
NewName: ToPointer("new-test-backend"),
Expand Down Expand Up @@ -158,7 +158,7 @@ func TestClient_Backends(t *testing.T) {
// NOTE: The following test validates empty values are NOT sent.
record(t, "backends/update_ignore_empty_values", func(c *Client) {
ub, err = c.UpdateBackend(&UpdateBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "new-test-backend",
})
Expand All @@ -180,7 +180,7 @@ func TestClient_Backends(t *testing.T) {
// and so the JSON unmarshal recognises that empty values are allowed.
record(t, "backends/update_allow_empty_values", func(c *Client) {
ub, err = c.UpdateBackend(&UpdateBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "new-test-backend",
OverrideHost: ToPointer(""),
Expand All @@ -200,7 +200,7 @@ func TestClient_Backends(t *testing.T) {
// Delete
record(t, "backends/delete", func(c *Client) {
err = c.DeleteBackend(&DeleteBackendInput{
ServiceID: testServiceID,
ServiceID: testDeliveryServiceID,
ServiceVersion: *tv.Number,
Name: "new-test-backend",
})
Expand Down
Loading
Loading