在GitLab CI中对Cloud Datastore进行测试

2 年 ago

科, 雅

3 minutes

太长没仔细看

Cloud Datastore Emulator を GitLab CI の services として起動しておけばテストできる

事件的起源

如果有以下的程式，試著使用 httpmock 方法將對 Cloud Datastore 的請求進行模擬並撰寫測試程式碼，但在建立 datastore.Client 時出現錯誤。

main.go

package main

import (
    "context"
    "log"
    "os"

    // golang.org
    "golang.org/x/oauth2/google"

    // google.golang.org
    "google.golang.org/api/option"

    // cloud.google.com
    "cloud.google.com/go/datastore"
)

func main() {
    ctx := context.Background()

    // 1. Get google application credentials.
    var credOption option.ClientOption
    credentials, err := google.FindDefaultCredentials(ctx)
        if err != nil {
        log.Fatal(err)
    }
    if credentials.JSON != nil {
        credOption = option.WithCredentialsJSON(credentials.JSON)
        } else {
                credOption = option.WithTokenSource(credentials.TokenSource)
        }

    // 2. Put value.
    err = PutValue("xxxxxxxxxxx", "example", credOption)
    if err != nil {
        log.Fatal(err)
    }
}

type Item struct{
    Key *datastore.Key `datastore:"__key__"`
    Value string       `datastore:"value"`
}

func PutValue(nameKey, value string, options ...option.ClientOption) error {
    ctx := context.Background()

    // 1. Create client.
    client, err := datastore.NewClient(ctx, os.Getenv("DATASTORE_PROJECT_ID"), options...)
    if err != nil {
        return err
    }
    defer client.Close()

    // 2. Put item.
    _, err = client.Put(
        ctx,
        datastore.NameKey("Entity", nameKey, nil),
        &Item{
            Value: value,
        },
    )
    if err != nil {
        return err
    }

    return nil
}

main_test.go

package main

import (
    "net/http"
    "testing"

    // github.com
    "github.com/jarcoal/httpmock"
    "github.com/stretchr/testify/assert"

    // google.golang.org
    "google.golang.org/api/option"
)

func TestDatastore(t *testing.T) {
    credOption := option.WithCredentialsJSON([]byte("hoge"))

    t.Run("PutValue", func(t *testing.T) {
        mockHTTPClient := &http.Client{}
        httpmock.ActivateNonDefault(mockHTTPClient)
        defer httpmock.Reset()

        err := PutValue(
            "xxxxxxxxxxx",
            "example",
            credOption,
            option.WithHTTPClient(mockHTTPClient),
        )
        assert.Nil(t, err)
    })
}

test

$ go test -v ./...
--- FAIL: TestDatastore (0.00s)
    --- FAIL: TestDatastore/PutValue (0.00s)
        main_test.go:29: 
                Error Trace:    main_test.go:29
                Error:          Expected nil, but got: &errors.errorString{s:"dialing: WithHTTPClient is incompatible with gRPC dial options"}
                Test:           TestDatastore/PutValue
FAIL
FAIL    command-line-arguments  0.036s

看了一下来源

https://github.com/googleapis/google-cloud-go/blob/master/datastore/datastore.go#L114
https://github.com/googleapis/google-api-go-client/blob/master/internal/settings.go#L86
https://github.com/googleapis/google-api-go-client/blob/master/transport/grpc/dial.go#L68

要测试Cloud Datastore

Cloud Datastore Emulator を使う

只需要一种中文选项，请改写以下内容：

1. 不再使用 httpmock 在 main_test.go 中。

package main

import (
    "testing"

    // github.com
    "github.com/stretchr/testify/assert"

    // google.golang.org
    "google.golang.org/api/option"
)

func TestDatastore(t *testing.T) {
    credOption := option.WithCredentialsJSON([]byte("hoge"))

    t.Run("PutValue", func(t *testing.T) {
        err := PutValue(
            "xxxxxxxxxxx",
            "example",
            credOption,
        )
        assert.Nil(t, err)
    })
}

2. 启动模拟器

$ docker run -d -it --rm --name cloud-datastore-emulator --entrypoint gcloud google/cloud-sdk:latest beta emulators datastore start --project=eg-example-01 --host-port=0.0.0.0:8081

3. 去测试

$ export DATASTORE_PROJECT_ID=eg-example-01
$ export DATASTORE_EMULATOR_HOST=datastore:8081

$ go test -v ./...
=== RUN   TestDatastore
=== RUN   TestDatastore/PutValue
--- PASS: TestDatastore (0.08s)
    --- PASS: TestDatastore/PutValue (0.08s)
PASS
ok      command-line-arguments  0.158s

将GitLab CI转化为CI/CD平台。

.gitlab-ci.yml

image: docker:stable

stages:
  - testing

variables:
  GIT_SUBMODULE_STRATEGY: recursive
  GO_VERSION: "1.12"

Test golang:
  stage: testing

  image: golang:${GO_VERSION}

  variables:
    DOCKERIZE_VERSION: v0.6.1
    DATASTORE_EMULATOR_HOST: datastore:8081
    DATASTORE_PROJECT_ID: eg-example-01

  services:
    - name: google/cloud-sdk:latest
      alias: datastore
      entrypoint: ["gcloud", "beta", "emulators", "datastore"]
      command: ["start", "--project", "eg-example-01", "--host-port", "0.0.0.0:8081"]

  before_script:
    - curl -sfL https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz | tar -C /usr/local/bin -xzv
    - dockerize -wait tcp://${DATASTORE_EMULATOR_HOST} -timeout 1m

  script:
    - go test -v ./...

当在本地进行测试时，可以指定 –host-port=localhost:8081，但是这样无法从容器外部进行通信，所以改为 –host-port=0.0.0.0:8081。

将DATASTORE_PROJECT_ID和DATASTORE_PROJECT_ID添加到环境变量中。

DATASTORE_PROJECT_ID应该设置为您在services中指定的别名作为主机名。

为了确保模拟器正常启动后再进行测试，使用dockerize进行等待。

参考了https://github.com/gcpug/nouhau/issues/8。

之后只需要进行测试。