[telemetry] godev/devtools: create cloud storage emulator script

5 views
Skip to first unread message

Jamal Carvalho (Gerrit)

unread,
May 25, 2023, 3:09:05 PM5/25/23
to goph...@pubsubhelper.golang.org, Hyang-Ah Hana Kim, Peter Weinberger, golang-co...@googlegroups.com

Attention is currently required from: Hyang-Ah Hana Kim, Peter Weinberger.

View Change

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: comment
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 2
    Gerrit-Owner: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Reviewer: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Comment-Date: Thu, 25 May 2023 19:09:03 +0000
    Gerrit-HasComments: No
    Gerrit-Has-Labels: No

    Hyang-Ah Hana Kim (Gerrit)

    unread,
    May 25, 2023, 9:19:24 PM5/25/23
    to Jamal Carvalho, goph...@pubsubhelper.golang.org, Hyang-Ah Hana Kim, Gopher Robot, Peter Weinberger, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Patch set 2:Code-Review +2

    View Change

    1 comment:

    • File godev/devtools/localstorage.sh:

      • Patch Set #2, Line 39: docker run --rm --name telemetry-gcs -p $port:$port $dir \

        q. does this mean that any tests that use this emulator need network access to fetch the docker image?

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: comment
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 2
    Gerrit-Owner: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Gopher Robot <go...@golang.org>
    Gerrit-Reviewer: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Reviewer: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Jamal Carvalho <ja...@golang.org>
    Gerrit-Comment-Date: Fri, 26 May 2023 01:19:21 +0000
    Gerrit-HasComments: Yes
    Gerrit-Has-Labels: Yes

    Jamal Carvalho (Gerrit)

    unread,
    May 26, 2023, 8:53:51 AM5/26/23
    to goph...@pubsubhelper.golang.org, Hyang-Ah Hana Kim, Gopher Robot, Peter Weinberger, golang-co...@googlegroups.com

    Attention is currently required from: Hyang-Ah Hana Kim, Jamal Carvalho, Peter Weinberger.

    View Change

    1 comment:

    • File godev/devtools/localstorage.sh:

      • q. […]

        Unit tests can use the Go API from the fake-gcs-server module. Use of this script should be limited to manual testing and prototyping a server that interacts with GCS. I will remove the use of docker and add a filesystem root.

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: comment
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 3
    Gerrit-Owner: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Gopher Robot <go...@golang.org>
    Gerrit-Reviewer: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Reviewer: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Jamal Carvalho <ja...@golang.org>
    Gerrit-Attention: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Comment-Date: Fri, 26 May 2023 12:53:48 +0000
    Gerrit-HasComments: Yes
    Gerrit-Has-Labels: No
    Comment-In-Reply-To: Hyang-Ah Hana Kim <hya...@gmail.com>

    Jamal Carvalho (Gerrit)

    unread,
    May 26, 2023, 8:53:51 AM5/26/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Hyang-Ah Hana Kim, Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #3 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 54 insertions(+), 0 deletions(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset

    Jamal Carvalho (Gerrit)

    unread,
    May 26, 2023, 10:32:01 AM5/26/23
    to goph...@pubsubhelper.golang.org, Gopher Robot, Hyang-Ah Hana Kim, Peter Weinberger, golang-co...@googlegroups.com

    Attention is currently required from: Hyang-Ah Hana Kim, Jamal Carvalho, Peter Weinberger.

    View Change

    1 comment:

    • File godev/devtools/localstorage.sh:

      • Unit tests can use the Go API from the fake-gcs-server module. […]

        Done

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: comment
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 4
    Gerrit-Owner: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Gopher Robot <go...@golang.org>
    Gerrit-Reviewer: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Reviewer: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Jamal Carvalho <ja...@golang.org>
    Gerrit-Attention: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Comment-Date: Fri, 26 May 2023 14:31:58 +0000
    Gerrit-HasComments: Yes
    Gerrit-Has-Labels: No
    Comment-In-Reply-To: Jamal Carvalho <ja...@golang.org>

    Hyang-Ah Hana Kim (Gerrit)

    unread,
    May 26, 2023, 11:27:33 AM5/26/23
    to Jamal Carvalho, goph...@pubsubhelper.golang.org, Gopher Robot, Hyang-Ah Hana Kim, Peter Weinberger, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Patch set 4:Code-Review +2

    View Change

    1 comment:

    • File godev/devtools/localstorage.sh:

      • Done

        Thanks!

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: comment
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 4
    Gerrit-Owner: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Gopher Robot <go...@golang.org>
    Gerrit-Reviewer: Hyang-Ah Hana Kim <hya...@gmail.com>
    Gerrit-Reviewer: Jamal Carvalho <ja...@golang.org>
    Gerrit-Reviewer: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Peter Weinberger <p...@google.com>
    Gerrit-Attention: Jamal Carvalho <ja...@golang.org>
    Gerrit-Comment-Date: Fri, 26 May 2023 15:27:30 +0000
    Gerrit-HasComments: Yes
    Gerrit-Has-Labels: Yes

    Jamal Carvalho (Gerrit)

    unread,
    May 27, 2023, 7:14:15 AM5/27/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #5 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 48 insertions(+), 1 deletion(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 5

    Jamal Carvalho (Gerrit)

    unread,
    May 27, 2023, 7:26:18 AM5/27/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #6 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 48 insertions(+), 1 deletion(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 6

    Jamal Carvalho (Gerrit)

    unread,
    May 27, 2023, 7:41:15 AM5/27/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #7 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 48 insertions(+), 1 deletion(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 7

    Jamal Carvalho (Gerrit)

    unread,
    May 27, 2023, 7:58:34 AM5/27/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #8 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 47 insertions(+), 0 deletions(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 8

    Jamal Carvalho (Gerrit)

    unread,
    May 27, 2023, 8:06:50 AM5/27/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #9 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 47 insertions(+), 0 deletions(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 9

    Jamal Carvalho (Gerrit)

    unread,
    May 27, 2023, 10:21:40 AM5/27/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #10 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 47 insertions(+), 0 deletions(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 10

    Jamal Carvalho (Gerrit)

    unread,
    May 30, 2023, 8:04:29 AM5/30/23
    to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

    Attention is currently required from: Jamal Carvalho, Peter Weinberger.

    Jamal Carvalho uploaded patch set #11 to this change.

    View Change

    The following approvals got outdated and were removed: Run-TryBot+1 by Jamal Carvalho, TryBot-Result+1 by Gopher Robot

    The change is no longer submittable: TryBots-Pass is unsatisfied now.

    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 47 insertions(+), 0 deletions(-)

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: newpatchset
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 11

    Jamal Carvalho (Gerrit)

    unread,
    May 30, 2023, 8:34:18 AM5/30/23
    to goph...@pubsubhelper.golang.org, golang-...@googlegroups.com, Gopher Robot, Hyang-Ah Hana Kim, Peter Weinberger, golang-co...@googlegroups.com

    Jamal Carvalho submitted this change.

    View Change



    4 is the latest approved patch-set.
    The change was submitted with unreviewed changes in the following files:

    ```
    The name of the file: godev/.gitignore
    Insertions: 1, Deletions: 1.

    @@ -1,2 +1,2 @@
    node_modules
    -.storage
    +.localstorage
    ```
    ```
    The name of the file: godev/devtools/localstorage.sh
    Insertions: 10, Deletions: 17.

    @@ -25,29 +25,22 @@
    # log.Fatal(err)
    # }
    #
    -# By default, the emulator will use the directory .storage as the filesystem root.
    -# Override this location by passing a directory as an argument. Subdirectories of
    -# the given location will map to storage buckets in the emulator.
    +# By default, the emulator will read and write from godev/.localstorage. Pass a
    +# directory to override that location.
    #
    -# ./devtools/localstorage.sh .gcs-data
    +# ./devtools/localstorage.sh ~/storage

    -version=v1.45.1
    +version=v0.0.0-20230523204811-eccb7d2267b0
    port=8081
    -dir=".storage"
    +dir=.localstorage
    if [ ! -z "$1" ]; then
    - dir=$1
    + dir="$1"
    fi

    -if ! command -v fake-gcs-server &> /dev/null
    +if ! command -v gcsemulator &> /dev/null
    then
    - echo "Command fake-gcs-server could not be found. Installing..."
    - go install github.com/fsouza/fake-gcs-server@$version
    + echo "Command gcsemulator could not be found. Installing..."
    + go install github.com/fullstorydev/emulators/storage/cmd/gcsemulator@$version
    fi

    -echo running GCS emulator at http://localhost:$port
    -
    -fake-gcs-server \
    - -filesystem-root $dir \
    - -scheme http \
    - -port $port \
    - -public-host localhost:$port
    +gcsemulator -port $port -dir $dir
    ```

    Approvals: Hyang-Ah Hana Kim: Looks good to me, approved Jamal Carvalho: Run TryBots Gopher Robot: TryBots succeeded
    godev/devtools: create cloud storage emulator script

    The script starts a GCS emulator for use in manual testing and
    prototyping the services that interact with GCS.

    Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Reviewed-on: https://go-review.googlesource.com/c/telemetry/+/498264
    Run-TryBot: Jamal Carvalho <ja...@golang.org>
    Reviewed-by: Hyang-Ah Hana Kim <hya...@gmail.com>
    TryBot-Result: Gopher Robot <go...@golang.org>

    ---
    M godev/.gitignore
    A godev/devtools/localstorage.sh
    2 files changed, 47 insertions(+), 0 deletions(-)

    
    
    diff --git a/godev/.gitignore b/godev/.gitignore
    index 3c3629e..46770c4 100644
    --- a/godev/.gitignore
    +++ b/godev/.gitignore
    @@ -1 +1,2 @@
    node_modules
    +.localstorage
    \ No newline at end of file
    diff --git a/godev/devtools/localstorage.sh b/godev/devtools/localstorage.sh
    new file mode 100755
    index 0000000..fb21cc1
    --- /dev/null
    +++ b/godev/devtools/localstorage.sh
    @@ -0,0 +1,46 @@
    +#!/usr/bin/env bash
    +
    +# Copyright 2023 The Go Authors. All rights reserved.
    +# Use of this source code is governed by a BSD-style
    +# license that can be found in the LICENSE file.
    +set -e
    +
    +# Script for running a Google Cloud Storage API emulator.
    +#
    +# Connect to the emulator by adding an endpoint option when creating a storage
    +# client.
    +#
    +# client, err := storage.NewClient(
    +# context.Background(),
    +# option.WithEndpoint("http://localhost:8081/storage/v1/"),
    +# )
    +#
    +# Or by setting STORAGE_EMULATOR_HOST=localhost:8081 when running your program.
    +#
    +# STORAGE_EMULATOR_HOST=localhost:8081 go run ./cmd/my-command
    +#
    +# OR
    +#
    +# if err := os.Setenv("STORAGE_EMULATOR_HOST", "localhost:8081"); err != nil {
    +# log.Fatal(err)
    +# }
    +#
    +# By default, the emulator will read and write from godev/.localstorage. Pass a
    +# directory to override that location.
    +#
    +# ./devtools/localstorage.sh ~/storage
    +
    +version=v0.0.0-20230523204811-eccb7d2267b0
    +port=8081
    +dir=.localstorage
    +if [ ! -z "$1" ]; then
    + dir="$1"
    +fi
    +
    +if ! command -v gcsemulator &> /dev/null
    +then
    + echo "Command gcsemulator could not be found. Installing..."
    + go install github.com/fullstorydev/emulators/storage/cmd/gcsemulator@$version
    +fi
    +
    +gcsemulator -port $port -dir $dir

    To view, visit change 498264. To unsubscribe, or for help writing mail filters, visit settings.

    Gerrit-MessageType: merged
    Gerrit-Project: telemetry
    Gerrit-Branch: master
    Gerrit-Change-Id: Ic4abe4f35fc7843ebd9b62b8d5323ba672cd7747
    Gerrit-Change-Number: 498264
    Gerrit-PatchSet: 12
    Reply all
    Reply to author
    Forward
    0 new messages