how to recompile .proto in envoy

592 views
Skip to first unread message

Gary Tang

unread,
Jul 29, 2020, 9:06:13 PM7/29/20
to envoy-dev

Hi,


I made a change to the proto file api/envoy/config/core/v3/address.proto


but bazel build did not pick up the change and thus build failed.


Looks like I need to recompile .proto to produce newer address.pb.h  and address.pb.cc


Anyone can show me how to do/recompile it in envoy?


The following is the build error:


% bazel build -s --verbose_failures --sandbox_debug source/common/network/address_lib



src/main/tools/process-tools.cc:118: sigaction(32, &sa, nullptr) failed

src/main/tools/process-tools.cc:118: sigaction(33, &sa, nullptr) failed

In file included from bazel-out/k8-fastbuild/bin/source/common/singleton/_virtual_includes/threadsafe_singleton/common/singleton/threadsafe_singleton.h:5:0,

                 from bazel-out/k8-fastbuild/bin/source/common/api/_virtual_includes/os_sys_calls_lib/common/api/os_sys_calls_impl.h:7,

                 from source/common/network/address_impl.cc:50:

source/common/network/address_impl.cc: In member function 'Envoy::Network::IoHandlePtr Envoy::Network::Address::InstanceBase::socketFromSocketType(uint8_t, Envoy::Network::Address::SocketType) const':

source/common/network/address_impl.cc:335:70: error: 'FP' is not a member of 'envoy::config::core::v3::SocketAddress'

       if (likely(provider == envoy::config::core::v3::SocketAddress::FP)) {

                                                                      ^

bazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/assert_lib/common/common/assert.h:152:42: note: in definition of macro 'likely'

 #define likely(expr) __builtin_expect(!!(expr), 1)

                                          ^~~~

src/main/tools/linux-sandbox-pid1.cc:437: waitpid returned 2

src/main/tools/linux-sandbox-pid1.cc:457: child exited with code 1

src/main/tools/linux-sandbox.cc:204: child exited normally with exitcode 1

Target //source/common/network:address_lib failed to build

INFO: Elapsed time: 12.934s, Critical Path: 8.52s

INFO: 0 processes.

FAILED: Build did NOT complete successfully

FAILED: Build did NOT complete successfully


Thanks


Gary

Lizan Zhou

unread,
Jul 29, 2020, 9:10:27 PM7/29/20
to Gary Tang, envoy-dev
You need run `./tools/proto_format/proto_format.sh fix` to sync proto files to api shadow

--
You received this message because you are subscribed to the Google Groups "envoy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/638bba29-5af0-48a5-875f-78d7880e4f8eo%40googlegroups.com.

Gary Tang

unread,
Jul 29, 2020, 9:32:07 PM7/29/20
to envoy-dev
Lizan,

Thank you for your quick help.

I run the command, but the following error was seen.
    "FileNotFoundError: [Errno 2] No such file or directory: 'clang-format': 'clang-format'

Is there anything I missed here?

Note: There is no clang install on my machine. I use g++. clang is not necessary for the recompile, right?

The following is the details of the error.

% ./tools/proto_format/proto_format.sh fix

.....

FileNotFoundError: [Errno 2] No such file or directory: 'clang-format': 'clang-format'

Traceback (most recent call last):

  File "/home/tang/do-porting/envoy-1.14.2/envoy/bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/protoxform/protoprint.py", line 611, in <module>

    dst_path.write_bytes(traverse.TraverseFile(file_proto, ProtoFormatVisitor()))

  File "/home/tang/do-porting/envoy-1.14.2/envoy/bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/api_proto_plugin/traverse.py", line 89, in TraverseFile

    return visitor.VisitFile(file_proto, package_type_context, services, msgs, enums)

  File "/home/tang/do-porting/envoy-1.14.2/envoy/bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/protoxform/protoprint.py", line 599, in VisitFile

    return ClangFormat(header + formatted_services + formatted_enums + formatted_msgs)

  File "/home/tang/do-porting/envoy-1.14.2/envoy/bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/protoxform/protoprint.py", line 92, in ClangFormat

    stdout=subprocess.PIPE).stdout

  File "/usr/lib/python3.6/subprocess.py", line 423, in run

    with Popen(*popenargs, **kwargs) as process:

  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__

    restore_signals, start_new_session)

  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child

    raise child_exception_type(errno_num, err_msg, err_filename)

FileNotFoundError: [Errno 2] No such file or directory: 'clang-format': 'clang-format'

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/type/matcher/v3/number.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/extensions/common/dynamic_forward_proxy/v3/dns_cache.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/service/discovery/v2/rtds.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/filter/http/grpc_web/v2/grpc_web.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/service/listener/v3/lds.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/trace/v3/xray.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/admin/v4alpha/certs.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/api/v2/route/route.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/extensions/filters/network/ratelimit/v3/rate_limit.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/cluster/v3/outlier_detection.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/grpc_credential/v2alpha/file_based_metadata.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/api/v2/core/grpc_method_list.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/extensions/transport_sockets/tap/v4alpha/tap.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/extensions/filters/http/dynamic_forward_proxy/v3/dynamic_forward_proxy.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/type/percent.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/filter/http/transcoder/v2/transcoder.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/type/v3/token_bucket.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/grpc_credential/v3/aws_iam.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/wasm/v2alpha/wasm.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/filter/http/aws_lambda/v2alpha/aws_lambda.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/extensions/filters/listener/tls_inspector/v3/tls_inspector.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/resource_monitor/fixed_heap/v2alpha/fixed_heap.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/admin/v4alpha/server_info.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/api/v2/listener.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/extensions/filters/http/lua/v3/lua.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/core/v3/grpc_service.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/config/filter/network/local_rate_limit/v2alpha/local_rate_limit.proto

ProtoPrint /tmp/tmpdvqf0qz2/b/envoy/admin/v2alpha/clusters.proto

multiprocessing.pool.RemoteTraceback:

"""

Traceback (most recent call last):

  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker

    result = (True, func(*args, **kwds))

  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar

    return list(map(*args))

  File "./tools/proto_format/proto_sync.py", line 161, in SyncProtoFile

    ProtoPrint(src, dst)

  File "./tools/proto_format/proto_sync.py", line 124, in ProtoPrint

    './bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/type_whisperer/api_type_db.pb_text'

  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output

    **kwargs).stdout

  File "/usr/lib/python3.6/subprocess.py", line 438, in run

    output=stdout, stderr=stderr)

subprocess.CalledProcessError: Command '['bazel-bin/tools/protoxform/protoprint', 'bazel-bin/external/envoy_api_canonical/envoy/extensions/transport_sockets/tap/v3/pkg/envoy/extensions/transport_sockets/tap/v3/tap.proto.next_major_version_candidate.proto', '/tmp/tmpdvqf0qz2/b/envoy/extensions/transport_sockets/tap/v4alpha/tap.proto', './bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/type_whisperer/api_type_db.pb_text']' returned non-zero exit status 1.

"""


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "./tools/proto_format/proto_sync.py", line 401, in <module>

    Sync(args.api_root, args.mode, args.labels, False)

  File "./tools/proto_format/proto_sync.py", line 344, in Sync

    pkg_deps = p.map(SyncProtoFile, dst_src_paths.items())

  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map

    return self._map_async(func, iterable, mapstar, chunksize).get()

  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get

    raise self._value

subprocess.CalledProcessError: Command '['bazel-bin/tools/protoxform/protoprint', 'bazel-bin/external/envoy_api_canonical/envoy/extensions/transport_sockets/tap/v3/pkg/envoy/extensions/transport_sockets/tap/v3/tap.proto.next_major_version_candidate.proto', '/tmp/tmpdvqf0qz2/b/envoy/extensions/transport_sockets/tap/v4alpha/tap.proto', './bazel-bin/tools/protoxform/protoprint.runfiles/envoy/tools/type_whisperer/api_type_db.pb_text']' returned non-zero exit status 1.

To unsubscribe from this group and stop receiving emails from it, send an email to envo...@googlegroups.com.

Gary Tang

unread,
Jul 29, 2020, 11:00:43 PM7/29/20
to envoy-dev
I installed clang-format.

However, the problem remains after "./tools/proto_format/proto_format.sh fix" was run.

Did I still miss anything?

Thanks,

Gary

Lizan Zhou

unread,
Jul 29, 2020, 11:10:48 PM7/29/20
to Gary Tang, envoy-dev
Do you have clang-format (without version suffix) in your PATH? If not set it to CLANG_FORMAT env var.

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/331518f9-1329-4943-b4d3-1db896f44de4o%40googlegroups.com.

George Hu

unread,
Jul 30, 2020, 12:14:42 AM7/30/20
to envoy-dev
Hello Everyone,

I would like to ask for help on how to build a body replace filter in Lua filter or C++ filter. Is there anything I can reference?  Thanks for your help.


Following is a similar feature in openresty. 

replace_filter

syntax: replace_filter <regex> <replace>

syntax: replace_filter <regex> <replace> <options>

default: no

context: http, server, location, location if

phase: output body filter


On Wed, Jul 29, 2020 at 6:06 PM Gary Tang <gcta...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "envoy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/638bba29-5af0-48a5-875f-78d7880e4f8eo%40googlegroups.com.

Gary Tang

unread,
Jul 30, 2020, 12:26:01 AM7/30/20
to envoy-dev

I installed clang-format-10, made a symbolic link to it in /usr/bin, then set env CLANG_FORMAT. Next './tools/proto_format/proto_format.sh fix' was run without error being reported. Then I did bazel build, but the same error occurred.


It appears the address.pb.h/cc still have not picked up the change. 


Is there any step I missed?


The following is the details:


u18-porting:/usr/bin ❯❯ which clang-format

/usr/bin/clang-format


u18-porting:/usr/bin ❯❯ clang-format --version

clang-format version 10.0.0-4ubuntu1~18.04.1 


u18-porting:/home/tang/do-porting/envoy-1.14.2/envoy ❯❯ export CLANG_FORMAT=/usr/bin/clang-format


u18-porting:/home/tang/do-porting/envoy-1.14.2/envoy ❯❯ env | g CLANG

CLANG_FORMAT=/usr/bin/clang-format


u18-porting:/home/tang/do-porting/envoy-1.14.2/envoy ❯❯ ./tools/proto_format/proto_format.sh fix


u18-porting:/home/tang/do-porting/envoy-1.14.2/envoy ❯❯ bazel build source/common/network/address_lib

WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:

/home/tang/do-porting/envoy-1.14.2/envoy/tools/bazel.rc

INFO: Analyzed target //source/common/network:address_lib (0 packages loaded, 7187 targets configured).

INFO: Found 1 target...

ERROR: /home/tang/do-porting/envoy-1.14.2/envoy/source/common/network/BUILD:11:1: C++ compilation of rule '//source/common/network:address_lib' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 125 argument(s) skipped)


Use --sandbox_debug to see verbose messages from the sandbox

In file included from bazel-out/k8-fastbuild/bin/source/common/singleton/_virtual_includes/threadsafe_singleton/common/singleton/threadsafe_singleton.h:5:0,

                 from bazel-out/k8-fastbuild/bin/source/common/api/_virtual_includes/os_sys_calls_lib/common/api/os_sys_calls_impl.h:7,

                 from source/common/network/address_impl.cc:50:

source/common/network/address_impl.cc: In member function 'Envoy::Network::IoHandlePtr Envoy::Network::Address::InstanceBase::socketFromSocketType(uint8_t, Envoy::Network::Address::SocketType) const':

source/common/network/address_impl.cc:335:70: error: 'FP' is not a member of 'envoy::config::core::v3::SocketAddress'

       if (likely(provider == envoy::config::core::v3::SocketAddress::FP)) {

                                                                      ^

bazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/assert_lib/common/common/assert.h:152:42: note: in definition of macro 'likely'

 #define likely(expr) __builtin_expect(!!(expr), 1)

                                          ^~~~

Target //source/common/network:address_lib failed to build

Use --verbose_failures to see the command lines of failed build steps.

INFO: Elapsed time: 25.480s, Critical Path: 23.71s

INFO: 0 processes.

FAILED: Build did NOT complete successfully



Lizan Zhou

unread,
Jul 30, 2020, 12:32:38 AM7/30/20
to Gary Tang, envoy-dev
It looks related to your change then. Unless you push your branch somewhere I can't help further.

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/7b284ac1-3429-465f-8a49-3ddfa7afc262o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages