how can remove the log field in gcp log viewer when log is captured by fluent-plugin-google-cloud

[Shawn Yang]

Hi, our service deploy in another cloud platform, and we transfer the log via fluent-plugin-google-cloud plugin. every work well but when i query the log, i found there are a duplicated field named log.  I want remove the that field. is there mistake about our configuration?

the detail configuration is at here:

1. our service output the log in console as a json like this:

{

    "date": "2021-10-13 08:35:10.083+0000",

    "logger": "com.airwallex.rest.core.api.TransactionNotificationServiceImpl",

    "logging.googleapis.com/labels": {

        "appName": "apptransactionservice",

        "profile": "staging",

        "region": "hk",

        "requestId": "unknown",

        "traceId": "5e3f8178-e496-45e5-80a2-3a0d5d2f461f",

        "version": "release-4.35.0"

    },

    "mdc": {

        "x-awx-account-id": "unknown",

        "x-awx-trace-request-host": "unknown",

        "x-awx-trace-request-service": "unknown",

        "x-awx-trace-request-version": "unknown"

    },

    "message": "accountID:8ea8409f-6e55-49be-acba-84b450902f47 ccy:ILS doesn't have settlementId, use default.",

    "severity": "INFO",

    "thread": "pool-5-thread-4"

}

2. we config fluent like this:

cp-fluentd:

  environment: prod

  externalSecrets:

    type: gcpSecretsManager

    projectId: fx-prod-99340f15

    name: application_default_credentials.json

    key: airlog-prod-stackdriver-log-collector

    isBinary: true

  configMaps:

    fluent.conf: |-

      @include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"

      @include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"

      @include kubernetes.conf

      @include filter.conf

      @include general.conf

      <match kubernetes.**>

        @type record_reformer

        enable_ruby true

        <record>

          # Extract local_resource_id from tag for 'k8s_container' monitored

          # resource. The format is:

          # 'k8s_container.<namespace_name>.<pod_name>.<container_name>'.

          "logging.googleapis.com/local_resource_id" ${"k8s_container.#{tag_suffix[4].rpartition('.')[0].split('_')[1]}.#{tag_suffix[4].rpartition('.')[0].split('_')[0]}.#{tag_suffix[4].rpartition('.')[0].split('_')[2].rpartition('-')[0]}"}

          # Rename the field 'log' to a more generic field 'message'. This way the

          # fluent-plugin-google-cloud knows to flatten the field as textPayload

          # instead of jsonPayload after extracting 'time', 'severity' and

          # 'stream' from the record.

          message ${record['log']}

          # If 'severity' is not set, assume stderr is ERROR and stdout is INFO.

          severity ${record['severity'] || if record['stream'] == 'stderr' then 'ERROR' else 'INFO' end}

        </record>

        tag ${if record['stream'] == 'stderr' then 'raw.stderr' else 'raw.stdout' end}

        remove_keys stream,log

      </match>

      <match **>

        @type copy

        <store>

          @type google_cloud

          @id out_stackdriver

          @log_level debug

          # https://blog.monotone.dev/infrastructure/2019/10/28/using-stackdriver-with-your-own-server.html

          use_metadata_service false

          project_id airlog-prod-19a4e083

          zone aliyun-jp-tokyo

          # will check if the vm trick could be avoided later

          vm_id aliyun-fluentd-prod-tk

          vm_name aliyun-fluentd-prod-tk

          type k8s_container

          k8s_cluster_name aliyun-prod-tokyo

          k8s_cluster_location asia-east2

          # IMPORTANT: You cannot mix v1.0 and v0.12 styles in one plugin directive. If you mix v1.0 and v0.12 styles, v1.0 style is used and v0.12 style is ignored.

          # https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-compat_parameters#compat_parameters_convert-conf-types-kwargs

          # Set the chunk limit conservatively to avoid exceeding the recommended

          # chunk size of 3MB per write request.

          buffer_chunk_limit 3M

          # Flush logs every 2 seconds, even if the buffer is not full.

          flush_interval 2s

          # Enforce some limit on the number of retries.

          disable_retry_limit false

          # After 5 retries, a given chunk will be discarded.

          retry_limit 5

          # Wait 10 seconds before the first retry. The wait interval will be doubled on

          # each following retry (20s, 40s...) until it hits the retry limit.

          retry_wait 10

          # Never wait longer than 5 minutes between retries. If the wait interval

          # reaches this limit, the exponentiation stops.

          # Given the default config, this limit should never be reached, but if

          # retry_limit and retry_wait are customized, this limit might take effect.

          max_retry_wait 300

          # Use multiple threads for processing.

          num_threads 8

          detect_json true

          buffer_type file

          buffer_path /var/log/gcp-fluentd-infra-service.buffer

          # Use the gRPC transport.

          use_grpc true

          # set compression algorithm for grpc, possible value, none, gzip

          grpc_compression_algorithm gzip

          # If a request is a mix of valid log entries and invalid ones, ingest the

          # valid ones and drop the invalid ones instead of dropping everything.

          partial_success true

          # Enable monitoring via Prometheus integration.

          enable_monitoring true

          monitoring_type prometheus

        </store>

        <store>

          @type prometheus

          <metric>

            name fluentd_output_status_num_records_total

            type counter

            desc The total number of outgoing records

            <labels>

              nodename $.kubernetes.host

            </labels>

          </metric>

        </store>

      </match>

3.  the log view query result: