promtail regex not match log with ANSI color

8 views
Skip to first unread message

Dongyu Shen

unread,
Sep 5, 2023, 9:34:43 PM9/5/23
to lokiproject
I am using regular expressions to process my logs.

My promtail config is:
```yaml
- job_name: kubernetes-pods
  pipeline_stages:
    - docker: {}
    - regex:
         expression: '^(?P<timestamp>[0-9 :.-]+?) \| \\u001b\[0;00;[0-9]+m(?P<logLevel>[DIWEF])\\u001b\[0m \| (?P<traceID>.*) \| (?P<location>[^\s]+:\d+) \\u001b\[0;00;[0-9]+m(?P<message>.+)\\u001b\[0m.*$'
    - labels:
        timestamp: timestamp
        logLevel: logLevel
        traceID: traceID
        location: location
        message: message
```

My log is like:
```
09-04 18:22:07.415 | \u001b[0;00;32mI\u001b[0m |  | handler.go:383 \u001b[0;00;32mget all robot stop plan info\u001b[0m\n
```

I got the error message from promtail debug log:
```
level=debug ts=2023-09-06T01:27:27.313448004Z caller=regex.go:121 component=file_pipeline component=stage type=regex msg="regex did not match" input="09-04 18:22:07.415 | \u001b[0;00;32mI\u001b[0m |  | handler.go:383 \u001b[0;00;32mget all robot stop plan info\u001b[0m\n" regex="^(?P<timestamp>[0-9 :.-]+?) \\| \\\\u001b\\[0;00;[0-9]+m(?P<logLevel>[DIWEF])\\\\u001b\\[0m \\| (?P<traceID>.*) \\| (?P<location>[^\\s]+:\\d+) \\\\u001b\\[0;00;[0-9]+m(?P<message>.+)\\\\u001b\\[0m.*$"
```

But I tested the regex in Go and it was correct.

```go
package main

import (
"fmt"
"regexp"
)

func main() {
logLine := `09-04 18:22:07.415 | \u001b[0;00;32mI\u001b[0m |  | handler.go:383 \u001b[0;00;32mget all robot stop plan info\u001b[0m\n`
re := regexp.MustCompile(`^(?P<timestamp>[0-9 :.-]+?) \| \\u001b\[0;00;[0-9]+m(?P<logLevel>[DIWEF])\\u001b\[0m \| (?P<traceID>.*) \| (?P<location>[^\s]+:\d+) \\u001b\[0;00;[0-9]+m(?P<message>.+)\\u001b\[0m.*$`)
match := re.FindStringSubmatch(logLine)

if match != nil {
groups := make(map[string]string)
for i, name := range re.SubexpNames() {
if name != "" {
groups[name] = match[i]
}
}

fmt.Printf("Timestamp: %s\n", groups["timestamp"])
fmt.Printf("Log Level: %s\n", groups["logLevel"])
fmt.Printf("Trace ID: %s\n", groups["traceID"])
fmt.Printf("Location: %s\n", groups["location"])
fmt.Printf("Message: %s\n", groups["message"])
} else {
fmt.Println("Log line does not match the expected format.")
}
}
```

output:
```
Timestamp: 09-04 18:22:07.415
Log Level: I
Trace ID:
Location: handler.go:383
Message: get all robot stop plan info
```

Who can tell me where the problem is? THX!
Reply all
Reply to author
Forward
0 new messages