Re: Multiline - Very Irregular Log File

[Mr. Fiber]

If you hard to parse the logs by regular expression,

writing parser plugins is better.

Masahiro

On Mon, Jul 18, 2016 at 11:09 PM, Kernel Panic <netwar...@gmail.com> wrote:

Hi guys.

I've got log file which is giving me a headache, this is a extract of it:


[time 18/07/2016 10:57:12] [region 441] [environment fesitesplus] [status DEBUG] [user_id 14019350] [code Array] [event Controller_Action_Helper_BillingPlazavip::editarCliente : 579] [client 128.177.98.212] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/home/user.php?micuentafull#micuentavtc][request Array(    [estadoRecurrencia] => 0    [id] => 14019350    [fechaNacimiento] =>     [ipCliente] => 128.177.98.212)] [message response]
[time 18/07/2016 10:57:12] [region 441] [environment fesitesplus] [status END] [user_id 14019350] [code editarCliente] [event Controller_Action_Helper_BillingPlazavip::editarCliente : 619] [client 128.177.98.212] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/home/user.php?micuentafull#micuentavtc][request Array(    [estadoRecurrencia] => 0    [id] => 14019350    [fechaNacimiento] =>     [ipCliente] => 128.177.98.212)] [message editarCliente Response ---> Array]
[time 18/07/2016 10:57:12] [region 441] [environment fesitesplus] [status NOTICE] [user_id 14019350] [code cancelarCliente] [event Controller_Action_Helper_BillingPlazavip::cancelarCliente : 930] [client 128.177.98.212] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/home/user.php?micuentafull#micuentavtc][request Array(    [estadoRecurrencia] => 0    [id] => 14019350)][response Array(    [codigo] => 0    [cliente] => Array        (            [estadoRecurrencia] => 0            [id] => 14019350            [fechaNacimiento] =>             [ipCliente] => 128.177.98.212        )    [error] =>     [errordesc] =>     [error_msg] => )] [message fin cancelarCliente]
[time 18/07/2016 10:57:57] [region 441] [environment fesitesplus] [status DEBUG] [user_id 13603379] [code Array] [event Controller_Action_Helper_BillingPlazavip::listarTarjetas : 1273] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser][request Array(    [idCliente] => 13603379)] [message response]
[time 18/07/2016 10:57:57] [region 441] [environment fesitesplus] [status END] [user_id 13603379] [code listarTarjetas] [event Controller_Action_Helper_BillingPlazavip::listarTarjetas : 1337] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser][request Array(    [idCliente] => 13603379)] [message listarTarjetas Response ---> Array]
[time 18/07/2016 10:58:41] [region 441] [environment fesitesplus] [status START] [user_id 17891889] [code step: ] [event TelmexmexicogateV2Import::vtcworkflow : 2274] [client 128.177.98.212] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/home/validacionusuariocompra.php?vtcworkflow&product_id=&product_type=package_id&onlyabono=true]
[time 18/07/2016 10:58:45] [region 441] [environment fesitesplus] [status END] [user_id 17891889] [code -1] [event TelmexmexicogateV2Import::vtcworkflow : 2985] [client 128.177.98.212] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/home/validacionusuariocompra.php?vtcworkflow&product_id=&product_type=package_id&onlyabono=true]
[time 18/07/2016 10:58:47] [region 441] [environment fesitesplus] [status START] [user_id 17891889] [code -1] [event HubgateV2Import::backgroundAction : 129] [client 128.177.98.212] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/home/validacionusuariocompra.php?vtcworkflow&product_id=&product_type=package_id&onlyabono=true]
[time 18/07/2016 11:01:26] [region 441] [environment fesitesplus] [status START] [user_id 3107408] [code step: confirm] [event TelmexmexicogateV2Import::vtcworkflow : 2662] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser/workflowstart.php?vtcworkflow&product_id=722461&product_type=group_id&object_id=722461&object_type=G&offer_id=13461046&from=abonos&url_ok=/&url_error=/&newplay=1]
[time 18/07/2016 11:01:28] [region 441] [environment fesitesplus] [status NOTICE] [user_id 3107408] [code showcardup] [event TelmexmexicogateV2Import::cargoUnicoTelmexPlazavip : 2061] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser/workflowstart.php?vtcworkflow&product_id=722461&product_type=group_id&object_id=722461&object_type=G&offer_id=13461046&from=abonos&url_ok=/&url_error=/&newplay=1][request Array(    [account] => )][response Array(    [idCliente] => 3107408    [telefonoTelmex] => 7262692223    [monto] => 39    [caja] => 1    [ticket] => 46462764871802164    [codigoPromo] =>     [titulo] => Cómo ser soltera    [hora] => 11:01:28)] [message RENTA: Parametros cargoUnicoTelmexPlazavip]
[time 18/07/2016 11:01:28] [region 441] [environment fesitesplus] [status START] [user_id 3107408] [code Inicio cargoUnicoTelmex] [event Controller_Action_Helper_BillingPlazavip::cargoUnicoTelmex : 631] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser/workflowstart.php?vtcworkflow&product_id=722461&product_type=group_id&object_id=722461&object_type=G&offer_id=13461046&from=abonos&url_ok=/&url_error=/&newplay=1][request Array(    [idCliente] => 3107408    [telefonoTelmex] => 7262692223    [monto] => 39    [caja] => 1    [ticket] => 46462764871802164    [codigoPromo] =>     [titulo] => Cómo ser soltera    [hora] => 11:01:28    [ipCliente] => 107.14.44.202)] [message REMOTE_ADDR]
[time 18/07/2016 11:01:29] [region 441] [environment fesitesplus] [status DEBUG] [user_id 3107408] [code _cargoUnicoTelmex response] [event Controller_Action_Helper_BillingPlazavip::cargoUnicoTelmex : 643] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser/workflowstart.php?vtcworkflow&product_id=722461&product_type=group_id&object_id=722461&object_type=G&offer_id=13461046&from=abonos&url_ok=/&url_error=/&newplay=1][request Array(    [idCliente] => 3107408    [telefonoTelmex] => 7262692223    [monto] => 39    [caja] => 1    [ticket] => 46462764871802164    [codigoPromo] =>     [titulo] => Cómo ser soltera    [hora] => 11:01:28    [ipCliente] => 107.14.44.202)][response Array(    [entry] => Array        (            [API_REMOTE_ADDR] => 189.139.35.138            [idCliente] => 3107408            [telefonoTelmex] => 7262692223            [monto] => 39            [caja] => 1            [ticket] => 46462764871802164            [codigoPromo] =>             [titulo] => Cómo ser soltera            [hora] => 11:01:28            [ipCliente] => 107.14.44.202        )    [response] => Array        (            [codigo] => 0            [autorizado] =>             [error] =>             [errordesc] =>             [bancoerror] =>         ))] [message error _cargoUnicoTelmex response]
[time 18/07/2016 11:01:29] [region 441] [environment fesitesplus] [status NOTICE] [user_id 3107408] [code cargoUnicoTelmex] [event TelmexmexicogateV2Import::cargoUnicoTelmexPlazavip : 2077] [client 107.14.44.202] [server 10.11.22.225] [referer https://www.clarovideo.com/mexico/homeuser/workflowstart.php?vtcworkflow&product_id=722461&product_type=group_id&object_id=722461&object_type=G&offer_id=13461046&from=abonos&url_ok=/&url_error=/&newplay=1][request Array(    [idCliente] => 3107408    [telefonoTelmex] => 7262692223    [monto] => 39    [caja] => 1    [ticket] => 46462764871802164    [codigoPromo] =>     [titulo] => Cómo ser soltera    [hora] => 11:01:28)][response Array(    [codigo] => 0    [autorizado] =>     [error] =>     [errordesc] =>     [bancoerror] =>     [id_cliente] => 3107408)] [message response cargoUnicoTelmex]


I need to parse every line which starts with [time, and there is no END of line match,  so that's much more difficult to achieve, and every new entry varies from the previous one,  I'm doing something like this:

<source>
  @type tail

  format multiline
  format_firstline '/^\[time/'
  format1 '/^\[(?<_time>.*?)\] \[(?<_region>.*?)\] \[(?<_environment>.*?)\] \[(?<_status>.*?)\] \[(?<_userid>.*?)\] \[(?<_codeinicioEditarLcient>.*?)\] \[(?<_eventControllerActionHelper>.*?)\] \[(?<_client>.*?)\] \[(?<_server>.*?)\] \[(?<_referer>.*?)\] \[(?<_requestArray>.*?)\]/'

  path /path_to_file/Billing.*.log.html
  pos_file /var/log/fluent/Billing.pos
  tag td.vtc.Billing
</source>

<match td.vtc.Billing>
  @type file
  path /var/log/fluent/Multilinetest.log
</match>

But I have no luck, I also tried to parse all the line as in  (?<time>.*?).* but no luck, any help appreciated
Regards

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.