Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Assistance required in log file ingestion
navdeepsingh83 created
Hi,
We have a following log file from open source password manager solution. It runs on tomcat. We have graylog server where we would like to send the log data and parse it. Now, we can send the log file to graylog however the entire line comes as one message block, instead of parsing into fields automatically. I am wondering how can i convert the file into csv and send to graylog.
Here is sample log. It doesn't come with any header.
2018-08-25T07:40:14Z, ERROR, http.PwmResponse, {117412} 5028 ERROR_BAD_SESSION (client unable to reply with session key) [xx.xx.47.82]
2018-08-25T07:40:15Z, ERROR, filter.SessionFilter, {117413} 5028 ERROR_BAD_SESSION (client unable to reply with session key) [xx.xx.47.82]
2018-08-25T07:40:15Z, ERROR, http.PwmResponse, {117413} 5028 ERROR_BAD_SESSION (client unable to reply with session key) [xx.xx.47.82]
2018-08-25T07:40:17Z, ERROR, filter.SessionFilter, {117415} 5028 ERROR_BAD_SESSION (client unable to reply with session key) [xx.xx.47.82]
2018-08-25T07:40:17Z, ERROR, http.PwmResponse, {117415} 5028 ERROR_BAD_SESSION (client unable to reply with session key) [xx.xx.47.82]
2018-08-25T10:04:28Z, ERROR, filter.RequestInitializationFilter, {117422} 5063 ERROR_SECURITY_VIOLATION (current network address 'yy.yy.185.123' has changed from original network address 'yy.yy.173.181') [yy.yy.173.181]
2018-08-25T10:04:28Z, ERROR, http.PwmResponse, {117422} 5063 ERROR_SECURITY_VIOLATION (current network address 'yy.yy.185.123' has changed from original network address 'yy.yy.173.181') [yy.yy.173.181]
2018-08-25T11:08:03Z, INFO , auth.LDAPAuthenticationRequest, {117467} authID=130, successful ldap authentication for UserIdentity{"userDN":"CN=UserA,CN=Users,DC=org,DC=com","ldapProfile":"default"} (606ms) type: AUTHENTICATED, using strategy BIND, using proxy connection: false, returning bind dn: CN=UserA,CN=Users,DC=org,DC=com [yy.yy.32.238]
2018-08-25T11:08:03Z, INFO , event.AuditService, audit event: {"perpetratorID":"UserA","perpetratorDN":"CN=UserA,CN=Users,DC=org,DC=com","perpetratorLdapProfile":"default","sourceAddress":"yy.yy.32.238","sourceHost":"yy.yy.32.238","type":"USER","eventCode":"AUTHENTICATE","guid":"941aa151-8998-4c89-b690-484e623429d8","timestamp":"2018-08-25T05:38:03Z","message":"type=AUTHENTICATED, source=LOGIN_FORM","narrative":"UserA (CN=UserA,CN=Users,DC=org,DC=com) has authenticated","xdasTaxonomy":"XDAS_AE_AUTHENTICATE_ACCOUNT","xdasOutcome":"XDAS_OUT_SUCCESS"}
2018-08-25T11:08:48Z, INFO , operations.PasswordUtility, {117467,UserA} user 'UserIdentity{"userDN":"CN=UserA,CN=Users,DC=org,DC=com","ldapProfile":"default"}' successfully changed password [yy.yy.32.238]
2018-08-25T11:08:49Z, INFO , event.AuditService, audit event: {"perpetratorID":"UserA","perpetratorDN":"CN=UserA,CN=Users,DC=org,DC=com","perpetratorLdapProfile":"default","sourceAddress":"yy.yy.32.238","sourceHost":"yy.yy.32.238","type":"USER","eventCode":"CHANGE_PASSWORD","guid":"00c158d5-0ea5-46aa-8c8c-cd279f783ecd","timestamp":"2018-08-25T05:38:49Z","narrative":"UserA (CN=UserA,CN=Users,DC=org,DC=com) has changed their password","xdasTaxonomy":"XDAS_AE_SET_CRED_ACCOUNT","xdasOutcome":"XDAS_OUT_SUCCESS"}
2018-08-25T11:10:04Z, ERROR, filter.RequestInitializationFilter, {117471} 5063 ERROR_SECURITY_VIOLATION (current network address 'yy.yy.112.147' has changed from original network address 'xx.xx.243.3') [xx.xx.243.3]
I wrote the following nxlog conf but it doesn't seems to be working.
<Extension tomcat>
Module xm_csv
Fields $DateTime,$Type,$Category,$Details
FieldTypes string,string,string,string
Delimiter ","
</Extension>
<Input in_pwm>
Module im_file
File "C:\\Users\\Documents\\TempOut\\PWM\\PWM.log"
PollInterval 1
ReadFromLast False
#Recursive True
SavePos False
Exec tomcat->parse_csv();
</Input>
Appreciate your assistance in getting this working.
navdeepsingh83 created