10
responses

lol so yeah my output is in another language??

I am running an XML input of data and trying to get it into an easy format to use for Elasitc. I followed the manual the best I could here https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_multiline_example_5 but my output is crazy.

##NxLog conf file##

<Extension multiline>
Module xm_multiline
HeaderLine /^\s*<Obj RefId="[0-9][0-9]?[0-9]?[0-9]?">/
</Extension>
<Extension _xml>
Module xm_xml
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Input in3>
Module im_file
File "C:\\Users\\administrator\\Desktop\\newtest.xml"
InputType multiline
SavePos FALSE
ReadFromLast FALSE
Exec parse_xml();
Exec to_json();
</Input>
<Output out3>
Module om_file
File "C:\\Users\\administrator\\Desktop\\testxml.txt"
</Output>
<Route>
Path in3 => out3
</Route>

##End conf##

##Data sample##

<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04">
<Obj RefId="12">
<TN RefId="4">
<T>System.Diagnostics.EventLogEntry#System/Microsoft-Windows-Kernel-General/16</T>
<T>System.Diagnostics.EventLogEntry#System/Microsoft-Windows-Kernel-General</T>
<T>System.Diagnostics.EventLogEntry</T>
<T>System.ComponentModel.Component</T>
<T>System.MarshalByRefObject</T>
<T>System.Object</T>
</TN>
<ToString>System.Diagnostics.EventLogEntry</ToString>
<Props>
<S N="MachineName">testserver</S>
<BA N="Data" />
<I32 N="Index">23749</I32>
<S N="Category">(0)</S>
<I16 N="CategoryNumber">0</I16>
<I32 N="EventID">16</I32>
<Obj N="EntryType" RefId="13">
<TNRef RefId="1" />
<ToString>Information</ToString>
<I32>4</I32>
</Obj>
<S N="Message">The description for Event ID '16' in Source 'Microsoft-Windows-Kernel-General' cannot be found. The local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them. The following information is part of the event:'109', '\??\C:\Users\testaccount\AppData\Local\Packages\Microsoft.Windows.Cortana_cw5n1h2txyewy\Settings\settings.dat', '12', '4'</S>
<S N="Source">Microsoft-Windows-Kernel-General</S>
<Obj N="ReplacementStrings" RefId="14">
<TNRef RefId="2" />
<LST>
<S>109</S>
<S>\??\C:\Users\testaccount\AppData\Local\Packages\Microsoft.Windows.Cortana_cw5n1h2txyewy\Settings\settings.dat</S>
<S>12</S>
<S>4</S>
</LST>
</Obj>
<I64 N="InstanceId">16</I64>
<DT N="TimeGenerated">2018-08-14T08:32:50-04:00</DT>
<DT N="TimeWritten">2018-08-14T08:32:50-04:00</DT>
<S N="UserName">testaccount</S>
<Nil N="Site" />
<Nil N="Container" />
</Props>
<MS>
<I32 N="EventID">16</I32>
</MS>
</Obj>
</Objs>

##End Sample##

##Output##

<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04">഍
਍  㰀伀戀樀 刀攀昀䤀搀㴀∀㄀㈀∀㸀ഀ਍ഀ
<TN RefId="4">഍
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍    㰀⼀吀一㸀ഀ਍ഀ笊䔢敶瑮敒散癩摥楔敭㨢㈢㄰ⴸ㠰㌭‰ㄱ〺㨱㐴Ⱒ匢畯捲䵥摯汵乥浡≥∺湩∳∬潓牵散潍畤敬祔数㨢椢彭楦敬索਍ഀ
<Props>഍
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍      㰀伀戀樀 一㴀∀䔀渀琀爀礀吀礀瀀攀∀ 刀攀昀䤀搀㴀∀㄀㌀∀㸀ഀ਍ഀ笊䔢敶瑮敒散癩摥楔敭㨢㈢㄰ⴸ㠰㌭‰ㄱ〺㨱㐴Ⱒ匢畯捲䵥摯汵乥浡≥∺湩∳∬潓牵散潍畤敬祔数㨢椢彭楦敬索਍ഀ笊䔢敶瑮敒散癩摥楔敭㨢㈢㄰ⴸ㠰㌭‰ㄱ〺㨱㐴Ⱒ匢畯捲䵥摯汵乥浡≥∺湩∳∬潓牵散潍畤敬祔数㨢椢彭楦敬索਍ഀ笊䔢敶瑮敒散癩摥楔敭㨢㈢㄰ⴸ㠰㌭‰ㄱ〺㨱㐴Ⱒ匢畯捲䵥摯汵乥浡≥∺湩∳∬潓牵散潍畤敬祔数㨢椢彭楦敬索਍ഀ
</Obj>഍
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍      㰀伀戀樀 一㴀∀刀攀瀀氀愀挀攀洀攀渀琀匀琀爀椀渀最猀∀ 刀攀昀䤀搀㴀∀㄀㐀∀㸀ഀ਍ഀ笊䔢敶瑮敒散癩摥楔敭㨢㈢㄰ⴸ㠰㌭‰ㄱ〺㨱㐴Ⱒ匢畯捲䵥摯汵乥浡≥∺湩∳∬潓牵散潍畤敬祔数㨢椢彭楦敬索਍ഀ
<LST>഍
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍        㰀⼀䰀匀吀㸀ഀ਍ഀ
</Obj>഍
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍    㰀⼀倀爀漀瀀猀㸀ഀ਍ഀ
<MS>഍
਍≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨱㄰㐺∴∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ
਍    㰀⼀䴀匀㸀ഀ਍ഀ
</Obj>഍
਍ 㰀⼀伀戀樀猀㸀ഀ

##End Output##

AskedAugust 30, 2018 - 5:12pm

Answer (1)

Looks like its time to break out the Rosetta Stone ;)

In seriousness this is most likely a character set issue.

Check out xm_charconv and maybe start with Exec convert_fields("auto", "utf-8"); in your Input.

https://nxlog.co/documentation/nxlog-user-guide#xm_charconv
https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_charconv

Comments (9)

  • motts's picture

    haha right? i tried to copy paste into google translate but it was uhhhhhh chinese? but provided no translation.

    I tried getting the latest version of NxLog, but the issue persisted.

    So I tried you suggestion, and that didn't seem too difficult to configure. When I added this to the code....

    <Extension _charconv>
    Module xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
    </Extension>
    <Input in3>
    Exec convert_fields("auto", "utf-8");
    Module im_file
    File "C:\\Users\\administrator\\Desktop\\newtest.xml"

    ...

    I get the same different language thing going on, but not I am getting this error

    2018-08-30 12:56:06 ERROR procedure 'parse_xml' failed at line 36, character 20
    in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been a
    orted; XML parse error at line 1: not well-formed (invalid token)

    So I commented out the parse_xml and I no longer get that error, but i get the same Chinese language, but it's all in one line now lol

    ≻癅湥剴捥楥敶呤浩≥∺〲㠱〭ⴸ〳ㄠ㨲㜵㔺∶∬潓牵散潍畤敬慎敭㨢椢㍮Ⱒ匢畯捲䵥摯汵呥灹≥∺浩晟汩≥ൽ笊䔢敶瑮敒散癩摥....

    Super strange

  • motts's picture

    I also just commented out the to_json section. and I am not getting half the data while half the lines are in Chinese and the other half are in Endlish lol wtffff lol I fee like someone is playing some kind of joke on me.

    駋ꏅ<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04">഍
    ਍  㰀伀戀樀 刀攀昀䤀搀㴀∀㄀㈀∀㸀ഀ਍ഀ
    <TN RefId="4">഍
    ਍      㰀吀㸀匀礀猀琀攀洀⸀䐀椀愀最渀漀猀琀椀挀猀⸀䔀瘀攀渀琀䰀漀最䔀渀琀爀礀⌀匀礀猀琀攀洀⼀䴀椀挀爀漀猀漀昀琀ⴀ圀椀渀搀漀眀猀ⴀ䬀攀爀渀攀氀ⴀ䜀攀渀攀爀愀氀⼀㄀㘀㰀⼀吀㸀ഀ਍ഀ
    <T>System.Diagnostics.EventLogEntry#System/Microsoft-Windows-Kernel-General</T>഍
    ਍      㰀吀㸀匀礀猀琀攀洀⸀䐀椀愀最渀漀猀琀椀挀猀⸀䔀瘀攀渀琀䰀漀最䔀渀琀爀礀㰀⼀吀㸀ഀ਍ഀ
    <T>System.ComponentModel.Component</T>഍
    ਍      㰀吀㸀匀礀猀琀攀洀⸀䴀愀爀猀栀愀氀䈀礀刀攀昀伀戀樀攀挀琀㰀⼀吀㸀ഀ਍ഀ
    <T>System.Object</T>഍
    ਍    㰀⼀吀一㸀ഀ਍ഀ
    <ToString>System.Diagnostics.EventLogEntry</ToString>഍
    ਍    㰀倀爀漀瀀猀㸀ഀ਍ഀ
    <S N="MachineName">testserver</S>഍
    ਍      㰀䈀䄀 一㴀∀䐀愀琀愀∀ ⼀㸀ഀ਍ഀ
    <I32 N="Index">23749</I32>഍
    ਍      㰀匀 一㴀∀䌀愀琀攀最漀爀礀∀㸀⠀ ⤀㰀⼀匀㸀ഀ਍ഀ
    <I16 N="CategoryNumber">0</I16>഍
    ਍      㰀䤀㌀㈀ 一㴀∀䔀瘀攀渀琀䤀䐀∀㸀㄀㘀㰀⼀䤀㌀㈀㸀ഀ਍ഀ
    <Obj N="EntryType" RefId="13">഍
    ਍        㰀吀一刀攀昀 刀攀昀䤀搀㴀∀㄀∀ ⼀㸀ഀ਍ഀ
    <ToString>Information</ToString>഍
    ਍        㰀䤀㌀㈀㸀㐀㰀⼀䤀㌀㈀㸀ഀ਍ഀ
    </Obj>഍

  • motts's picture

    Just tried to put the parse_xml here

    <Extension multiline>
    Module xm_multiline
    HeaderLine /^\s*<Obj RefId="[0-9][0-9]?[0-9]?[0-9]?">/
    Exec parse_xml();
    </Extension>

    And I am again getting the every other line mismatched language

    also I am getting this error

    2018-08-30 13:07:12 ERROR procedure 'parse_xml' failed at line 18, character 20
    in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been ab
    orted; XML parse error at line 2: no element found

  • b0ti's picture
    (NXLog)

    I have a feeling that newtest.xml uses a multibyte encoding, most likely UTF-16. You check this first.
    Once the encoding is known it can be converted to utf-8 and the rest should work.

  • motts's picture

    Just for giggles, I copied some of the stuff into Google translate. It didn't directly translate to English, but instead put the pronunciation directly below the Chinese characters. I copied the words it had there and placed them in the translation and this is what it came back with...

    ≻Forum of the floor of the floor of the floor of the floor with the light bulb of the light bulb ∺〲 ∺ ൽ ൽ ൽ ൽ ൽ ൽ ൽ ൽ ൽ The floor is covered with a pile of floor coverings. The floor is covered with floor tiles and the floor is covered with floor tiles. Bring your grandchildren to dinner at the bar floor with a glass of wine and a glass of wine with your favorite dish. shēn floor is a museum of xiē yáng pī ㈢ ㄰ ⴸ

    lol

  • motts's picture

    Ok. Took me a sec to try to figure out the encoding. Easiest way for me to find out was to open the xml in Notepad++ and in the bottom right it displays the character set, which is "UCS-2 LE BOM".

    I am not sure how to put that into NxLog because I cannot seem to find ucs-2 le bom as an option unless it's also called something else.

    <Extension multiline>
    Module xm_multiline
    HeaderLine /^\s*<Obj RefId="[0-9][0-9]?[0-9]?[0-9]?">/
    </Extension>
    <Extension _xml>
    Module xm_xml
    </Extension>
    <Extension _json>
    Module xm_json
    </Extension>
    <Extension _charconv>
    Module xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32, ucs-2
    </Extension>
    <Input in3>
    Exec convert_fields("ucs-2 le bom", "utf-8");
    Module im_file
    File "C:\\Users\\administrator\\Desktop\\newtest.xml"
    InputType multiline
    SavePos FALSE
    ReadFromLast FALSE
    Exec parse_xml();
    Exec to_json();
    </Input>
    <Output out3>
    Module om_file
    File "C:\\Users\\administrator\\Desktop\\testxml.txt"
    </Output>

    This config will not take "ucs-2 le bom". I have also tried other combinations

    Did some googling and found results for utf-16, so I input that in place of ucs-2 le bom, but still in foreign characters.

  • motts's picture

    Thanks

    I cannot seem to get it to work though. I have this here just before the input section

    <Extension _charconv>
    Module xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32, ucs-2
    Exec convert_fields("utf-2", "utf-8");
    Exec $raw_event = convert($raw_event, 'UCS-2LE', 'UTF-8');
    </Extension>

    Still getting the foreign output :(

    I tried placing it here and it also didn't work

    <Input in3>
    Module im_file
    File "C:\\Users\\administrator\\Desktop\\newtest.xml"
    InputType multiline
    SavePos FALSE
    ReadFromLast FALSE
    Exec $raw_event = convert($raw_event, 'UCS-2LE', 'UTF-8');
    Exec parse_xml();
    Exec to_json();
    </Input>