1
answer

Hi,

I'm trying to use nxlog to to extract three metrics from a .set file. My OS is Windows 10. I edited the .conf file in "C:\Program Files (x86)\nxlog\conf" and it looks like this:

Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

<Extension json>
Module xm_json
</Extension>

<Extension sikora>
Module xm_set
Fields $Nominal, $PlusTol, $Oval # fields of interest (metrics)
FieldTypes string, string, string # type of variable
Delimiter ;
EscapeControl FALSE
</Extension>

<Input sikora_logs>
Module im_file
File "C:\\Users\50051145\Desktop\nx_log\\\\*.set" # imput file
#ReadFromLast True
#Recursive True
#SavePos True
ReadFromLast False
Recursive False
SavePos False

<Exec>

if $raw_event =~ /^Macrosezione : (.+)/ # creazione variabile
{ #
create_var('macrosection'); #
set_var('macrosection', $1); #
drop(); #
}

sikora->parse_set();
delete($EventReceivedTime);
delete($SourceModuleName);
delete($SourceModuleType);

if $raw_event =~ /^Operatore /
{ # variable definition for the
if not defined get_var('start_time') # timestamp
{ #
log_debug("parsed_time: " + strptime($time, "%d/%m/%Y %I:%M:%S")); #
create_var('start_time'); #
set_var('start_time', strptime($time, "%d/%m/%Y %I:%M:%S"));
drop();
}
else
{
if get_var('start_time') != strptime($time,"%d/%m/%Y %I:%M:%S")
{
log_debug("old_time: " + get_var('start_time'));
log_debug("new_time: ", $time);
set_var('start_time', strptime($time,"%d/%m/%Y %I:%M:%S"));
drop();
}
}
}

$time = (integer(get_var('start_time')) / 1000000 + integer($time)) * 1000; # formula to convert timestamp in milliseconds
$pressure = integer($pressure);
$macrosection = get_var('macrosection');
$nominal = get_var('nominal');
$type = get_var('type');

to_json();

</Exec>
</Input>

<Output out>
Module om_file #
CreateDir TRUE #
File "C:\\Users\\50051145\\Desktop\\temp" + $fileName # output file
</Output>

<Route 1>
Path sikora_logs => out
</Route>

When I run the program I expect an output file in a folder on my desktop "C:\\Users\\50051145\\Desktop\\temp" but I get nothing. I checked the logs and I get this:

2019-09-10 18:20:34 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\extension\xm_set.dll, The specified module could not be found. ;
The specified module could not be found.
2019-09-10 18:20:34 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\output\om_file #.dll, The specified module
could not be found. ; The specified module could not be found.
2019-09-10 18:20:34 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:38; couldn't parse statement at line 45, character 28 in
C:\Program Files (x86)\nxlog\conf\nxlog.conf; module sikora not found
2019-09-10 18:20:34 ERROR module 'sikora_logs' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:93
2019-09-10 18:20:34 ERROR module 'out' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:93
2019-09-10 18:20:34 ERROR route 1 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:93
2019-09-10 18:20:34 WARNING no routes defined!
2019-09-10 18:20:34 WARNING not starting unused module sikora_logs
2019-09-10 18:20:34 INFO nxlog-ce-2.10.2150 started
2019-09-11 11:10:27 WARNING stopping nxlog service
2019-09-11 11:10:27 WARNING nxlog-ce received a termination request signal, exiting...

It appers that xm_set.dll library is missing, "The specified module could not be found". I found out in "C:\Program Files (x86)\nxlog\modules\extension" that nxlog doesn't come with a .set library. How can I add this library ?

Thank you

AskedSeptember 12, 2019 - 11:31am

Answer (1)

Your main issue starts here. There is no xm_set module. You are likely wanting xm_csv instead.

<Extension sikora>
    Module xm_set
    Fields $Nominal, $PlusTol, $Oval # fields of interest (metrics)
    FieldTypes string, string, string # type of variable
    Delimiter ;
    EscapeControl FALSE
</Extension>

Another issue is with the inline comments on your Output module:

<Output out>
    Module om_file #
    CreateDir TRUE #
    File "C:\\Users\\50051145\\Desktop\\temp" + $fileName # output file
</Output>

This is what is causing:

2019-09-10 18:20:34 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\output\om_file #.dll, The specified module
could not be found. ; The specified module could not be found.
AnsweredSeptember 12, 2019 - 8:12pm

Comments (8)

  • IoT_fra_87's picture

    Hi Zhengshi,

    thanks for your help! So you are telling me there is no way to add a xm_set module? Do I have to work with a csv file ? I will run some tests. Could you please tell me if there are some other issues with my code ? I'll appreciate that a lot.

    September 13, 2019 - 10:48am
  • Zhengshi's picture
    (NXLog)

    I would remove all in-line comments and place them above or below. These will cause issues for you.

    I am unfamiliar with .set files. From your config, it looks like the contents of the .set files may be an ascii file with CSV layout delimited by a semi-colon (;). If this is accurate, then xm_csv is what you need as we do not have an xm_set module.

    September 13, 2019 - 8:19pm
  • IoT_fra_87's picture

    I removed all in-line comments completely. Yes the content of the .set file is an ascii file except data is not delimited by a semi-colon ( ; )
    but is written as a column:

    [ANALYSIS]
    TUBE=1
    CABLE=0
    [OTHERS]
    RECIPE=Tab73e112 DN16 2SCR16_1E4198
    VALUE_TAB_NUMBER=3
    [D0]
    Nominal=21.40
    PlusTol=0.40
    MinusTol=0.40
    PlusWarnTol=0.40
    MinusWarnTol=0.40
    Oval=0.00e+000
    [D2]
    Nominal=23.80
    PlusTol=0.10
    MinusTol=0.10
    PlusWarnTol=0.05
    MinusWarnTol=0.05
    Oval=0.00e+000
    TrAvMax=29.38
    TrAvMin=28.66
    [SECTOR]
    TwistLength=0.000000e+000
    [XRAY]
    CableType=60
    Nominal=1.20
    PlusTol=0.30
    MinusTol=0.20
    TrAvMin=0.81
    TrAvMax=1.32
    PlusWarnTol=0.00e+000
    MinusWarnTol=0.00e+000
    [XRAYE]
    PlusTol=0.10
    [XRAYC]
    PlusTol=0.20
    PlusWarnTol=0.00e+000

    As I told you from this file I have to extract three metrics: Nominal, PlusTol and Oval. I also renamed the file .set in a .csv file as you suggested.
    Now my code looks like this:

    Panic Soft
    #NoFreeOnExit TRUE

    define ROOT C:\Program Files (x86)\nxlog
    define CERTDIR %ROOT%\cert
    define CONFDIR %ROOT%\conf
    define LOGDIR %ROOT%\data
    define LOGFILE %LOGDIR%\nxlog.log
    LogFile %LOGFILE%

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data

    <Extension json>
    Module xm_json
    </Extension>

    <Extension sikora>
    Module xm_csv
    Fields $Nominal, $PlusTol, $Oval
    FieldTypes string, string, string
    Delimiter ;
    EscapeControl FALSE
    </Extension>

    <Input sikora_logs>
    Module im_file
    File "C:\Users\50051145\Desktop\CAT\nx_log\\\\*.csv"
    #ReadFromLast True
    #Recursive True
    #SavePos True
    ReadFromLast False
    Recursive False
    SavePos False

    <Exec>

    if $raw_event =~ /^Macrosezione : (.+)/
    {
    create_var('macrosection');
    set_var('macrosection', $1);
    drop();
    }

    sikora->parse_csv();
    delete($EventReceivedTime);
    delete($SourceModuleName);
    delete($SourceModuleType);

    if $raw_event =~ /^Operatore /
    {
    if not defined get_var('start_time')
    {
    log_debug("parsed_time: " + strptime($time, "%d/%m/%Y %I:%M:%S"));
    create_var('start_time');
    set_var('start_time', strptime($time, "%d/%m/%Y %I:%M:%S"));
    drop();
    }
    else
    {
    if get_var('start_time') != strptime($time,"%d/%m/%Y %I:%M:%S")
    {
    log_debug("old_time: " + get_var('start_time'));
    log_debug("new_time: ", $time);
    set_var('start_time', strptime($time,"%d/%m/%Y %I:%M:%S"));
    drop();
    }
    }
    }

    $time = (integer(get_var('start_time')) / 1000000 + integer($time)) * 1000;
    $pressure = integer($pressure);
    $macrosection = get_var('macrosection');
    $nominal = get_var('nominal');
    $type = get_var('type');

    to_json();

    </Exec>
    </Input>

    <Output out>
    Module om_file
    CreateDir TRUE
    File "C:\\Users\\50051145\\Desktop\\temp" + $fileName
    </Output>

    <Route 1>
    Path sikora_logs => out
    </Route>

    When I run the program I expect an output file in "C:\\Users\\50051145\\Desktop\\temp" but I get nothing. I checked the logs and I have this:

    2019-09-17 10:31:07 INFO nxlog-ce-2.10.2150 started
    2019-09-17 10:31:07 ERROR failed to open directory: C:\Users\50051145\Desktop\CAT;x_log\: The filename, directory name, or volume label syntax is incorrect.
    2019-09-17 10:31:07 WARNING Module sikora_logs has no input files to read
    2019-09-17 10:33:11 WARNING stopping nxlog service
    2019-09-17 10:33:11 WARNING nxlog-ce received a termination request signal, exiting...

    I don't understand why the program reads the directory of the input file with a semicolon. As you can see from my code in <Input sikora_logs>
    the input file has this path: "C:\Users\50051145\Desktop\CAT\nx_log\\\\*.csv".
    Any help would be appreciated.

    Thank you again for your help Zhengshi.

    September 17, 2019 - 10:54am
  • b0ti's picture
    (NXLog)

    File "C:\Users\50051145\Desktop\CAT\nx_log\\\\*.csv"

    See the NOTE in the im_file documentation.

    September 17, 2019 - 8:33pm
  • IoT_fra_87's picture

    I saw the NOTE in the im_file documentation. I edited the file path: "C:\\Users\\\\*.csv". Now the program seems to
    get the input file (CSV). I edited my code and now looks like this:

    Panic Soft
    #NoFreeOnExit TRUE

    define ROOT C:\Program Files (x86)\nxlog
    define CERTDIR %ROOT%\cert
    define CONFDIR %ROOT%\conf
    define LOGDIR %ROOT%\data
    define LOGFILE %LOGDIR%\nxlog.log
    LogFile %LOGFILE%

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data

    <Extension json>
    Module xm_json
    </Extension>

    <Extension sikora>
    Module xm_csv
    Fields $Nominal, $PlusTol, $Oval
    FieldTypes string, string, string
    Delimiter ;
    EscapeControl FALSE

    </Extension>

    <Input sikora_logs>
    Module im_file
    File "C:\\Users\\\\*.csv"
    ReadFromLast False
    Recursive False
    SavePos False

    <Exec>

    if $raw_event =~ /^macrosezione : (.+)/
    {
    create_var('macrosection');
    set_var('macrosection', $1);
    drop();
    }

    if $raw_event =~ /^Nominale : (.+)/
    {
    create_var('Nominal');
    set_var('Nominal', $1);
    drop();
    }

    if $raw_event =~ /^Plustotale : (.+)/
    {
    create_var('PlusTol');
    set_var('PlusTol', $1);
    drop();
    }

    if $raw_event =~ /^Ovale : (.+)/
    {
    create_var('Oval');
    set_var('Oval', $1);
    drop();
    }

    sikora->parse_csv();
    delete($EventReceivedTime);
    delete($SourceModuleName);
    delete($SourceModuleType);

    $macrosection = get_var('macrosection');
    $Nominal = get_var('Nominal');
    $PlusTol = get_var('PlusTol');
    $Oval = get_var('Oval');

    to_json();

    </Exec>
    </Input>

    <Output out>
    Module om_file
    CreateDir TRUE
    File "C:\\Users\\temp"
    </Output>

    <Route 1>
    Path sikora_logs => out
    </Route>

    This is the input file (CSV):

    [ANALYSIS]
    TUBE=1
    CABLE=0
    [OTHERS]
    RECIPE=Tab73e112 DN16 2SCR16_1E4198
    VALUE_TAB_NUMBER=3
    [D0]
    Nominal=21.40
    PlusTol=0.40
    MinusTol=0.40
    PlusWarnTol=0.40
    MinusWarnTol=0.40
    Oval=0.00e+000
    [D2]
    Nominal=23.80
    PlusTol=0.10
    MinusTol=0.10
    PlusWarnTol=0.05
    MinusWarnTol=0.05
    Oval=0.00e+000
    TrAvMax=29.38
    TrAvMin=28.66
    [SECTOR]
    TwistLength=0.000000e+000
    [XRAY]
    CableType=60
    Nominal=1.20
    PlusTol=0.30
    MinusTol=0.20
    TrAvMin=0.81
    TrAvMax=1.32
    PlusWarnTol=0.00e+000
    MinusWarnTol=0.00e+000
    [XRAYE]
    PlusTol=0.10
    [XRAYC]
    PlusTol=0.20
    PlusWarnTol=0.00e+000

    When I run the program I still get errors. These are the logs:

    2019-09-18 11:43:15 INFO nxlog-ce-2.10.2150 started
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Too many fields in CSV input, expected 3, got 4 in input 'PK'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ua¡¨¡0tÞÖ‚e|ž»{÷»Ðf"ÕÎìO†¬Ÿ3–¾çé½Óž‰‡Rò§FÒÐQñ¼ ù'
    2019-09-18 11:43:15 ERROR failed to open C:\Users\temp; Access is denied.
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input '}ëC);4ÐЗþ˜…„6ý=3’­™õ8›Ë¦´%kX¤ÑwÎ|sÎÑ7]¼t/¦ÎN9aIÛ­^¨¸NÆlB’YÛ½5”š®ÃJ&ˆ²·Ý%æî¥Ý?ºˆvD„cì€}ÂwPۍ„˜ï”Ë| ͈_`sœÀ³)Kc$à6•'):¿1-×*• #’¸N‚bp{}:%cì ¥Kwwå¼Oá6\6Œiz ]cÃBa'‡U‰àKÒÔ9B´íB?v<Ä÷„ëPÄ<h»õç–w/–ÑNnDÅ[Ín þr»Ü`rXS}¦³ÑºSÏó½ ³ö¯'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Too many fields in CSV input, expected 3, got 4 in input '*c^^Ü5¸DéBKÏW£Ø'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'Ÿi…õ©2{°´d H÷A49vFt‘ÞDPb~£*8!ŽªY4'Î3×BVÔß©‰)—]ý@QÕPÖŽè<BùŒ¢‹yW"º¦£îÖ1Ðîò1C@7C8šÉ ögݳ§j9M4‹9ÓP9kÚÅôýMò«b5XeÒ­¶;¼ÐºÖJë P­³Ä³îkLµ¢3ƒšd¼)ÃR³óV“Ú9.´H[⶞#¬‘xÛ™ìNW­œ VëJUøêÇþm‚î‚xôàxAW©„/)‚E_vŽœÉ¼"÷D¾F„+g‘’¶{¿âw¼°æ‡¥JÓï—¼ºW)5ýN½ÔñýzµïW+½níL,"Š«~öÑe'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ƒn©„Þ úÍÖàë)°×©‡^Ðo–‚j–¼ "é7[¥†W«u¼F§Ù÷:òe Œ<“<^Åk÷o'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ÞZ'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input '$nOº¦']ªö«1†¸ñ ²—+´êïìÚxg$Kâ°UtwØaŸyvföÙݱ÷F?n³Tû—URäcݺ2u-Σb‘䫱þÛCÐêZEÂ|¦Eõ]\é?N¾ÿnT‘]}Œc¢D^õGBÖ׆QEqVWÅ:Îá7Ë¢ÌB—åʨÖe.*Ú(K;Û4ûF&¹^#\g‘ H–O›u/*²uH’y’&dÇ°t-‹®?­ò¢ ç)PÝZni[«_Ú{ ìց‘,‰Ê¢*–ä'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Too many fields in CSV input, expected 3, got 4 in input '@b¹L¢ø«oøFq$€}’å¦]w|2Z9©´¨ØäÜÎf¯Ÿòâ¯< ¿ƒ»zýµÉ¨ú[û¦pÇҍÉ(*Ò¢Ôx:Ëîäa×ߘ…i2/úµe˜%鮾mÓ,@Í÷²ÜEo”ˆ2;Ãþ8ôÎA´ÏÉꑼޫðÏ#½šÓ¾ï=èIZ|£[¬/B´Ž÷® [‡™Ñ©­cX®æc=`4[¦y4l¦acÌŸ™`ïbÆúÞÅzæN0è´gB.¢üh\I;:A—®|Å`p3¸½˜;»7vªw_jÐ×­¬‡y2èÏ%òäÂSZGc3hSh’¦íŒî8tò†;“¬VH\æ\hÍç‡Ý¦îVÔ­Fý½W¾½*Ýe³éK®AU¤É‚²XÍØ‚¡ö³þ]0»cv3Y'@ƒ`6ø'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input '°2Þ3€å €Î Yñžq`Qak„ r€›ûe4æ¨Éìå)ù…._d¸[Ho!ѹ›áâ£Ü|JbUI '
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ªGB•Dbª4…B•D"'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ªGB•DbêW‘ŽúU¤ó?XE:,‘.›ÖETT?¥‡!Î/ŸjÛe[Geø€ˆê³bu¶µ¯ÑócÝ6ÐzÚMA=ü_CY47I'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input ' Æ:ÿÌ-Ã(†—8@n>WðÊü«mÊd¬ÿs7ø·wÝšÓaÏub¯ç{ÓÛžçΦ··oÚæì_t~ý§×ÙY{Ø>Yîu•Â÷²élCþ+¿7ÖÑEMŸ½´1wßî›7žeöÇ´zn?ö†}ÇëžeßöÝéxˆ»÷Æóò¦aYûóò[Ë»&I§I¾Õ>Bø. ._脱„Áÿ#ƒÉ'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Too many fields in CSV input, expected 3, got 4 in input 'V3‡D¼<±?Á³zxkêàUjS©v²( Ùf*¯ÚÝ:üóûùfÆŠ6µjå:|—&|Ø|ý²:*½7¥”6'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input '¥ìéû¦±›Ýü'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'fr¦-<ZmÀz’r93%ÞyorBہ¤.Ä­¶’úp´;1”½ÓH–$s"ÁSN=%=02‘œMHóaÛÀ$(ïH§äÛëÁJ÷ëÀ œ9¥ð:ãž³9Åɽwb2v]w³!FÈŸ’×õÃÓP5ªß\œåÌõÚV+*…^ Úö¢cÕ93ôËl©óë°÷­'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'X¦­Œ'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ƒbPä,ZÔ€•dL©‘E„"ÎD°ûõ¥$Ä–›¬º›ÇÅåá%Õõ®uY‘lð…X.r‘¡7¡²~[ˆÇòþü›ÈˆÁWà‚ÇBì‘ĵ>û¢Ö1tÙ"eÉÂS!æn%%™[ EZû´©ClS·2Ôµ5xÌ[‹žåEž_IÜ1ú'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ÞZ'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'X¦­Œ'
    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 0 in input ''
    2019-09-18 11:43:38 WARNING stopping nxlog service
    2019-09-18 11:43:38 WARNING nxlog-ce received a termination request signal, exiting...

    I'm trying to understand what's the cause of these errors. I was thinking that maybe the variable "Oval" in the CSV file has too many digits.
    Any help would be appreciated.
    Thanks

    September 18, 2019 - 1:54pm
  • Zhengshi's picture
    (NXLog)

    2019-09-18 11:43:15 ERROR procedure 'parse_csv' failed at line 61, character 28 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 3, got 1 in input 'ua¡¨¡0tÞÖ‚e|ž»{÷»Ðf"ÕÎìO†¬Ÿ3–¾çé½Óž‰‡Rò§FÒÐQñ¼ù'
    2019-09-18 11:43:15 ERROR failed to open C:\Users\temp; Access is denied.
    

    These are the kinds of errors you are getting. The second one is easiest. You can not write to your Output due to filesystem permissions.

    The first one indicates possibly two issues. First is you may need to use xm_charconv to convert to a character set that is more common. utf-8 for instance.
    https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_charconv
    The other bit is you are using xm_csv that declares there are 3 fields, and very commonly you are not getting 3 fields. It is detecting 1 or 4 fields in the errors. This is likely because of the structure of the file.

    It may be useful to post the actual source file so we can see it. It would also be useful to know how the file structure works. Is the entire file expected to be one event? Is each section preceded with brackets supposed to be an event? i.e. [ANALYSIS],[OTHERS],[D0] are those three their own events and should be treated as such, or are they all part of one event?

    September 18, 2019 - 5:45pm
  • IoT_fra_87's picture

    I thought of permission issues on my output folder. I'll try to edit the permissions. So do you think I have to use this module (xm_charconv)
    to convert characters to utf-8 ? Do I have to use it in place of the xm_csv module or do I have to add it to my script ?
    The original one is a .set file. Then because of the impossibility of nxlog to work with a .set file (there is no xm_set module available) I edited the source file to a .csv. Yes the entire file is expected to be one event. [ANALYSIS],[OTHERS],[D0]... etc are all part of the same event. How can I upload the source file ? I don't see the upload button. Anyway I'll copy the source file here:

    [ANALYSIS]
    TUBE=1
    CABLE=0
    [OTHERS]
    RECIPE=Tab73e112 DN16 2SCR16_1E4198
    VALUE_TAB_NUMBER=3
    [D0]
    Nominal=21.40
    PlusTol=0.40
    MinusTol=0.40
    PlusWarnTol=0.40
    MinusWarnTol=0.40
    Oval=0.00e+000
    [D2]
    Nominal=23.80
    PlusTol=0.10
    MinusTol=0.10
    PlusWarnTol=0.05
    MinusWarnTol=0.05
    Oval=0.00e+000
    TrAvMax=29.38
    TrAvMin=28.66
    [SECTOR]
    TwistLength=0.000000e+000
    [XRAY]
    CableType=60
    Nominal=1.20
    PlusTol=0.30
    MinusTol=0.20
    TrAvMin=0.81
    TrAvMax=1.32
    PlusWarnTol=0.00e+000
    MinusWarnTol=0.00e+000
    [XRAYE]
    PlusTol=0.10
    [XRAYC]
    PlusTol=0.20
    PlusWarnTol=0.00e+000

    Thank you again Zhengshi!

    September 19, 2019 - 10:53am
  • Zhengshi's picture
    (NXLog)

    It is hard to say about the Character conversion without the actual file. It is also hard to say about the format, etc because the original code looks like Italian words filtering in the REGEX.

    The original one is a .set file. Then because of the impossibility of nxlog to work with a .set file (there is no xm_set module available) I edited the source file to a .csv.

    If the file is a standard ascii or text file, we can work with it no problem. Again, it is hard to know without the source file.

    I don't really know what format you are expecting out, but since the entire file is one record/event, then you will need to treat the file as a multiline event.

    I copied what you gave as the source into a file and wrote up a quick config to process it given the requirement of only needing the Nominal, PlusTol, and Oval values from the sections.
    I am using xm_multiline setting the HeaderLine to the fist value of [ANALYSIS] and running an Exec against every line. If the line doesnt start with Nominal, PlusTol, Oval, or [ I am dropping it and adding what is left to a syslog event.

    <Extension syslog>
        Module xm_syslog
    </Extension>
    <Extension multi>
        Module xm_multiline
        HeaderLine /^\[ANALYSIS\]/
        Exec if $raw_event !~ /^(\[|Nominal|PlusTol|Oval)/ drop();
    </Extension>
    
    <Input sikora_logs>
        Module im_file
        File 'set.log'
        InputType multi
        ReadFromLast False
        Recursive False
        SavePos False
        <Exec>
            to_syslog_bsd();
        </Exec>
    </Input>
    

    Output:

    <13>Sep 19 05:14:11 node1 [ANALYSIS] [OTHERS] [D0] Nominal=21.40 PlusTol=0.40 Oval=0.00e+000 [D2] Nominal=23.80 PlusTol=0.10 Oval=0.00e+000 [SECTOR] [XRAY] Nominal=1.20 PlusTol=0.30 [XRAYE] PlusTol=0.10 [XRAYC] PlusTol=0.20
    

    These tools should help get the data more into a format that you are looking for.

    September 20, 2019 - 12:17am