~~Please confirm about HPUX server how to install NXLOG server or client.

Hello, I'm catching up logs, but when they arrived to my Graylog, they take the actual date and not the old date.  I saw my logs and they don't have format date. They're csv, and I have this field : FW_date_time:10Nov2016;21:40:10 and I need to assign it on the timestamp, but I don't kno how. Could you help me please to convert this on a good format? I tried this (I have a field called: FW_date_time on my source csv file): Exec          FW->parse_csv(); Exec            $EventReceivedTime =strptime($FW_date_time, '%Y %b %d %T'); Exec              $EventTime =strptime($FW_date_time, '%Y %b %d %T'); But it didn't work. Thank you very much for your help. Cordially,   Ana      

I'm having some issues sending event logs from a windows 2012 R2 client using nxlog ce agent to a linux syslog-ng server.  The tcp connection appears to be getting dropped by syslog-ng due to an invalid frame header. syslog-ng says: 2016-11-22T09:10:35+00:00 server syslog-ng[8317]: Invalid frame header; header='' 2016-11-22T09:10:35+00:00 server syslog-ng[8317]: Syslog connection closed; fd='8', client='AF_INET(ip.of.nxlog.client:55473)', local='AF_INET(0.0.0.0:6514)' 2016-11-22T09:10:36+00:00 server syslog-ng[8317]: Syslog connection accepted; fd='8', client='AF_INET(ip.of.nxlog.client:55474)', local='AF_INET(0.0.0.0:6514)' nxlog says: 2016-11-22 15:24:27 ERROR om_tcp send failed; An established connection was aborted by the software in your host machine.   2016-11-22 15:24:28 INFO connecting to ip.of.syslog-ng.server:6514 2016-11-22 15:24:28 INFO reconnecting in 1 seconds I've tried several variations in the nxlog.conf file to work around this issue.  Here is my nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_syslog </Extension> <Input in>     Module      im_msvistalog # For windows 2003 and earlier use the following: #   Module      im_mseventlog     ReadFromLast TRUE     <QueryXML>        <QueryList>          <Query Id='1'>             <Select Path='Application'>*</Select>             <Select Path='Security'>*</Select>             <Select Path='System'>*</Select>          </Query>        </QueryList>    </QueryXML> </Input> <Processor eventlog_transformer>   Module pm_transformer   Exec $Hostname = hostname();  OutputFormat syslog_rfc5424  </Processor> <Output out>    Module om_tcp    Host        ip.of.syslog-ng.server    Port 6514        Exec $raw_event = replace($raw_event, "\r\n", " ");     Exec $raw_event = replace($raw_event, "\t", " ");     #Exec to_syslog_ietf();     #Exec        to_syslog_bsd(); </Output> <Route 1>     Path        in => eventlog_transformer => out </Route> Here is the syslog-ng configuration: @version:3.5 @include "scl.conf" # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # Note: it also sources additional configuration files (*.conf) #       located in /etc/syslog-ng/conf.d/ #Options # ## Warnings # # options {     create_dirs(yes);     dir_perm(0755);     perm(0644);     log_msg_size(65536);     log_fifo_size(10000);     threaded(yes);     ts_format(iso);     keep_hostname(no);     use_dns(no);     dns_cache(no);     use_fqdn(no);     flush_lines(100);     stats_freq(60);     mark_freq(36400); }; #Sources source s_sys {     system();     internal(); }; source s_network {     syslog( port(6514) flags(syslog-protocol)  transport("tcp") keep-alive(yes) so-keepalive(yes) so_rcvbuf(1073741824) log-fetch-limit(100) log-iw-size(100) max-connections(5000));     syslog( port(514) transport("udp") so_rcvbuf(1073741824)); }; #Destinations #destination d_all       { file("/var/log/logs/all_logs/$HOST/$HOST-$YEAR-$MONTH-$DAY-all_logs.log" create_dirs(yes)); }; destination d_all       { file("/var/log/logs/all_logs/$HOST/$HOST.log" create_dirs(yes)); }; #Filters #Logs #   { source(s_sys); filter(f_kernel); destination(d_cons); }; log { source(s_sys); destination(d_all); }; log { source(s_network); destination(d_all); flags(flow-control); }; # Source additional configuration files (.conf extension only) @include "/etc/syslog-ng/conf.d/*.conf"   I can get things to work using udp, but can't get it to work over tcp.  Has anyone else had any success getting nxlog to send events from windows to a syslog-ng server over tcp?

Hello,   I have NXLOG installed, and use UDP to receive and sent the servers logs. However, when I run the netstat command I found that there are packets on error and other received through unknown ports. See the lines shown :   UDP:  123234944 packets received 223432 packets to unknown port received. 523455 packet receive errors 2111 packets sent Why does it happen? I didn't configure any other port on my NXLOG and the firewall rules only accept 514 port to communicate with my NXLOG server. How could I verify what is this? Is it normal?   Thank you very much for your help and answer   Cordially,   Ana

After a bit of trial and and lots of reading, I managed to get Graylog2 working like a charm I'm using NXLOG to send the logs to Graylog via GELF UDP Right now I'm just testing and trying diffrent things. Right now I';m testing with just 4 servers (have close to 100) has consumed a fair bit of space So to evaluate usage, I figured I'd just send the security logs (these are all Windows Servers) Original working configuration __________________________________________________________ define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Input in> # Use ’im_mseventlog’ for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="System">*</Select>\ <Select Path="Application">*</Select>\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> </Input> <Output out> Module om_udp Host 10.60.10.62 Port 12201 OutputType GELF </Output> <Route r> Path in => out </Route> _______________________________________________________ Now I put a REM statement at the beginning of the file # Just capturing security logs The service won't start. If I rem out Application and System path, it won't start. Any suggestions?

  I am using nxlog-ce clients in windows machines, with om_ssl.  NXlog seems to keep the connection open, but does not send anything, except for the bursts for every Poll Interval.  The load balancer seems confused, and tries to send every second burst to a different server.  Is there any option to tell nxlog-ce that it should make a new connection for each burst, so that the load balancer can easily balance the load for different clients to different log servers?

Hello,   I have 2 infrastructures installed one on Amazon and another on premises. On Amazon every thing is OK, but when I send logs on premises I have a 5 to 7 minutes delayed and I loss almost 30% of the data, I activated the buffer option and I pass through it before sending logs to my on premise server but it doesn't work Somebody know how to correct that?   Thank you for your help and answer   Ana

We are pushing logs from file with the im_file module to logstash and then to elasticsearch. However, some of these logs only have second accuracy, and therefore not returned in order by elasticsearch when sorted by just the time. To get around this problem, we would like to add the position of the log event to a field that we store (for example beginning line number or byte offset within the source file). Is there a way I can add a separate field for this in the events pushed by nxlog, either in the input or output configuration?

I have the following regexp: ^\s*(\d{2}:\d{2}:\d{2}\.\d{4})\s+(\[\S+\])\s+([\s\S]*)$ and a line from the log that I am trying to parse (there are some spaces at the beginning of the line):  13:33:00.1205 [-] Persisting VDOM path / trying to process a file using pm_pattern module with LogLevel DEBUG this is what I get: 2016-11-16 13:38:00 DEBUG regexp '^\s*(\d{2}:\d{2}:\d{2}\.\d{4})\s+(\[S+\])s+([\s\S]*)$' did not match against '  13:38:00.1857 [-] Persisting VDOM path /' Part of the xml file used to process the input: <pattern>     <id>1</id>     <name>log_line</name>     <description>Regular log line</description>     <matchfield>         <name>raw_event</name>         <type>regexp</type>         <value>^\s*(\d{2}:\d{2}:\d{2}\.\d{4})\s+(\[S+\])s+([\s\S]*)$</value>                                 <capturedfield>             <name>EventTime</name>             <type>string</type>         </capturedfield>         <capturedfield>             <name>SourceName</name>             <type>string</type>         </capturedfield>         <capturedfield>             <name>Message</name>             <type>string</type>         </capturedfield>     </matchfield> </pattern> What is wrong with my expression?   

I am wanting to see if this is possiable to put in a line for Input event log.     <Input eventlog>    Module im_msvistalog    SavePos FALSE    ReadFromLast TRUE    Query  <QueryList>\             <Query Id="0">\              <Select Path="System">*[System[(EventID=22 or EventID=1076 or EventID=6005 or EventID=6006)]] and *[System/Level=2]</Select>\              </Query>\ </QueryList> </Input>     Can i have this line in the select path along with just detecting event logs? I also want to monitor all Error messages.  *[System/Level=2]

Hello.  The case is: I am creating Stream with rules using API. Question is: Is there a way to set the input for a stream using GrayLog API?   Thanks.

Hi, I want to store my logs in .evtx file in windows. I tried following configuration. <Output out2>         Module      om_file   File     '%ROOT%\tmp\test.evtx' </Output> This created evtx file but it was also opening with notepad, wordpad,etc. For security purpose, I want to make it open with MS EventViewer API only. Is this possible using nxlog om_file module? Is there any plugin for nxlog to store data in .evtx files?

Hi, I am a new at this and would like to know the command line argument that will let me specify what config file to use. I need to have this done during installation and avoid changing the file post installation. Any help would be greatly appreciated.   Thanks.

Hi,   I am totally new to nxlog and trying to set up a config file to watch a file on my local hard drive and send it to Loggly whenever it changes.  My conf file is below.  I copied this file from the loggly documentation and tried modifiying it as per their instructions so there may be some unnecessary lines in it. The file I want to watch and send is in the C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\ folder.  The file name is JobProcessor.log.  When I uncomment the "#Exec file_write("C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\JobProcessor.log",  $raw_event);" line, it does send it out so I know the connection is working.  But I have not been able to 'trigger' the file to process whenever there is a change to it.  Any advice is appreciated.  Thanks.   #define ROOT_STRING C:\\Program Files\\nxlog define ROOT C:\\Program Files (x86)\\nxlog define ROOT_STRING C:\\Program Files (x86)\\nxlog define CERTDIR %ROOT%\\cert   Moduledir %ROOT%\\modules CacheDir %ROOT%\\data Pidfile %ROOT%\\data\\nxlog.pid SpoolDir %ROOT%\\data LogFile %ROOT%\\data\\nxlog.log   # Include fileop while debugging, also enable in the output module below <Extension fileop>     Module      xm_fileop </Extension>   <Extension json>     Module      xm_json </Extension>   <Extension syslog>     Module xm_syslog </Extension>   <Input internal>     Module im_internal     Exec  $Message = to_json();  </Input> ##Module to watch a file <Input file_watch_1>     Module im_file     File "C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\"     Exec   $SourceName = 'JobProcessor.log';      SavePos TRUE     ##include the message and add meta data     Exec $Message = $raw_event;     Exec to_syslog_ietf(); </Input>   <Output out>     Module om_tcp     Host logs-01.loggly.com     Port 514       Exec to_syslog_ietf();     Exec $raw_event; #Use the following line for debugging (uncomment the fileop extension above as well)     #Exec file_write("C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\JobProcessor.log",  $raw_event); </Output>   <Route 1>     #Path internal, eventlog => out     Path    in => out </Route>

Hi! I have a problem using module im_odbc. I use sql query to receive some data from sql server. For example, query looks like "SELECT [Name] From [MyTable] WHERE [Id] > ?". In some moment, the "?" has stored incorrect value (I don't know how, but it's bigger then MAX(Id) value in my table) and sinse that moment, query has not return any data.  Please, help me reset value of "?" (nxlog service restart not help). And, maybe, you can give me some recommendations, how to avoid this problem in future?   UPD: I reset value of "?" by renaming input name in config file. The first query after input renaming was executed with "0" as value of "?" and return all rows from table. But next time query executed with "?"-value "161027120023", wich bigger than max value of "Id" column in my table (max value was 161027115813 at that time). What am i doing wrong? <Input odbc1>     Module                im_odbc     ConnectionString    DSN=im_odbc_test;database=MyDB;     SQL                    SELECT [Name] From [MyTable] WHERE [Id] > ?;     PollInterval        120     SavePos                TRUE     IdIsTimestamp        FALSE </Input>

Hello, This is jsut a quick question on the following lines of a config for Nxlog. Basically is the code suppressing all security events with the exception of those event IDs with in the brackets or are the event IDs within the brackets being suppress? # Windows Event Log <Input eventlog> # Uncomment im_msvistalog for Windows Vista/2008 and later    Module im_msvistalog Query    <QueryList>\                    <Query Id="0">\                      <Select Path="Security">*</Select>\                      <Suppress Path="Security">*[System[(EventID=4624 or EventID=4776 or EventID=4634 or EventID=4672 or EventID=4688 or EventID=4662 or EventID=5159) or EventID=5136 or EventID=4932 or EventID=4933]]</Suppress>\                    </Query>\         </QueryList>    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json(); </Input>

Does it work to send IIS logs in snare format ??? when i use the "Exec to_syslog_snare();" option in output, it sends logs like they are MSWinEvents...   <13>Oct 21 06:26:36 SRV-00-20-21 MSWinEventLog    1    N/A    17    Fri Oct 21 06:26:36 2016    N/A    N/A    N/A    N/A    N/A    N/A    N/A        N/A    N/A

I was reading the Enterprise Edition documentation on Check Point OPSEC LEA. Can this be done on a Windows server or do I need to use a Linux server?

I have    <Extension w3c>     Module            xm_csv     Fields            $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $time-taken     FieldTypes        string, string, string, string, string, string, integer, string, string, string, string, string, integer, integer, integer, integer     Delimiter        ' '     QuoteChar        '"'     EscapeControl    FALSE     UndefValue        - </Extension>   And i have    <Input iis>     Module            im_file     File            "C:\\inetpub\\logs\\LogFiles\\u_ex*.log"     SavePos            TRUE     Exec \                     if $raw_event =~ /^#/ drop(); \                     else \                         { \                             w3c->parse_csv(); \                             $EventTime = parsedate($date + " " + $time); \                             $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); \                             $SourceName = "IIS"; \                         } </Input>   When im trying to add something like if $raw_event =~ /^#/ or $sc-status == 101 drop(); \ I am getting error in log like that:  couldn't parse statement at line 91, character 46 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected -, expecting ( I have tried to use similar options, but it doesnt works.  What im doing wrong, help me please. Thanks.  

I'm attempting to use nxlog to pull in audit files from ModSecurity the root of the log files reside at /var/log/modsec_audit/. The actual log files are two folders lower, as modsecurity creates the folders that are timestamped by day and then minute. Inside my nxlog input config, I have my logfile path as /var/log/modsec_audit/*, and I have enabled recursive file lookup, but to this point, no modsecurity logs are being picked up. The same sidecar instance is grabbing apache logs perfectly fine, however. I feel like the issue is related to the directory structure, but I'm not seeing an issue with my config. Any thoughts?   <Input 579d7bd625b5bc05cc7f1614>         Module im_file         File "/var/log/modsec_audit/*"         PollInterval 5         SavePos True         ReadFromLast True         Recursive True         RenameCheck False         Exec $FileName = file_name(); # Send file name with each message </Input>