1 Aug 2012 00:06
Re: Quick question or two about Argus
Carter Bullard <carter <at> qosient.com>
2012-07-31 22:06:52 GMT
2012-07-31 22:06:52 GMT
It is very difficult to comment on any of your possible bugs, as you aren't providing enough information
to understanding where the problem is. For example, without seeing your rasplit() command,
I can't comment on what the data differences could be. The tools aren't bad, but one would
have to imagine that with all the problems you're seeing that something is terribly wrong.
What version of the tools are you using ? Looks like you're doing some things that are
deprecated, such as printing " avgdur ".
On Jul 31, 2012, at 3:42 PM, Craig Merchant <cmerchant <at> responsys.com> wrote:
I ran the following command against a directory with 10 minute raw files produced by rasplit:racluster -R /space/argus/rasplit/07/29 -m saddr daddr dport -t 1343587314-1343587914 -w /space/argus/testing/clusterrecursive.raw - src net 10.0.0.0/8 and dst net not 10.0.0.0/8 or 172.16.0.0/15 and dst port not 53The output file was 72597624.I ran this search against a single file spanning several days:racluster -r /space/argus/radium/radium.out -m saddr daddr dport -t 1343587314-1343587914 -w /space/argus/testing/clusterbigfile.raw - src net 10.0.0.0/8 and dst net not 10.0.0.0/8 or 172.16.0.0/15 and dst port not 53The output file was 7068656, almost ten times smaller.I’m guessing that we’ve got a lot of flows that rasplit is segmenting, so more records appear? If I try to run rahisto against a day’s worth of 10 minute files, I get a segmentation fault. If I specify “-t 1343587314-1343587914”, it doesn’t produce any results.I also tried to sort the output from the single big file by avgdur to see how long our flows tend to be:rasort -r clusterbigfile.raw -m avgdur12:41:30.279031 sort syntax error. 'avgdur' not supportedThx.CFrom: Carter Bullard [mailto:carter <at> qosient.com]
Sent: Tuesday, July 31, 2012 6:44 AM
To: Craig Merchant
Cc: Argus; Steve Slater; Vishnu Shankar; Paul DeBone
Subject: Re: Quick question or two about Argus
Hey Craig,My bad, I wasn't thinking last night. rabins() is not the tool for your situation. rabins() is a time series tool when you bin based on time, and netflow streams are pretty with regard to the out of order nature of the data and massive hold times ( unless of course you have a lot of time and a lot of memory ).
The -B option is incredibly important to rabins(), without it, rabins will process data until EOF, so you won't get any output until the file is over or you exit. -B defines the hold buffer for bin formation, data whose timestamps are before the ( current time + the hold buffer time ) are rejected when reading real time data.
As a result, you generally can't use rabins on netflow data, because of the huge hold times and out of order nature of the output.
Better to write netflow into a file using rasplit(), and process the netflow data later, after all the data for a given time period has arrived.
With regard to filenames and rasplit(). Your debug information is saying you have toooooo many ' % ' in your specification, and putting addresses in the output filename may not scale for you. The total number of output files is limited by the kernel, not by the application. The error is saying you saw more than x addresses during he last 60 seconds. I see > 1000 per minute all the time. Your kernel may be tuned as low as 64 open files.
On Jul 30, 2012, at 10:48 PM, Craig Merchant <cmerchant <at> responsys.com> wrote:Thanks for the quick response!rabinsI just tried the following test with rabins:rabins -S localhost -M time 1m -w argus%M &No bins appeared after several minutes, though when I killed the process, argus%M was created. It didn’t translate the %M into the decimal minute though. When connected to a stream, when does rabins write out a bin? I see the rabins process steadily consuming more memory, but nothing seems to get written to disk.I tried the –B 15 switch, but files aren’t being written out either. I should say that we’re using radium to listen for incoming Cisco netflow data, so maybe rabins isn’t seeing the keep alives that argusd sends out?ratopInstalling ncurses-devel solved the compilation problem.rasplitI get the following error when running rasplit and using the $saddr field in the path or filename: ArgusWriteNewLogfile(/space/argus/rasplit/tcp/19/argus.%%T.0.000000.10.raw, 0x2e8ea5e0) fopen Too many open files$proto gets translated correctly, but not $saddr or $daddrThe command was:rasplit -S localhost -M time 1m -w /space/argus/rasplit/\$proto/%H/argus.\$saddr.%M.raw &What’s the limit on the amount of open files that rasplit can handle?A couple more questions…Does argus use any kind of indexing of the data files so that when client applications search recursively, they know they don’t have to scan an entire file because a particular saddr or dport doesn’t exist in the data?Is it possible to make rasplit output data by netblock or a label?Thanks!CraigFrom: Carter Bullard [mailto:carter <at> qosient.com]
Sent: Monday, July 30, 2012 5:32 PM
To: Craig Merchant
Subject: Re: Quick question or two about ArgusHey Craig,Answers in line.CarterOn Jul 30, 2012, at 7:38 PM, Craig Merchant <cmerchant <at> responsys.com> wrote:
Hi. My name is Craig Merchant. I’m in charge of implementing a security monitoring solution using Argus. I’ve got a few questions for you if you’ve got a second…1. The documentation says rabins supports time prefixes in the output file name (%Y,%m, etc.). In my experience on CentOS, it writes those as a literal string.Not sure how you're calling rabins, such that the strftime() directives in the output file are not being done.Just in case, you do need to indicate the time period, so that the logic knows to do the substitution.rabins -M time 1d -w argus_%Y_%m_%d2. Is there any benefit to having rabins forward the data from the sensor to radium in a multi-sensor environment?The preferred strategy would be to do the rabins() at the end of the data flow. So it could beargus -> radium -> .... -> rabins
3. Ratop complains that it wasn’t compiled with ncurses support (ncurses-5.5-24.20060715 is installed). –M nocurses works, but I get the impression that’s not very efficient… I tried the “make clobber” step you recommended, but no luck.When you run ./configure, the output will state if curses was found in the search paths. You can searchthe file ./include/argus-config.h to see what its saying about CURSES. If its defined, then you won't getthe error message. ratop without the curses window isn't really very gratifying, so try to figure out why./configure isn't finding ncurses.
4. What other ra commands/fields/switches does ratop support?ratop is an aggregator, so it does everything that racluster() can do, it can sort, so it does all that rasort() does, and it can print every field that ra* programs support, and a few others. Type ' :h ' and you'll get a help screen. If that doesn't do it for you, send some more email, and we'll add to that screen.
5. When I tried following the video for ratop, I see you specify “-M rmon”. When I tried that, I get: ArgusClientInit: ArgusNewAggregator errorWhen you report a bug, we need at least how it was called, to see what the issues might be.
6. Using rasplit and specifying $saddr in the output file path ends up with a directory named %T.00000. Same with $daddrThis error is sometimes caused by the shell, which wants to interpret the ' $ ' as special. Escape the ' $ ' using ' \$ '.
Thanks!Hopefully this is helpful!!