Peter Bates | 29 Mar 14:54 2012
Picon
Picon

Suricata's http-log


Hello all

Suricata's inbuilt 'http log' is quite useful for adding context to
alerts and reducing the need for running additional software.

As far as I can see, this file just grows and grows until restart.

Would it be possible to add one of the following:

1) Allowing the rotation of the file on SIGHUP
2) Creating a new file when the current one is moved away (as per Argus)
3) Adding a filesize option to auto-rotate when a limit is reached

I'm trying to avoid just using logrotate to move the file and then
restarting Suricata to pick up the change - if at all possible.

--

-- 
Peter Bates
Senior Computer Security Officer    Phone: +44(0)2076792049
Information Services Division	    Internal Ext: 32049
University College London
London WC1E 6BT
Martin Holste | 29 Mar 15:46 2012
Picon

Re: Suricata's http-log

One other thing that would be nice and would be easier: can it log to
the syslog facility?  Then you could have your system's syslog handle
rotation, etc.

On Thu, Mar 29, 2012 at 7:54 AM, Peter Bates <peter.bates@...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Hello all
>
> Suricata's inbuilt 'http log' is quite useful for adding context to
> alerts and reducing the need for running additional software.
>
> As far as I can see, this file just grows and grows until restart.
>
> Would it be possible to add one of the following:
>
> 1) Allowing the rotation of the file on SIGHUP
> 2) Creating a new file when the current one is moved away (as per Argus)
> 3) Adding a filesize option to auto-rotate when a limit is reached
>
> I'm trying to avoid just using logrotate to move the file and then
> restarting Suricata to pick up the change - if at all possible.
>
> - --
> Peter Bates
> Senior Computer Security Officer    Phone: +44(0)2076792049
> Information Services Division       Internal Ext: 32049
> University College London
(Continue reading)

Victor Julien | 29 Mar 16:17 2012
Picon

Re: Suricata's http-log


On 03/29/2012 03:46 PM, Martin Holste wrote:
> One other thing that would be nice and would be easier: can it log
> to the syslog facility?  Then you could have your system's syslog
> handle rotation, etc.
> 
> On Thu, Mar 29, 2012 at 7:54 AM, Peter Bates
> <peter.bates@...> wrote:
> 
> Hello all
> 
> Suricata's inbuilt 'http log' is quite useful for adding context
> to alerts and reducing the need for running additional software.
> 
> As far as I can see, this file just grows and grows until restart.
> 
> Would it be possible to add one of the following:
> 
> 1) Allowing the rotation of the file on SIGHUP 2) Creating a new
> file when the current one is moved away (as per Argus) 3) Adding a
> filesize option to auto-rotate when a limit is reached
> 
> I'm trying to avoid just using logrotate to move the file and then 
> restarting Suricata to pick up the change - if at all possible.

Shouldn't be hard to do. The output API for those line based logs like
http.log, fast.log, etc already supports unix socket, and I think
adding syslog shouldn't be very hard. Might be a nice project for
someone that wants to get familiar with our code base and dev procedures.

(Continue reading)

Victor Julien | 29 Mar 16:14 2012
Picon

Re: Suricata's http-log


On 03/29/2012 02:54 PM, Peter Bates wrote:
> 
> Hello all
> 
> Suricata's inbuilt 'http log' is quite useful for adding context
> to alerts and reducing the need for running additional software.
> 
> As far as I can see, this file just grows and grows until restart.
> 
> Would it be possible to add one of the following:
> 
> 1) Allowing the rotation of the file on SIGHUP 2) Creating a new
> file when the current one is moved away (as per Argus) 3) Adding a
> filesize option to auto-rotate when a limit is reached
> 
> I'm trying to avoid just using logrotate to move the file and then 
> restarting Suricata to pick up the change - if at all possible.
> 

You can use the trick described here:
https://redmine.openinfosecfoundation.org/issues/265#note-4

--

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

(Continue reading)

Peter Bates | 30 Mar 15:04 2012
Picon
Picon

Re: Suricata's http-log


Hello again all

On 29/03/2012 15:14, Victor Julien wrote:
>> I'm trying to avoid just using logrotate to move the file and
>> then restarting Suricata to pick up the change - if at all
>> possible.
> 
> You can use the trick described here: 
> https://redmine.openinfosecfoundation.org/issues/265#note-4

Thanks for the advice - and also Martin's suggestion that syslog
support for http-log might be useful.

I've been running httpry up until recently - and generally manage a
logfile from that of around 700-800Mb an hour, dropping to 200-300Mb
at quiet times.

Just testing with the Suricata http-log I've ended up with a 7Mb
logfile from 1pm-2pm (BST).

Httpry does also log the HTTP responses so you could argue the log
should be double the size - but there seems a big difference here
between the two.

--

-- 
Peter Bates
Senior Computer Security Officer    Phone: +44(0)2076792049
Information Services Division	    Internal Ext: 32049
University College London
(Continue reading)

Peter Manev | 30 Mar 15:12 2012
Picon

Re: Suricata's http-log

Hi Peter,
 
Is there any way that you could compare the two logs by the ways of scripting/bashing ? - if Suri and httpry are running at the same time (maybe just 10 min time span)?
 
thanks

On Fri, Mar 30, 2012 at 3:04 PM, Peter Bates <peter.bates <at> ucl.ac.uk> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hello again all

On 29/03/2012 15:14, Victor Julien wrote:
>> I'm trying to avoid just using logrotate to move the file and
>> then restarting Suricata to pick up the change - if at all
>> possible.
>
> You can use the trick described here:
> https://redmine.openinfosecfoundation.org/issues/265#note-4

Thanks for the advice - and also Martin's suggestion that syslog
support for http-log might be useful.

I've been running httpry up until recently - and generally manage a
logfile from that of around 700-800Mb an hour, dropping to 200-300Mb
at quiet times.

Just testing with the Suricata http-log I've ended up with a 7Mb
logfile from 1pm-2pm (BST).

Httpry does also log the HTTP responses so you could argue the log
should be double the size - but there seems a big difference here
between the two.

- --
Peter Bates
Senior Computer Security Officer    Phone: +44(0)2076792049
Information Services Division       Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPda91AAoJELhVoVpEMS6RPbwH/1nXjmMEbDzE6CQhGAgfYb6c
ebsrxKam3owvkOL2A/LHGeo4Y0nfjQ622jwZPhUHwEMl0FGNf6L7BNq9g//HOqhi
NKZQFAhYa45J6Fk2DpPAp6KUYb/RLHA0z3OflJtzFn18jAK9QE9POuRMiYSoqo18
XWZoxs3OuVi+UOxuWb97GAOoScsRrC5mQ2EI4LdodC9rjqy0RqJDhPxOVauOss7B
e65jJBxVgCCM2SfnnBoKy4PJR2XO0i3UguU6CGILiKFjb0SVScIzTvpxOelCR7bA
/TXZd/rnfhGHKFAhrx38bnfDgDvjFyQF/GbJkAfX3Cu7aEXWa1L5oA0oepJi868=
=MopT
-----END PGP SIGNATURE-----

_______________________________________________
Oisf-users mailing list
Oisf-users-SIp8cRv7ZWaXGvOITfQg64YkZiVZrdSR2LY78lusg7I@public.gmane.org
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users



--
Regards,
Peter Manev

_______________________________________________
Oisf-users mailing list
Oisf-users@...
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Martin Holste | 30 Mar 15:56 2012
Picon

Re: Suricata's http-log

Use wc -l to count the log lines for both.  httpry should be double.

On Fri, Mar 30, 2012 at 8:12 AM, Peter Manev <petermanev@...> wrote:
> Hi Peter,
>
> Is there any way that you could compare the two logs by the ways of
> scripting/bashing ? - if Suri and httpry are running at the same time (maybe
> just 10 min time span)?
>
> thanks
>
> On Fri, Mar 30, 2012 at 3:04 PM, Peter Bates <peter.bates@...> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>
>> Hello again all
>>
>> On 29/03/2012 15:14, Victor Julien wrote:
>> >> I'm trying to avoid just using logrotate to move the file and
>> >> then restarting Suricata to pick up the change - if at all
>> >> possible.
>> >
>> > You can use the trick described here:
>> > https://redmine.openinfosecfoundation.org/issues/265#note-4
>>
>> Thanks for the advice - and also Martin's suggestion that syslog
>> support for http-log might be useful.
>>
>> I've been running httpry up until recently - and generally manage a
>> logfile from that of around 700-800Mb an hour, dropping to 200-300Mb
>> at quiet times.
>>
>> Just testing with the Suricata http-log I've ended up with a 7Mb
>> logfile from 1pm-2pm (BST).
>>
>> Httpry does also log the HTTP responses so you could argue the log
>> should be double the size - but there seems a big difference here
>> between the two.
>>
>> - --
>> Peter Bates
>> Senior Computer Security Officer    Phone: +44(0)2076792049
>> Information Services Division       Internal Ext: 32049
>> University College London
>> London WC1E 6BT
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.17 (MingW32)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iQEcBAEBAgAGBQJPda91AAoJELhVoVpEMS6RPbwH/1nXjmMEbDzE6CQhGAgfYb6c
>> ebsrxKam3owvkOL2A/LHGeo4Y0nfjQ622jwZPhUHwEMl0FGNf6L7BNq9g//HOqhi
>> NKZQFAhYa45J6Fk2DpPAp6KUYb/RLHA0z3OflJtzFn18jAK9QE9POuRMiYSoqo18
>> XWZoxs3OuVi+UOxuWb97GAOoScsRrC5mQ2EI4LdodC9rjqy0RqJDhPxOVauOss7B
>> e65jJBxVgCCM2SfnnBoKy4PJR2XO0i3UguU6CGILiKFjb0SVScIzTvpxOelCR7bA
>> /TXZd/rnfhGHKFAhrx38bnfDgDvjFyQF/GbJkAfX3Cu7aEXWa1L5oA0oepJi868=
>> =MopT
>> -----END PGP SIGNATURE-----
>>
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users@...
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>
>
>
> --
> Regards,
> Peter Manev
>
>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users@...
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
Peter Bates | 30 Mar 16:35 2012
Picon
Picon

Re: Suricata's http-log


Hello all

On 30/03/2012 14:12, Peter Manev wrote:
> Is there any way that you could compare the two logs by the ways
> of scripting/bashing ? - if Suri and httpry are running at the same
> time (maybe just 10 min time span)?

Running both for ten minutes (both sniffing from eth1):
-rw-r-----. 1 snort  snort  2.0M Mar 30 14:30 http.log.10mins
-rw-r--r--. 1 httpry httpry 268M Mar 30 14:30 httpry.log.10mins

Httpry is only compiled with libpcap and I was running Suricata with
AFPACKET so I tried a test for 10 seconds with both using pcap:

-rw-r--r--. 1 root   root   2.1M Mar 30 14:39 httpry.log
-rw-r-----. 1 snort  snort  590K Mar 30 14:35 http.log

Httpry by default also logs the server responses on a seperate line,
but I've removed those and still see the difference above.

Taking an arbitrary host from the log, 'grooveshark.com':
http.log:

03/30/2012-15:16:00.459591 /more.php?getStreamKeyFromSongIDEx
03/30/2012-15:16:00.716673 /more.php?albumGetAllSongs
03/30/2012-15:16:00.749612 /more.php?markSongDownloadedEx
03/30/2012-15:16:00.801473 /more.php?getArtistByID
03/30/2012-15:16:01.024251 /more.php?getAlbumRecentListeners
03/30/2012-15:16:01.889693 /more.php?getArtistProfileFeed
03/30/2012-15:16:01.995698 /more.php?artistGetAllSongs
03/30/2012-15:16:06.547887 /more.php?addSongsToQueue
03/30/2012-15:16:18.436115
/dfpAds.html?p=artist_profile&w=728&h=90&2=865&3=4331&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1
03/30/2012-15:16:18.873468 /more.php?addSongsToQueue
03/30/2012-15:16:21.134086
/dfpAds.html?p=artist_profile&w=300&h=250&2=865&3=4315&4=41&5=0&9=0&11=592&15=41&18=30&19=0&0=1
03/30/2012-15:16:21.134257 /more.php?addSongsToQueue
03/30/2012-15:16:21.135646 /more.php?artistGetFans

httpry.log:

2012-03-30 15:16:00 POST /more.php?getStreamKeyFromSongIDEx
2012-03-30 15:16:00 POST /more.php?getArtistByID
2012-03-30 15:16:00 POST /more.php?albumGetAllSongs
2012-03-30 15:16:00 POST /more.php?markSongDownloadedEx
2012-03-30 15:16:00 POST /more.php?getAlbumRecentListeners
2012-03-30 15:16:01 POST /more.php?getPageNameByIDType
2012-03-30 15:16:01 POST /more.php?getArtistProfileFeed
2012-03-30 15:16:01 POST /more.php?artistGetAllSongs
2012-03-30 15:16:01 POST /more.php?artistGetSimilarArtists
2012-03-30 15:16:01 POST /more.php?getSongkickEventsFromArtists
2012-03-30 15:16:01 GET
/dfpAds.html?p=artist_profile&w=300&h=250&2=865&3=4315&4=41&5=0&9=0&11=592&15=41&18=30&19=0&0=1
2012-03-30 15:16:02 POST /more.php?artistGetFans
2012-03-30 15:16:02 GET
/dfpAds.html?p=artist_profile&w=728&h=90&2=865&3=4315&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1
2012-03-30 15:16:02 POST /more.php?getArtistRecentListeners
2012-03-30 15:16:04 POST /more.php?addSongsToQueue
2012-03-30 15:16:05 POST /more.php?addSongsToQueue
2012-03-30 15:16:05 POST /more.php?getStreamKeyFromSongIDEx
2012-03-30 15:16:06 POST /more.php?markSongDownloadedEx
2012-03-30 15:16:06 POST /more.php?addSongsToQueue
2012-03-30 15:16:07 POST /more.php?addSongsToQueue
2012-03-30 15:16:08 POST /more.php?addSongsToQueue
2012-03-30 15:16:09 GET
/dfpAds.html?p=song_overview&w=300&h=250&2=401367&3=20839&4=193&5=0&9=0&11=592&15=38&18=30&19=0&0=1
2012-03-30 15:16:09 GET
/dfpAds.html?p=song_overview&w=728&h=90&2=401367&3=20839&4=193&5=0&9=0&11=592&15=38&18=30&19=0&0=1
2012-03-30 15:16:10 POST /more.php?addSongsToQueue
2012-03-30 15:16:17 GET
/dfpAds.html?p=artist_profile&w=300&h=250&2=865&3=4331&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1
2012-03-30 15:16:17 GET
/dfpAds.html?p=artist_profile&w=728&h=90&2=865&3=4331&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1

--

-- 
Peter Bates
Senior Computer Security Officer    Phone: +44(0)2076792049
Information Services Division	    Internal Ext: 32049
University College London
London WC1E 6BT
Peter Manev | 30 Mar 16:48 2012
Picon

Re: Suricata's http-log

Hi,
 
Is there a chance that you can share a small pcap for this? privately if you would like - lets say for a smaller amount of time - that would be possible to be mailed....
Please have in mind that Suricata actually logs only properly terminated connections in terms of http (FA received, proper tcp teardown).
 
Thanks
On Fri, Mar 30, 2012 at 4:35 PM, Peter Bates <peter.bates <at> ucl.ac.uk> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hello all

On 30/03/2012 14:12, Peter Manev wrote:
> Is there any way that you could compare the two logs by the ways
> of scripting/bashing ? - if Suri and httpry are running at the same
> time (maybe just 10 min time span)?

Running both for ten minutes (both sniffing from eth1):
- -rw-r-----. 1 snort  snort  2.0M Mar 30 14:30 http.log.10mins
- -rw-r--r--. 1 httpry httpry 268M Mar 30 14:30 httpry.log.10mins

Httpry is only compiled with libpcap and I was running Suricata with
AFPACKET so I tried a test for 10 seconds with both using pcap:

- -rw-r--r--. 1 root   root   2.1M Mar 30 14:39 httpry.log
- -rw-r-----. 1 snort  snort  590K Mar 30 14:35 http.log

Httpry by default also logs the server responses on a seperate line,
but I've removed those and still see the difference above.

Taking an arbitrary host from the log, 'grooveshark.com':
http.log:

03/30/2012-15:16:00.459591 /more.php?getStreamKeyFromSongIDEx
03/30/2012-15:16:00.716673 /more.php?albumGetAllSongs
03/30/2012-15:16:00.749612 /more.php?markSongDownloadedEx
03/30/2012-15:16:00.801473 /more.php?getArtistByID
03/30/2012-15:16:01.024251 /more.php?getAlbumRecentListeners
03/30/2012-15:16:01.889693 /more.php?getArtistProfileFeed
03/30/2012-15:16:01.995698 /more.php?artistGetAllSongs
03/30/2012-15:16:06.547887 /more.php?addSongsToQueue
03/30/2012-15:16:18.436115
/dfpAds.html?p=artist_profile&w=728&h=90&2=865&3=4331&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1
03/30/2012-15:16:18.873468 /more.php?addSongsToQueue
03/30/2012-15:16:21.134086
/dfpAds.html?p=artist_profile&w=300&h=250&2=865&3=4315&4=41&5=0&9=0&11=592&15=41&18=30&19=0&0=1
03/30/2012-15:16:21.134257 /more.php?addSongsToQueue
03/30/2012-15:16:21.135646 /more.php?artistGetFans

httpry.log:

2012-03-30 15:16:00 POST /more.php?getStreamKeyFromSongIDEx
2012-03-30 15:16:00 POST /more.php?getArtistByID
2012-03-30 15:16:00 POST /more.php?albumGetAllSongs
2012-03-30 15:16:00 POST /more.php?markSongDownloadedEx
2012-03-30 15:16:00 POST /more.php?getAlbumRecentListeners
2012-03-30 15:16:01 POST /more.php?getPageNameByIDType
2012-03-30 15:16:01 POST /more.php?getArtistProfileFeed
2012-03-30 15:16:01 POST /more.php?artistGetAllSongs
2012-03-30 15:16:01 POST /more.php?artistGetSimilarArtists
2012-03-30 15:16:01 POST /more.php?getSongkickEventsFromArtists
2012-03-30 15:16:01 GET
/dfpAds.html?p=artist_profile&w=300&h=250&2=865&3=4315&4=41&5=0&9=0&11=592&15=41&18=30&19=0&0=1
2012-03-30 15:16:02 POST /more.php?artistGetFans
2012-03-30 15:16:02 GET
/dfpAds.html?p=artist_profile&w=728&h=90&2=865&3=4315&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1
2012-03-30 15:16:02 POST /more.php?getArtistRecentListeners
2012-03-30 15:16:04 POST /more.php?addSongsToQueue
2012-03-30 15:16:05 POST /more.php?addSongsToQueue
2012-03-30 15:16:05 POST /more.php?getStreamKeyFromSongIDEx
2012-03-30 15:16:06 POST /more.php?markSongDownloadedEx
2012-03-30 15:16:06 POST /more.php?addSongsToQueue
2012-03-30 15:16:07 POST /more.php?addSongsToQueue
2012-03-30 15:16:08 POST /more.php?addSongsToQueue
2012-03-30 15:16:09 GET
/dfpAds.html?p=song_overview&w=300&h=250&2=401367&3=20839&4=193&5=0&9=0&11=592&15=38&18=30&19=0&0=1
2012-03-30 15:16:09 GET
/dfpAds.html?p=song_overview&w=728&h=90&2=401367&3=20839&4=193&5=0&9=0&11=592&15=38&18=30&19=0&0=1
2012-03-30 15:16:10 POST /more.php?addSongsToQueue
2012-03-30 15:16:17 GET
/dfpAds.html?p=artist_profile&w=300&h=250&2=865&3=4331&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1
2012-03-30 15:16:17 GET
/dfpAds.html?p=artist_profile&w=728&h=90&2=865&3=4331&4=42&5=0&9=0&11=592&15=42&18=30&19=0&0=1

- --
Peter Bates
Senior Computer Security Officer    Phone: +44(0)2076792049
Information Services Division       Internal Ext: 32049
University College London
London WC1E 6BT
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPdcSdAAoJELhVoVpEMS6R17AH/3E4Yvs5X00yka73fftD5RAk
DrXGILyM5lO0O4t7fQBGt2u4704ECfl3071k4AY9qLew/hl/4UqkTRtOf7OL2lOq
nqNKgoSUJD0iNZGv/K1Gi3M0osm4wv73NZd+vo2AUNQtBDduEIVehu0ksVxkl6CL
IVPXHwaRgzwRpyV41Z7PseLeJkJdxHKNxpjifqX5gAbGT2HLbjWBZukgK8Y6a+qo
HHOWT1RS6kTtKC2p5umO1PKwQvcv3b4OCntLhTjF1ylhp2x7amnuMQc7X5JGDaR7
qIWQkZol5t+DJq1cfucZUSUZ3PpY70gNnUELiiQ+jCPYqxUQEmPqppax4/T/1+o=
=jd2v
-----END PGP SIGNATURE-----




--
Regards,
Peter Manev

_______________________________________________
Oisf-users mailing list
Oisf-users@...
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Victor Julien | 30 Mar 16:49 2012
Picon

Re: Suricata's http-log

On 03/30/2012 04:48 PM, Peter Manev wrote:
> Please have in mind that Suricata actually logs only properly terminated
> connections in terms of http (FA received, proper tcp teardown).

TCP sessions that time out (no RST or FIN sequence) will be logged as well.

--

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
Martin Holste | 30 Mar 17:05 2012
Picon

Re: Suricata's http-log

Please use wc -l to count lines instead of file sizes when comparing.

On Fri, Mar 30, 2012 at 9:49 AM, Victor Julien <victor@...> wrote:
> On 03/30/2012 04:48 PM, Peter Manev wrote:
>> Please have in mind that Suricata actually logs only properly terminated
>> connections in terms of http (FA received, proper tcp teardown).
>
> TCP sessions that time out (no RST or FIN sequence) will be logged as well.
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users@...
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Victor Julien | 30 Mar 17:14 2012
Picon

Re: Suricata's http-log

Also, since Suri's http engine is stateful packet loss may be a factor
as well. The "stream.gap" counter is one indication of streams affecting
packet loss.

On 03/30/2012 05:05 PM, Martin Holste wrote:
> Please use wc -l to count lines instead of file sizes when comparing.
> 
> On Fri, Mar 30, 2012 at 9:49 AM, Victor Julien <victor@...> wrote:
>> On 03/30/2012 04:48 PM, Peter Manev wrote:
>>> Please have in mind that Suricata actually logs only properly terminated
>>> connections in terms of http (FA received, proper tcp teardown).
>>
>> TCP sessions that time out (no RST or FIN sequence) will be logged as well.
>>
>> --
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users@...
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> 

--

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
Martin Holste | 30 Mar 17:28 2012
Picon

Re: Suricata's http-log

Ah, along with that, if you have a stream cutoff (as most do), then
requests after that stream cutoff won't get logged.  There can be many
HTTP requests on the same stream, so this may be affecting things.
Still, this seems to be quite a disparity, so I'm a bit concerned.

On Fri, Mar 30, 2012 at 10:14 AM, Victor Julien <victor@...> wrote:
> Also, since Suri's http engine is stateful packet loss may be a factor
> as well. The "stream.gap" counter is one indication of streams affecting
> packet loss.
>
> On 03/30/2012 05:05 PM, Martin Holste wrote:
>> Please use wc -l to count lines instead of file sizes when comparing.
>>
>> On Fri, Mar 30, 2012 at 9:49 AM, Victor Julien <victor@...> wrote:
>>> On 03/30/2012 04:48 PM, Peter Manev wrote:
>>>> Please have in mind that Suricata actually logs only properly terminated
>>>> connections in terms of http (FA received, proper tcp teardown).
>>>
>>> TCP sessions that time out (no RST or FIN sequence) will be logged as well.
>>>
>>> --
>>> ---------------------------------------------
>>> Victor Julien
>>> http://www.inliniac.net/
>>> PGP: http://www.inliniac.net/victorjulien.asc
>>> ---------------------------------------------
>>>
>>> _______________________________________________
>>> Oisf-users mailing list
>>> Oisf-users@...
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
Peter Bates | 30 Mar 17:30 2012
Picon
Picon

Re: Suricata's http-log


Hello all

On 30/03/2012 16:05, Martin Holste wrote:
> Please use wc -l to count lines instead of file sizes when
> comparing.

Running httpry and Suricata with a BPF of a known host and generating
various GET requests seems to elicit identical logs (when eliminating
the fact that httpry logs the response as Martin noted so the log is
double the size).

I'll dig a bit more - there is obviously a bit of a difference between
testing against one destination from one source and the traffic I
usually see.

--

-- 
Peter Bates
Senior Computer Security Officer    Phone: +44(0)2076792049
Information Services Division	    Internal Ext: 32049
University College London
London WC1E 6BT
Martin Holste | 30 Mar 17:49 2012
Picon

Re: Suricata's http-log

This is what I was afraid of.  It sounds to me like Suricata can't
keep up logging at medium to high volumes.

On Fri, Mar 30, 2012 at 10:30 AM, Peter Bates <peter.bates@...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Hello all
>
> On 30/03/2012 16:05, Martin Holste wrote:
>> Please use wc -l to count lines instead of file sizes when
>> comparing.
>
> Running httpry and Suricata with a BPF of a known host and generating
> various GET requests seems to elicit identical logs (when eliminating
> the fact that httpry logs the response as Martin noted so the log is
> double the size).
>
> I'll dig a bit more - there is obviously a bit of a difference between
> testing against one destination from one source and the traffic I
> usually see.
>
> - --
> Peter Bates
> Senior Computer Security Officer    Phone: +44(0)2076792049
> Information Services Division       Internal Ext: 32049
> University College London
> London WC1E 6BT
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJPddF9AAoJELhVoVpEMS6RfCkIAJV8KggdatFHFZsb5NNMRcc9
> IgUR6Y7TVknwfUZL9uJi7P/gOeJqJlmAcl4tcuG8CfWy5tZEWDJQ0UoOKV/GCeU7
> 1iSn0aL6eAhB46xjiI3vGGiAPiZ0SjKD4yCEDJCoUX1SV8h+Ov+7H7sHOzjHX9Da
> D6KV+4B4UKSor96n/Fbfvnk70BmvygrL4QNe/AYw7G77MykXh3uIGFfwHKdZW3dw
> uCjScrtiWfA6gHUBaxKM9syScZU1OMRGr9gaVTBNRZXrC2Kz9T5LSvJtY7KvYbc/
> QBdlOtIYn4/hqVpCi1iV2P2Qm6B2l+F/T3zngjGxHfUBiCFYKW8k5fMvNpJEttc=
> =8bHp
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users@...
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Victor Julien | 30 Mar 17:55 2012
Picon

Re: Suricata's http-log


On 03/30/2012 05:49 PM, Martin Holste wrote:
> This is what I was afraid of.  It sounds to me like Suricata can't 
> keep up logging at medium to high volumes.

Btw, we identified a scalability issue wrt http logging. Fix should be
in git sometime next week:

https://redmine.openinfosecfoundation.org/issues/438

Cheers,
Victor

> On Fri, Mar 30, 2012 at 10:30 AM, Peter Bates
> <peter.bates@...> wrote:
> 
> Hello all
> 
> On 30/03/2012 16:05, Martin Holste wrote:
>>>> Please use wc -l to count lines instead of file sizes when 
>>>> comparing.
> 
> Running httpry and Suricata with a BPF of a known host and
> generating various GET requests seems to elicit identical logs
> (when eliminating the fact that httpry logs the response as Martin
> noted so the log is double the size).
> 
> I'll dig a bit more - there is obviously a bit of a difference
> between testing against one destination from one source and the
> traffic I usually see.
> 
>> 
>> _______________________________________________ Oisf-users
>> mailing list Oisf-users@... 
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>> 
_______________________________________________
> Oisf-users mailing list
Oisf-users@... 
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> 

--

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

Martin Holste | 30 Mar 18:41 2012
Picon

Re: Suricata's http-log

Ok, cool.  Looks like we'll need Peter to re-run his tests on the new
code next week then!

On Fri, Mar 30, 2012 at 10:55 AM, Victor Julien <victor@...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 03/30/2012 05:49 PM, Martin Holste wrote:
>> This is what I was afraid of.  It sounds to me like Suricata can't
>> keep up logging at medium to high volumes.
>
> Btw, we identified a scalability issue wrt http logging. Fix should be
> in git sometime next week:
>
> https://redmine.openinfosecfoundation.org/issues/438
>
> Cheers,
> Victor
>
>> On Fri, Mar 30, 2012 at 10:30 AM, Peter Bates
>> <peter.bates@...> wrote:
>>
>> Hello all
>>
>> On 30/03/2012 16:05, Martin Holste wrote:
>>>>> Please use wc -l to count lines instead of file sizes when
>>>>> comparing.
>>
>> Running httpry and Suricata with a BPF of a known host and
>> generating various GET requests seems to elicit identical logs
>> (when eliminating the fact that httpry logs the response as Martin
>> noted so the log is double the size).
>>
>> I'll dig a bit more - there is obviously a bit of a difference
>> between testing against one destination from one source and the
>> traffic I usually see.
>>
>>>
>>> _______________________________________________ Oisf-users
>>> mailing list Oisf-users@...
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>>>
> _______________________________________________
>> Oisf-users mailing list Oisf-users@...
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>
> - --
> - ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> - ---------------------------------------------
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk9113EACgkQiSMBBAuniMci+ACfRUgOyXcf0qmangDHv586ibeV
> PwkAn17Mcri1nZx6Y/qaJeexUsSTndUK
> =4tiC
> -----END PGP SIGNATURE-----
> _______________________________________________
> Oisf-users mailing list
> Oisf-users@...
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Seth Hall | 30 Mar 18:45 2012

Re: Suricata's http-log


On Mar 30, 2012, at 11:30 AM, Peter Bates wrote:

> Running httpry and Suricata with a BPF of a known host and generating
> various GET requests seems to elicit identical logs (when eliminating
> the fact that httpry logs the response as Martin noted so the log is
> double the size).

We wouldn't even complain if you threw Bro in the mix for comparing logs. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/
Martin Holste | 30 Mar 19:44 2012
Picon

Re: Suricata's http-log

Agree:  Of special interest are any requests greater than the packet
MTU as httpry does not truncate, it *drops* any requests greater than
the MTU.

On Fri, Mar 30, 2012 at 11:45 AM, Seth Hall <seth@...> wrote:
>
> On Mar 30, 2012, at 11:30 AM, Peter Bates wrote:
>
>> Running httpry and Suricata with a BPF of a known host and generating
>> various GET requests seems to elicit identical logs (when eliminating
>> the fact that httpry logs the response as Martin noted so the log is
>> double the size).
>
>
> We wouldn't even complain if you threw Bro in the mix for comparing logs. :)
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users@...
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users

Gmane