Development discussion for the Yesod Web Framework. ()

Chris Swires | 5 Sep 2012 00:51

Benchmarking Suggestions?

Hi all,

As some of you know I'm currently completing a comparative study of Yesod against ASP.NET for my MSc, and I'm curious to see if anyone has any suggestions for further tests for me to implement. I have completed the standard three (Pong, big-table and static file twice; small and 'smaller'), and I'm scouting around for further tests.

I was toying with the idea of a database query rps test but feel that this is fraught with issue. If I were to standardise the database used (Sqlite3 presumably) then I imagine that the results would be practically identical as the database would effectively be the bottleneck. If I were to use the fastest available for each it would degrade to a simple database comparison.

Ideally these tests will need to be fairly processor light (the online version are using ab with c20 n1000) to avoid ec2 'cpu stealing', but I will be performing a number of them on a larger instance before the end of the project (the end of the month) to allow for concurrency testing.

The four tests completed will be available online (partly so that anyone interested can run the tests themselves and see the results rather than simply read a dry academic paper, and partly as a project req) tomorrow I just need to write the hamlet up for descriptions.

I'd also like to say that my experience with Yesod has been great thus far (it will be a shame to graduate and almost certainly have to work with another framework), and this group has been a great help. I've gone from being relatively new to web-development and with no experience in Haskell at all to writing tutorials in three months, and it's largely down to a great framework and the people here.

Any suggestions are welcome,

Best,

Chris.

headers

Felipe Almeida Lessa | 5 Sep 2012 02:52

Re: Benchmarking Suggestions?

On Tue, Sep 4, 2012 at 7:51 PM, Chris Swires <490415@...> wrote:
> I was toying with the idea of a database query rps test but feel that this
> is fraught with issue. If I were to standardise the database used (Sqlite3
> presumably) then I imagine that the results would be practically identical
> as the database would effectively be the bottleneck. If I were to use the
> fastest available for each it would degrade to a simple database comparison.

Something database-related would make for a nice test.  Even if the
tests show that the database is the only bottleneck, that's a very
nice result on its own (which basically means that you do not need to
be too much concerned about your code's performance if you're
accessing the database).

However, I'd expect you to use PostgreSQL.  Or, given that you're
using AWS, MySQL using Amazon RDS.  SQLite is known to be an extremely
poor database choice for web servers.

> Ideally these tests will need to be fairly processor light (the online
> version are using ab with c20 n1000) to avoid ec2 'cpu stealing', but I will
> be performing a number of them on a larger instance before the end of the
> project (the end of the month) to allow for concurrency testing.

Are you using micro instances?  The results are meaningless when doing
so, there's too much variation depending on external sources.

Cheers, =)

--

-- 
Felipe.

headers

Chris Swires | 5 Sep 2012 09:48

Re: Benchmarking Suggestions?

I will look into starting a database test today then thank you, (maybe in fact a couple to get a broader overview, for example, a simple query as an upper bounds test and then a text based search perhaps).

I am using Micro instances atm, but this is merely to save on my budget until everything is set up, at which point I will probably upgrade to at least medium (preferably large or bigger but I'll have to see how money goes), I believe the small instances share the same fate as micro but with shorter durations.

Thanks again,

Chris.

On Wednesday, 5 September 2012 01:52:46 UTC+1, Felipe Lessa wrote:

On Tue, Sep 4, 2012 at 7:51 P M, Chris Swires <490...-QvkbJbr/y3w2EctHIo1CcQ@public.gmane.org> wrote:
> I was toying with the idea of a database query rps test but feel that this
> is fraught with issue. If I were to standardise the database used (Sqlite3
> presumably) then I imagine that the results would be practically identical
> as the database would effectively be the bottleneck. If I were to use the
> fastest available for each it would degrade to a simple database comparison.

Something database-related would make for a nice test. Even if the
tests show that the database is the only bottleneck, that's a very
nice result on its own (which basically means that you do not need to
be too much concerned about your code's performance if you're
accessing the database).

However, I'd expect you to use PostgreSQL. Or, given that you're
using AWS, MySQL using Amazon RDS. SQLite is known to be an extremely
poor database choice for web servers.

> Ideally these tests will need to be fairly processor light (the online
> version are using ab with c20 n1000) to avoid ec2 'cpu stealing', but I will
> be performing a number of them on a larger instance before the end of the
> project (the end of the month) to allow for concurrency testing.

Are you using micro instances? The results are meaningless when doing
so, there's too much variation depending on external sources.

Cheers, =)

--
Felipe.