Ernesto Rodriguez | 12 Apr 23:10 2013

GSoC proposal: Data Visualization

Dear Haskell Community,


During the last months I used Haskell for machine learning, particularly in the field of Echo State Neural Networks. The main drawback I encountered is that its difficult to visualize and plot data in Haskell in spite the fact there are a couple of plotting libraries. Data visualization is very important in the field of machine learning research (not so much in machine learning implementation) since humans are very efficient to analyze graphical input to figure out what is going on in order to determine possible adjustments. I was wondering if other members of the community have experienced this drawback and would be interested in improved data visualization for Haskell, especially if there is interest to use Haskell for machine learning research. I collected my ideas in the following page: https://github.com/netogallo/Visualizer . Please provide me with feedback because if the proposal is interesting for the community I would start working with it, even if it doesn't make it to this GSoC, but a project like this will need a lot of collaboration for it to be successful. 

Thank you very much,

Best Regards,

Ernesto

--
Ernesto Rodriguez

Bachelor of Computer Science - Class of 2013
Jacobs University Bremen



_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Carter Schonwald | 12 Apr 23:55 2013
Picon

Re: GSoC proposal: Data Visualization

Hello Ernesto,

There are a number of efforts underway to provide better data vis libraries for haskell. Likewise, there was some recent discussion on the Diagrams mailing list about data vis tooling, and there should be a few interesting tools surfacing over the coming few months.


My immediate concern is that this project is too broad and undefined in scope to be a successful Haskell GSOC.
A successful GSOC project should have
a) a clear notion of what project's goal is 
b) clear evidence that the planned work can reasonably be done over the summer
c) the result of a successful project would be  valuable to the general haskell community

It sounds like the core of what you want to do is write a small lib that transforms a data set from some initial "schema" into the "schema" thats suitable for some underlying choice in plotting tool.  This is a useful thing to do, but not large enough in scope for a GSOC project. 

On the flip side, interactive data vis tools are *hard* to do well, and a GSOC that proposed to work on that from scratch would be very very risky unless you've spent a lot of time working on building such tools. 


You're definitely pointing at region of library space where more nice tools for haskell would be very valuable, and which a number of folks are trying to address.  But, for GSOC, unless its a very very clearly laid out proposal, it will be deemed too risky. 

I warmly recommend you look at prior years' Haskell GSOC projects to get a feel for what strong successful projects/proposals look like. 


cheers
-Carter






On Fri, Apr 12, 2013 at 5:10 PM, Ernesto Rodriguez <neto <at> netowork.me> wrote:
Dear Haskell Community,

During the last months I used Haskell for machine learning, particularly in the field of Echo State Neural Networks. The main drawback I encountered is that its difficult to visualize and plot data in Haskell in spite the fact there are a couple of plotting libraries. Data visualization is very important in the field of machine learning research (not so much in machine learning implementation) since humans are very efficient to analyze graphical input to figure out what is going on in order to determine possible adjustments. I was wondering if other members of the community have experienced this drawback and would be interested in improved data visualization for Haskell, especially if there is interest to use Haskell for machine learning research. I collected my ideas in the following page: https://github.com/netogallo/Visualizer . Please provide me with feedback because if the proposal is interesting for the community I would start working with it, even if it doesn't make it to this GSoC, but a project like this will need a lot of collaboration for it to be successful. 

Thank you very much,

Best Regards,

Ernesto

--
Ernesto Rodriguez

Bachelor of Computer Science - Class of 2013
Jacobs University Bremen




_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Heinrich Apfelmus | 13 Apr 10:33 2013
Picon

Re: GSoC proposal: Data Visualization

Ernesto Rodriguez wrote:
> Dear Haskell Community,
> 
> During the last months I used Haskell for machine learning, particularly in
> the field of Echo State Neural Networks. The main drawback I encountered is
> that its difficult to visualize and plot data in Haskell in spite the fact
> there are a couple of plotting libraries. Data visualization is very
> important in the field of machine learning research (not so much in machine
> learning implementation) since humans are very efficient to analyze
> graphical input to figure out what is going on in order to determine
> possible adjustments. I was wondering if other members of the community
> have experienced this drawback and would be interested in improved data
> visualization for Haskell, especially if there is interest to use Haskell
> for machine learning research. I collected my ideas in the following page:
>  https://github.com/netogallo/Visualizer . Please provide me with feedback
> because if the proposal is interesting for the community I would start
> working with it, even if it doesn't make it to this GSoC, but a project
> like this will need a lot of collaboration for it to be successful.

Your project is very ambitious! In fact, too ambitious.

Essentially, you want to build an interactive environment for evaluating 
Haskell expressions. The use case you have in mind is data visualization 
for machine learning, but that is just a special case. If you can zoom 
in and out of plots of infinite time series, you can zoom in and out of 
audio data, and then why not add an interactive synthesizer widget to 
create that audio data in the first place.

Your idea decomposes into many parts, each of which would easily fill an 
entire GSoC project on their own.

* GUI. Actually, we currently don't have a GUI library that is easy to 
install for everyone. Choosing wxHaskell or gtk2hs immediately separates 
your user base into three disjoint parts. I think it's possible to use 
the web browser as GUI instead 
(<https://github.com/HeinrichApfelmus/threepenny-gui>).

* Displaying Haskell values in a UI. You mentioned that you want 
matrices to come with a contextual menu where you can select different 
transformations on them. It's just a minor step to allow any Haskell 
function operating on them. I have a couple of ideas on how to do this 
is in a generic fashion. Unfortunately, the project from last year 
<http://hackage.haskell.org/trac/summer-of-code/ticket/1609> did not 
succeed satisfactorily. There were some other efforts, but I haven't 
seen anything released.

* UI programming is hard. You could easily spend an entire project on 
implementing a single visualization, for instance an infinite time 
series with responsive zoom. It's not difficult to implement something, 
but adding the right level of polish so that people want to use it takes 
effort. There's a reason that Matlab costs money, and there's a reason 
that your mentor relies on it.

* Functionality specific to machine learning. Converting Vector to a 
format suitable for representation of matrices, etc. This is your 
primary interest.

Note that, unfortunately, the parts depend on each other from top to 
bottom. It's possible to write functionality specific to machine 
learning, but it would be of little impact if it doesn't come with a 
good UI.

Best regards,
Heinrich Apfelmus

--
http://apfelmus.nfshost.com
Ernesto Rodriguez | 13 Apr 11:45 2013

Re: GSoC proposal: Data Visualization

Hi Hienrich,


It is indeed a big scope as you mentioned. Matlab has been working for years to get this functionality right. On the other hand, the project you linked is interesting. For me it would already be a huge advantage if I could edit and re-evaluate expressions interactively (in a comfortable GUI, not ghci). Also a plot widget with sliders would also help. I was wondering if you know any reason the project has not been worked on for various months (as I see in the repo). Is there anyone working in this project and has a later version? I mean these are features that are even available in free math packages such as Sage.

Best regards,

Ernesto Rodriguez

On Sat, Apr 13, 2013 at 10:33 AM, Heinrich Apfelmus <apfelmus <at> quantentunnel.de> wrote:
Ernesto Rodriguez wrote:
Dear Haskell Community,

During the last months I used Haskell for machine learning, particularly in
the field of Echo State Neural Networks. The main drawback I encountered is
that its difficult to visualize and plot data in Haskell in spite the fact
there are a couple of plotting libraries. Data visualization is very
important in the field of machine learning research (not so much in machine
learning implementation) since humans are very efficient to analyze
graphical input to figure out what is going on in order to determine
possible adjustments. I was wondering if other members of the community
have experienced this drawback and would be interested in improved data
visualization for Haskell, especially if there is interest to use Haskell
for machine learning research. I collected my ideas in the following page:
 https://github.com/netogallo/Visualizer . Please provide me with feedback
because if the proposal is interesting for the community I would start
working with it, even if it doesn't make it to this GSoC, but a project
like this will need a lot of collaboration for it to be successful.

Your project is very ambitious! In fact, too ambitious.

Essentially, you want to build an interactive environment for evaluating Haskell expressions. The use case you have in mind is data visualization for machine learning, but that is just a special case. If you can zoom in and out of plots of infinite time series, you can zoom in and out of audio data, and then why not add an interactive synthesizer widget to create that audio data in the first place.

Your idea decomposes into many parts, each of which would easily fill an entire GSoC project on their own.

* GUI. Actually, we currently don't have a GUI library that is easy to install for everyone. Choosing wxHaskell or gtk2hs immediately separates your user base into three disjoint parts. I think it's possible to use the web browser as GUI instead (<https://github.com/HeinrichApfelmus/threepenny-gui>).

* Displaying Haskell values in a UI. You mentioned that you want matrices to come with a contextual menu where you can select different transformations on them. It's just a minor step to allow any Haskell function operating on them. I have a couple of ideas on how to do this is in a generic fashion. Unfortunately, the project from last year <http://hackage.haskell.org/trac/summer-of-code/ticket/1609> did not succeed satisfactorily. There were some other efforts, but I haven't seen anything released.

* UI programming is hard. You could easily spend an entire project on implementing a single visualization, for instance an infinite time series with responsive zoom. It's not difficult to implement something, but adding the right level of polish so that people want to use it takes effort. There's a reason that Matlab costs money, and there's a reason that your mentor relies on it.

* Functionality specific to machine learning. Converting Vector to a format suitable for representation of matrices, etc. This is your primary interest.


Note that, unfortunately, the parts depend on each other from top to bottom. It's possible to write functionality specific to machine learning, but it would be of little impact if it doesn't come with a good UI.


Best regards,
Heinrich Apfelmus

--
http://apfelmus.nfshost.com



_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe



--
Ernesto Rodriguez

Bachelor of Computer Science - Class of 2013
Jacobs University Bremen



_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Heinrich Apfelmus | 15 Apr 10:32 2013
Picon

Re: GSoC proposal: Data Visualization

Ernesto Rodriguez wrote:
> 
> For me it would already be a huge advantage if I
> could edit and re-evaluate expressions interactively (in a comfortable GUI,
> not ghci). Also a plot widget with sliders would also help. I was wondering
> if you know any reason the project has not been worked on for various
> months (as I see in the repo). Is there anyone working in this project and
> has a later version? I mean these are features that are even available in
> free math packages such as Sage.

I was actually the initial mentor for this project. I'm not particularly 
happy about the result. As you can see, it hasn't been picked up by 
anyone else, including me, and I think that's because it missed the 
modularity goals I had in mind.

Best regards,
Heinrich Apfelmus

--
http://apfelmus.nfshost.com
Carter Schonwald | 14 Apr 01:41 2013
Picon

Re: GSoC proposal: Data Visualization

Heinrich, you hit the nail on the head.

for an interactive plotting story to work well, we wind up needing to have better tools in the ecosystem on the gui / computational notebook side.


on the other hand, similar work was done last summer, as heinrich mentions, in the form of ghc live https://github.com/shapr/ghclive, by 
a very strong GSOC participant, and while it works, its not really being used, and is still quite immature.

Additionally, the folks at FPcomplete have their browser based haskell interaction tool thats seeing quite a lot of use by folks learning haskell, which in turn raises the quality bar that any other effort must achieve to see serious usage.

its also worth wondering if the scala notebook port of the ipython notebook could be used to guid writing a similar tools for haskell.

 Either way,  would require a very concrete plan of attack to be a tractable GSOC project

-Carter




On Sat, Apr 13, 2013 at 4:33 AM, Heinrich Apfelmus <apfelmus <at> quantentunnel.de> wrote:
Ernesto Rodriguez wrote:
Dear Haskell Community,

During the last months I used Haskell for machine learning, particularly in
the field of Echo State Neural Networks. The main drawback I encountered is
that its difficult to visualize and plot data in Haskell in spite the fact
there are a couple of plotting libraries. Data visualization is very
important in the field of machine learning research (not so much in machine
learning implementation) since humans are very efficient to analyze
graphical input to figure out what is going on in order to determine
possible adjustments. I was wondering if other members of the community
have experienced this drawback and would be interested in improved data
visualization for Haskell, especially if there is interest to use Haskell
for machine learning research. I collected my ideas in the following page:
 https://github.com/netogallo/Visualizer . Please provide me with feedback
because if the proposal is interesting for the community I would start
working with it, even if it doesn't make it to this GSoC, but a project
like this will need a lot of collaboration for it to be successful.

Your project is very ambitious! In fact, too ambitious.

Essentially, you want to build an interactive environment for evaluating Haskell expressions. The use case you have in mind is data visualization for machine learning, but that is just a special case. If you can zoom in and out of plots of infinite time series, you can zoom in and out of audio data, and then why not add an interactive synthesizer widget to create that audio data in the first place.

Your idea decomposes into many parts, each of which would easily fill an entire GSoC project on their own.

* GUI. Actually, we currently don't have a GUI library that is easy to install for everyone. Choosing wxHaskell or gtk2hs immediately separates your user base into three disjoint parts. I think it's possible to use the web browser as GUI instead (<https://github.com/HeinrichApfelmus/threepenny-gui>).

* Displaying Haskell values in a UI. You mentioned that you want matrices to come with a contextual menu where you can select different transformations on them. It's just a minor step to allow any Haskell function operating on them. I have a couple of ideas on how to do this is in a generic fashion. Unfortunately, the project from last year <http://hackage.haskell.org/trac/summer-of-code/ticket/1609> did not succeed satisfactorily. There were some other efforts, but I haven't seen anything released.

* UI programming is hard. You could easily spend an entire project on implementing a single visualization, for instance an infinite time series with responsive zoom. It's not difficult to implement something, but adding the right level of polish so that people want to use it takes effort. There's a reason that Matlab costs money, and there's a reason that your mentor relies on it.

* Functionality specific to machine learning. Converting Vector to a format suitable for representation of matrices, etc. This is your primary interest.


Note that, unfortunately, the parts depend on each other from top to bottom. It's possible to write functionality specific to machine learning, but it would be of little impact if it doesn't come with a good UI.


Best regards,
Heinrich Apfelmus

--
http://apfelmus.nfshost.com



_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Gmane