Re: Rmpi and ncdf4
I suggest you start by looking through the netcdf documentation and
ask about this on their mailing lists. It seems to me that the
problem is one of multiple processes accessing the same file
simultaneously. I would guess that the netcdf people know whether it
is possible or not, and also what the interface should be (my guess is
that there is an interface). The next step is then to figure out
whether ncdf4 implements this interface.
Kasper
On Fri, Sep 10, 2010 at 12:12 PM, clement
<clement.tisseuil@...> wrote:
> Dear members,
>
> I have started to adapt the "task pull" approach with Rmpi to parallelize my
> calculations on a NetCDF file of dimension lat[1:6], lon[1:6] and
> time[1:11323]. The basic idea is to parallelize the calculation by splitting
> the process according to each latitudinal coordinate i.e. for lat=1, lat=2,
> ..., lat=6. Please, see the code below and the link to the NetCDF file if
> you want to try :
> https://www.transferbigfiles.com/e4d7acd2-c186-4867-b4bf-82fe5aee4023?rid=jIcbZN%2blDcQz3kMeEHCQog%3d%3d
>
> In the given example, the function works perfectly because the NetCDF file
> is quite small (~ 2 Mo). However, some difficulties occur when I try the
> function with larger NetCDF files (~ 8 Go), covering the entire globe, of
> dimensions lat[1:576], lon[1:1152] and time[1:11323]. The problem is that
> each processor (slave) needs to "connect" the NetCDF file before extracting
> the data required according to the latitudinal coordinate of interest. This
> connection takes more than 2 minutes.
>
(Continue reading)
RSS Feed