headers
Sergi Pons Freixes | 29 Jun 2012 11:52
Picon

Dates interpretation on csv2rec

Dear all,

I have a CSV file with the first column with timestamps:

0:00:00 01/01/2007,       0.000,     10,       0.000,     10,
0.000,     10:
00:00:00 02/01/2007,       0.000,     10,       0.000,     10,
0.000,     10
00:00:00 03/01/2007,       0.000,     10,       0.000,     10,
0.000,     10
...
00:00:00 29/12/2009,       0.000,     10,       0.000,     10,
0.000,     10
00:00:00 30/12/2009,       0.000,     10,       0.000,     10,
0.000,     10
00:00:00 31/12/2009,       0.000,     10,       0.000,     10,
0.000,     10

As you can see, the format is hour:minute:second (nor relevant, always
00) day/month/year.

When loaded with mlab.csv2rec, it automatically detects them as dates
and creates datetime objects, but it is not consistent with the
interpretation of the dates. When the day is < 13, it interprets it as
month/day/year. It's loaded correctly otherwise (as day/month/year). I
could pre-process the files and change the dates format, but imho it's
not very elegant. Any suggestion about how to address this issue?

Regards,
Sergi
(Continue reading)

Permalink | Reply |
headers
Andreas Hilboll | 29 Jun 2012 12:02
Picon

Re: Dates interpretation on csv2rec

> Dear all,
>
> I have a CSV file with the first column with timestamps:
>
> 0:00:00 01/01/2007,       0.000,     10,       0.000,     10,
> 0.000,     10:
> 00:00:00 02/01/2007,       0.000,     10,       0.000,     10,
> 0.000,     10
> 00:00:00 03/01/2007,       0.000,     10,       0.000,     10,
> 0.000,     10
> ...
> 00:00:00 29/12/2009,       0.000,     10,       0.000,     10,
> 0.000,     10
> 00:00:00 30/12/2009,       0.000,     10,       0.000,     10,
> 0.000,     10
> 00:00:00 31/12/2009,       0.000,     10,       0.000,     10,
> 0.000,     10
>
> As you can see, the format is hour:minute:second (nor relevant, always
> 00) day/month/year.
>
> When loaded with mlab.csv2rec, it automatically detects them as dates
> and creates datetime objects, but it is not consistent with the
> interpretation of the dates. When the day is < 13, it interprets it as
> month/day/year. It's loaded correctly otherwise (as day/month/year). I
> could pre-process the files and change the dates format, but imho it's
> not very elegant. Any suggestion about how to address this issue?
>
> Regards,
> Sergi
(Continue reading)

Permalink | Reply |
headers
Sergi Pons Freixes | 29 Jun 2012 15:00
Picon

Re: Dates interpretation on csv2rec

On Fri, Jun 29, 2012 at 12:02 PM, Andreas Hilboll <lists@...> wrote:
>
> You could use numpy.genfromtxt together with its `converters` parameter:
>
>     converters : variable, optional
>         The set of functions that convert the data of a column to a value.
>         The converters can also be used to provide a default value
>         for missing data: ``converters = {3: lambda s: float(s or 0)}``.

I finally used:

def mkdate(text):
    return(dt.datetime.strptime(text.decode('utf8'),\
            '%H:%M:%S %d/%m/%Y'))

and

data = np.genfromtxt(\
            os.path.join(sys.argv[1],f),\
            delimiter=',',\
            skip_header=4,\
            usecols = (0, 1, 2),\
            dtype=(object, float, float),\
            converters={0: mkdate},\
            autostrip=True,\
            )

Thank you!

------------------------------------------------------------------------------
(Continue reading)

Permalink | Reply |

Gmane