MATLAB Language Reading large files Date and time strings to numeric array fast
Example
Converting date and time strings to numeric arrays can be done with datenum, though it may take as much as half the time of reading a large data file.
Consider the data in example Textscan. By, again, using textscan and interpret date and time as integers, they can rapidly be converted into a numeric array.
I.e. a line in the example data would be interpreted as:
Data , 2015 - 09 - 16 , 15 : 41 : 52 ; 801 , 800.000000 , 1.5123 , 0.0043
ignore double double double double double double double double double double
which will be read as:
fid = fopen('path/to/myfile');
data = textscan(fid,'%*s %f %f %f %f %f %f %f %f %f %f','Delimiter',',-:;');
fclose(fid);
Now:
y = data{1}; % year
m = data{2}; % month
d = data{3}; % day
H = data{4}; % hours
M = data{5}; % minutes
S = data{6}; % seconds
F = data{7}; % milliseconds
% Translation from month to days
ms = [0,31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334];
n = length(y); % Number of elements
Time = zeros(n,1); % Declare numeric time array
% Algorithm for calculating numeric time array
for k = 1:n
Time(k) = y(k)*365 + ms(m(k)) + d(k) + floor(y(k)/4)...
- floor(y(k)/100) + floor(y(k)/400) + (mod(y(k),4)~=0)...
- (mod(y(k),100)~=0) + (mod(y(k),400)~=0)...
+ (H(k)*3600 + M(k)*60 + S(k) + F(k)/1000)/86400 + 1;
end
Using datenum on 566,678 elements required 6.626570 seconds, whilst the method above required 0.048334 seconds, i.e. 0.73% of the time for datenum or ~137 times faster.
