Progress monitor (or progress bar) within a MATLAB parfor loop
In MATLAB, it is really easy to do parallel processing of trivially parallelizable problems with a parfor loop. I do it all the time. It's great. A problem with this is that, if you need to parallelize something in the first place, it's typically something that takes a really long time to run. Some type of progress monitor is normally easier to make, but because parfor does not iterate in order and the workers cannot communicate with one another, it's a little tricky to do in the parallel case.
So I made a very hacky solution to this problem. Basically, I start with a blank file (parfor_progress.txt) and add a line to it each time an iteration of the loop finishes. At that time, I also count the number of lines in parfor_progress.txt and display that as a percentage of the total number of iterations.
I put all this in a function that you can download from the MATLAB Central File Exchange or GitHub or just copy/paste parfor_progress.m from here:
function percent = parfor_progress(N)
%PARFOR_PROGRESS Progress monitor (progress bar) that works with parfor.
% PARFOR_PROGRESS works by creating a file called parfor_progress.txt in
% your working directory, and then keeping track of the parfor loop's
% progress within that file. This workaround is necessary because parfor
% workers cannot communicate with one another so there is no simple way
% to know which iterations have finished and which haven't.
%
% PARFOR_PROGRESS(N) initializes the progress monitor for a set of N
% upcoming calculations.
%
% PARFOR_PROGRESS updates the progress inside your parfor loop and
% displays an updated progress bar.
%
% PARFOR_PROGRESS(0) deletes parfor_progress.txt and finalizes progress
% bar.
%
% To suppress output from any of these functions, just ask for a return
% variable from the function calls, like PERCENT = PARFOR_PROGRESS which
% returns the percentage of completion.
%
% Example:
%
% N = 100;
% parfor_progress(N);
% parfor i=1:N
% pause(rand); % Replace with real code
% parfor_progress;
% end
% parfor_progress(0);
%
% See also PARFOR.
% By Jeremy Scheff - jdscheff@gmail.com - http://www.jeremyscheff.com/
narginchk(0, 1);
if nargin < 1
N = -1;
end
percent = 0;
w = 50; % Width of progress bar
if N > 0
f = fopen('parfor_progress.txt', 'w');
if f<0
error('Do you have write permissions for %s?', pwd);
end
fprintf(f, '%d\n', N); % Save N at the top of progress.txt
fclose(f);
if nargout == 0
disp([' 0%[>', repmat(' ', 1, w), ']']);
end
elseif N == 0
delete('parfor_progress.txt');
percent = 100;
if nargout == 0
disp([repmat(char(8), 1, (w+9)), char(10), '100%[', repmat('=', 1, w+1), ']']);
end
else
if ~exist('parfor_progress.txt', 'file')
error('parfor_progress.txt not found. Run PARFOR_PROGRESS(N) before PARFOR_PROGRESS to initialize parfor_progress.txt.');
end
f = fopen('parfor_progress.txt', 'a');
fprintf(f, '1\n');
fclose(f);
f = fopen('parfor_progress.txt', 'r');
progress = fscanf(f, '%d');
fclose(f);
percent = (length(progress)-1)/progress(1)*100;
if nargout == 0
perc = sprintf('%3.0f%%', percent); % 4 characters wide, percentage
disp([repmat(char(8), 1, (w+9)), char(10), perc, '[', repmat('=', 1, round(percent*w/100)), '>', repmat(' ', 1, w - round(percent*w/100)), ']']);
end
end
Then, to get a progress bar for a parfor loop, all you have to do is something like this:
N = 100;
parfor_progress(N);
parfor i=1:N
pause(rand); % Replace with real code
parfor_progress;
end
parfor_progress(0);
You should thank commenter Ben Aernouts for helping me make the progress bar pretty.
Hey Jeremy,
I found this while Googling for something similar. I ran into a problem while trying to run this, though, in which a test loop of mine would finish, but the indicator would be around 50%. I think it may have been that two or more processors were trying to read/write to the file at the same time. Did you happen to encounter something like that?
The way I fixed it for my particular application was to create a directory instead, and have each call create a new file in that directory, then compare the number of files to the range of the loop. Still kind of hacky, and not sure how well that would scale, but it worked for me.
Comment by Kevin — October 11, 2011 @ 2:23 pm
I found a solution to your ‘pretty text progress bar' problem:
%Parameters
length_waitbar = 50;
nr_loops = 100;
%Parfor loop
if matlabpool(‘size') ==0; matlabpool; end
parfor_progress(nr_loops);
disp([‘Progress:' char(10) repmat(‘ ‘,1,length_waitbar)]);
parfor i = 1:nr_loops
pause(rand) % Calculations
percent = parfor_progress;
disp([repmat(char(8),1,(length_waitbar+2)) char(10) repmat(‘.',1,round(percent*length_waitbar/100))...
repmat(‘*',1,length_waitbar-round(percent*length_waitbar/100))]);
end
parfor_progress(0);
Thanks for your ‘parfor_progress.m' file.
Best regards,
Ben Aernouts
Comment by Ben Aernouts — October 20, 2011 @ 10:59 am
I added code that calculates and displays the estimated time remaining by saving the start time in the parfor_progress.txt file.
http://cl.ly/2b2O1F3x1i3a0r231p3x
-Albert
Comment by Albert Wu — July 16, 2012 @ 4:40 pm