day_reduce

 Example usage 

 day_reduce('02-Nov-2010','_jamie')
 Will find all obs for that day and reduce them one by one.
 will save to the concat of date and output_stub e.g. '02-Nov-2010_jamie'

 Use the optional path_to_redscript to specify which redscript you want
 Use the optional sched_to_reduce to specify only specific schedule names 
 to be reduced. 

 JL 24 Oct 2013 - I changed the behaviour of optional parameter sched_to_reduce. It will now
 reduce all observations with have a schedule name which has sched_to_reduce as a substring. 
 E.g. to reduce all the observations which have schedules with the substring "cal" in them
 (i.e the calibrator observations), you can do 

 day_reduce('02-Nov-2010','_jamie','/reduc/redScript.red','cal') 

 this would reduce shedules like scan_calibrators_2013.sch, but not
 e.g. cbass_survey_5speed.sch.

 You can also invert this behaviour by prepending a ~. E.g. to reduce all the schedules which
 do *not* have "cal" as a subsring you can do
 
 day_reduce('02-Nov-2010','_jamie','/reduc/redScript.red','~cal')  
  

 ACT 4th Jan 2012 - fixed a bug whereby if the observation needed to be
 split into chunks, anything starting/ending at 00:00:00 hours crashed.
 ACT/JL 3rd Jan - new paramater max_doable_obs_len added to allow whatever
 length obs you fancy through before chopping up.

 MAS 23-Feb-2012 - added a parameter "runhealth" to allow antenna health
 to be run optionally.  Run antenna health by default.

 SJCM 09-aug-2012:  bug was present when splitting into chunks.  was
 interpreting the number of hours as teh number of chunks needed to reduce.
 also added a final_dir for transferring all the reduced files to a
 specific location

0001 function day_reduce(date,output_stub,path_to_redscript,sched_to_reduce,runhealth, final_dir)
0002 % Example usage
0003 %
0004 % day_reduce('02-Nov-2010','_jamie')
0005 % Will find all obs for that day and reduce them one by one.
0006 % will save to the concat of date and output_stub e.g. '02-Nov-2010_jamie'
0007 %
0008 % Use the optional path_to_redscript to specify which redscript you want
0009 % Use the optional sched_to_reduce to specify only specific schedule names
0010 % to be reduced.
0011 %
0012 % JL 24 Oct 2013 - I changed the behaviour of optional parameter sched_to_reduce. It will now
0013 % reduce all observations with have a schedule name which has sched_to_reduce as a substring.
0014 % E.g. to reduce all the observations which have schedules with the substring "cal" in them
0015 % (i.e the calibrator observations), you can do
0016 %
0017 % day_reduce('02-Nov-2010','_jamie','/reduc/redScript.red','cal')
0018 %
0019 % this would reduce shedules like scan_calibrators_2013.sch, but not
0020 % e.g. cbass_survey_5speed.sch.
0021 %
0022 % You can also invert this behaviour by prepending a ~. E.g. to reduce all the schedules which
0023 % do *not* have "cal" as a subsring you can do
0024 %
0025 % day_reduce('02-Nov-2010','_jamie','/reduc/redScript.red','~cal')
0026 %
0027 %
0028 % ACT 4th Jan 2012 - fixed a bug whereby if the observation needed to be
0029 % split into chunks, anything starting/ending at 00:00:00 hours crashed.
0030 % ACT/JL 3rd Jan - new paramater max_doable_obs_len added to allow whatever
0031 % length obs you fancy through before chopping up.
0032 %
0033 % MAS 23-Feb-2012 - added a parameter "runhealth" to allow antenna health
0034 % to be run optionally.  Run antenna health by default.
0035 %
0036 % SJCM 09-aug-2012:  bug was present when splitting into chunks.  was
0037 % interpreting the number of hours as teh number of chunks needed to reduce.
0038 % also added a final_dir for transferring all the reduced files to a
0039 % specific location
0040 
0041 if(nargin < 6)
0042   final_dir = [];
0043 end
0044 
0045 
0046 max_doable_obs_len=360*4;
0047 
0048 if(~exist('path_to_redscript','var'))
0049   path_to_redscript='reduc/redScript.red';
0050 end
0051 if(~exist('runhealth','var'))
0052   runhealth=1;
0053 end
0054 
0055 [home,installeddir]=where_am_i();
0056 location = [home,'/',installeddir];
0057 
0058 date_number=datenum(date);
0059 num_date = datestr(date_number, 'yyyymmdd');
0060 
0061 antenna_health_dir = ['reduc/',num_date,output_stub];
0062 
0063 outputdir = [num_date,output_stub];
0064 errorfile = [num_date,output_stub,'_errors.txt'];
0065 errorfilepath = [location,'/','reduc/',outputdir,'/',errorfile];
0066 mkdir([location,'/','reduc/',outputdir]);
0067 disp(['day_reduce:: Writing errors to ',errorfilepath]);
0068 errfile = fopen(errorfilepath,'w');
0069 
0070 file = [location,'/','log/obs_log.html'];
0071 
0072 [start_dates,end_dates,schedule] =get_days_obs(file,date);
0073 
0074 disp(['day_reduce:: FOUND ',int2str(length(start_dates)),' observations...']);
0075 
0076 for i=1:length(start_dates)
0077     
0078     sched_bits = regexp(char(schedule(i)), '/', 'split');
0079     sched_name = char(sched_bits(length(sched_bits)));
0080     % strip out possible open bracket if there is one.
0081     sched_name = strrep(sched_name, '(','');
0082     disp(['day_reduce:: Schedule is ',sched_name]);
0083     
0084     % Check for error state.
0085     if strcmp(char(start_dates(i)),'Error')
0086         continue;
0087     end; %endif
0088  
0089 
0090     
0091     if (exist('sched_to_reduce','var'))
0092       negate = 0;
0093       stripped_sched_to_reduce=sched_to_reduce;
0094       if (~isempty(regexp(sched_to_reduce,'^~'))) 
0095       % if we have a tilde up front we should prepare to negate and remove
0096       % the tildee in prep. for future match
0097       negate = 1;
0098       stripped_sched_to_reduce = regexprep(sched_to_reduce,'^~',''); 
0099       end
0100       does_sched_match = ~isempty(regexp(sched_name,stripped_sched_to_reduce,'match'));       
0101       xor_exp = ~xor(does_sched_match,negate);
0102 
0103       if (xor_exp) % This is a case sensitive regex style match
0104     disp(['day_reduce:: SKIPPING this observation ', sched_name,' - does not match the requested schedule substring, namely ''',sched_to_reduce,'''']);   
0105             continue;
0106         end;
0107     end;
0108     
0109     start_string =  remove_colon_datestring(char(start_dates(i)));
0110     end_string =  remove_colon_datestring(char(end_dates(i)));
0111     start_num = datenum(start_string);
0112     end_num = datenum(end_string);
0113     
0114     obs_length_mins = (end_num-start_num)*24*60;
0115     %disp(['day_reduce:: Date num start= ',int2str(start_num), '  date num end= ',int2str(end_num),' Time in mins = ',int2str(obs_length_mins)]);
0116     
0117     if (obs_length_mins<max_doable_obs_len)
0118         
0119         
0120         try
0121             disp(['day_reduce:: READING Obs No. ', int2str(i),' of ',int2str(length(start_dates)),' ->  ',char(start_dates(i)),' --- ',char(end_dates(i))]);
0122             d=read_arc(char(start_dates(i)),char(end_dates(i)));
0123             
0124         catch exception
0125             error_string= ['day_reduce:: FAILED TO READ ', int2str(i),' of ',int2str(length(start_dates)), ' ->  ',char(start_dates(i)),' --- ',char(end_dates(i))];
0126             disp(error_string);
0127             rep = getReport(exception, 'basic');
0128             fprintf(errfile,'%s\n',error_string);
0129             fprintf(errfile,'%s\n',rep);
0130             continue;
0131         end; %endtry
0132 
0133         try
0134             disp(['day_reduce:: REDUCING Obs No. ', int2str(i),' of ',int2str(length(start_dates)),' -> ',char(start_dates(i)),' --- ',char(end_dates(i))]);
0135             dcal2 = reduceData(d, path_to_redscript, 0, outputdir,sched_name);
0136         catch exception
0137             error_string = ['day_reduce:: FAILED TO REDUCE ',int2str(i),' of ',int2str(length(start_dates)), ' ->  ',char(start_dates(i)),' --- ',char(end_dates(i))];
0138             disp(error_string);
0139             rep = getReport(exception, 'basic');
0140             fprintf(errfile,'%s\n',error_string);
0141             fprintf(errfile,'%s\n',rep);
0142             continue;
0143         end;  %endtry
0144         
0145     else
0146         %disp(['day_reduce:: Detected observation longer than 3 hours...']);
0147         % observation greater than 3hrs must split into chunks
0148         
0149         start_num_mins =  start_num*24*60;
0150         end_num_mins =  end_num*24*60;
0151         
0152         no_chunks_reqd =  ceil( obs_length_mins/max_doable_obs_len);
0153 
0154     
0155         disp(['day_reduce:: Obs longer than 360 mins will split obs from ',datestr(start_num),' to ', datestr(end_num), ' into ',int2str(no_chunks_reqd),' sub-observations']);
0156         for j=1:no_chunks_reqd
0157       if (j== no_chunks_reqd)
0158                 new_start_mins= start_num_mins + (j-1)*max_doable_obs_len;
0159                 new_start_days= new_start_mins/(24*60);
0160                 new_start_string = datestr(new_start_days,0);
0161                 new_start_string = insert_colon_datestring(new_start_string);
0162                 new_end_string = char(end_dates(i));
0163             else
0164                 new_start_mins= start_num_mins + (j-1)*max_doable_obs_len;
0165                 new_start_days= new_start_mins/(24*60);
0166                 new_start_string = datestr(new_start_days,0);
0167                 new_end_mins= start_num_mins + (j)*max_doable_obs_len;
0168                 new_end_days= new_end_mins/(24*60);
0169                 %keyboard;
0170                 new_end_string = datestr(new_end_days);
0171                 whos
0172                 % Re-insert semicolon into date string.
0173                 
0174                 new_start_string = insert_colon_datestring(new_start_string) ;
0175                 new_end_string = insert_colon_datestring(new_end_string) ;
0176                 
0177             end
0178             
0179             
0180             
0181             try
0182                 disp(['day_reduce:: READING Obs No. ', int2str(i),', chunk ',int2str(j),' of ',int2str(no_chunks_reqd),'   ->  ',...
0183                     new_start_string, ' --- ',new_end_string]);
0184                 d=read_arc(new_start_string,new_end_string);
0185                 
0186             catch exception
0187                 error_string= ['day_reduce:: FAILED TO READ ',int2str(i),':',int2str(j), '  of ',int2str(length(start_dates)),':',int2str(no_chunks_reqd),'   ->  ',...
0188                     new_start_string, ' --- ',new_end_string];
0189                 
0190                 disp(error_string);
0191                 rep = getReport(exception, 'basic');
0192                 fprintf(errfile,'%s\n',error_string);
0193                 fprintf(errfile,'%s\n',rep);
0194                 continue;
0195             end; %endtry
0196             
0197             try
0198                 disp(['day_reduce:: REDUCING Obs No. ', int2str(i),':',int2str(j), '  of ',int2str(length(start_dates)),':',int2str(no_chunks_reqd),'   ->  ',...
0199                     new_start_string, ' --- ',new_end_string]);
0200                 dcal2 = reduceData(d, path_to_redscript, 0, outputdir,sched_name);
0201             catch exception
0202                 error_string = ['day_reduce:: FAILED TO REDUCE ',int2str(i),':',int2str(j), '  of ',int2str(length(start_dates)),':',int2str(no_chunks_reqd),'   ->  ',...
0203                     new_start_string, ' --- ',new_end_string];
0204                 
0205                 disp(error_string);
0206                 rep = getReport(exception, 'basic');
0207                 fprintf(errfile,'%s\n',error_string);
0208                 fprintf(errfile,'%s\n',rep);
0209                 continue;
0210             end; %endtry
0211             
0212         end; %endfor no_chunk_reqd
0213         
0214     end; % end if > 3 hrs
0215 end; % end main for
0216 
0217 disp( ['day_reduce:: FINISHED ',date,' output dir is ',outputdir] );
0218 
0219 % Now try running the antenna health script
0220 
0221 if runhealth ~= 0
0222     try
0223         disp( ['day_reduce:: Attempting to run antenna health script : location',antenna_health_dir]);
0224         antHealth(date,antenna_health_dir);
0225     catch exception
0226         error_string = ['day_reduce:: FAILED to run antenna health script : location',antenna_health_dir];
0227         disp(error_string);
0228         rep = getReport(exception, 'basic');
0229         fprintf(errfile,'%s\n',error_string);
0230         fprintf(errfile,'%s\n',rep);
0231 
0232     end; %endtry
0233 else
0234     disp('day_reduce:: NOT attempting to run antenna health script.');
0235 end
0236 
0237 
0238 fclose(errfile);
0239 
0240 if(~isempty(final_dir))
0241   txt = sprintf('mv reduc/%s/ %s/', outputdir, final_dir);
0242   unix(txt);
0243 end
0244 
0245 
0246 return;
0247 
0248 
0249 
0250 function processed_string = remove_colon_datestring(date_string)
0251 splitline = regexp(char(date_string), ':', 'split');
0252 processed_string=[char(splitline(1)),' ',char(splitline(2)),':',char(splitline(3)),':',char(splitline(4))];
0253 
0254 function processed_string = insert_colon_datestring(date_string)
0255 splitline = regexp(char(date_string), ' ', 'split');
0256 processed_string=[char(splitline(1)),':',char(splitline(2))];