| | |
- auto()
- Process all platforms, all packages, latest data
Notes
-----
1. determine which platforms (all platforms with currently active
config files i.e. config_end_date is None
2. for each platform
get latest config
for each package
(determine process for 'latest' data) copy to new area when grabbed
parse recent data
yyyy_mm is the current month
load this months netcdf, if new month, create this months netcdf
update modified date and append new data in netcdf
- create_spin_list(plats, packs, dates, config_dir='/opt/env/haines/dataproc/raw2proc')
- create list of params needed to run manual() mutiple ways
:Returns:
spin_list : list of three-tuple each tuple with form (platform, package, yyyy_mm)
Notes
-----
1. plats -- 'ALL' or ['b1', 'b2']
2. packs -- 'ALL' or ['ctd1', 'ctd2']
3. dates -- 'ALL' or ['2011_11', '2011_12'] or [dt.datetime(2006,1,1), dt.nowutc()]
For each platform determin packages for given dates
also a good way to get listing platforms and packages for specified dates
- find_active_configs(config_dir='/opt/env/haines/dataproc/raw2proc')
- Find which configuration files are active
:Returns:
cns : list of str
List of configurations that overlap with desired month
If empty [], no configs were found
- find_configs(platform, yyyy_mm, config_dir='')
- Find which configuration files for specified platform and month
:Parameters:
platform : string
Platfrom id to process (e.g. 'bogue')
yyyy_mm : string
Year and month of data to process (e.g. '2007_07')
:Returns:
cns : list of str
List of configurations that overlap with desired month
If empty [], no configs were found
- find_raw(si, yyyy_mm)
- Determine which list of raw files to process for month
- get_all_packages(platform, config_dir='/opt/env/haines/dataproc/raw2proc')
- Get all package ids -- all defined packages in sensor_info{} from all configs for the platform
:Returns:
sids : list of str
Sorted list of all the sensor ids for package
- get_all_platform_configs(platform, config_dir='/opt/env/haines/dataproc/raw2proc')
- Get all the config files for a platform
:Returns:
cns : list of config names
Sorted list of all the sensor ids for package
- get_all_platforms(config_dir='/opt/env/haines/dataproc/raw2proc')
- Get all platform ids
:Returns:
pids : list of str
Sorted list of all the platforms
- get_config(name)
- Usage Example >>>sensor_info = get_config('bogue_config_20060918.sensor_info')
- get_config_dates(pi)
- Get datetime of both start and end setting within config file
Example
-------
>>> pi = get_config(cn+'.platform_info')
>>> (config_start_dt, config_end_dt) = get_config_dates(pi)
- get_config_packages(cn)
- Get active packages set in platform_info{} from specific config file
:Returns:
sids : list of str
Sorted (default) or unsorted list of all the sensor ids for package
If empty [], no platform ids were found
- import_parser(name)
- import_processors(mod_name)
- list_months(dts, dte)
- list of datetimes for all months inclusively within given date range
- load_data(inFile)
- manual(platform, package, yyyy_mm)
- Process data for specified platform, sensor package, and month
Notes
-----
1. determine which configs
2. for each config for specific platform
if have package in config
which raw files
- process(pi, si, raw_files, yyyy_mm)
- raw2proc(proctype, platform=None, package=None, yyyy_mm=None)
- Process data either in auto-mode or manual-mode
If auto-mode, process newest data for all platforms, all
sensors. Otherwise in manual-mode, process data for specified
platform, sensor package, and month.
:Parameters:
proctype : string
'auto' or 'manual' or 'spin'
platform : string
Platfrom id to process (e.g. 'bogue')
package : string
Sensor package id to process (e.g. 'adcp')
yyyy_mm : string
Year and month of data to process (e.g. '2007_07')
Examples
--------
>>> raw2proc(proctype='manual', platform='bogue', package='adcp', yyyy_mm='2007_06')
>>> raw2proc('manual', 'bogue', 'adcp', '2007_06')
Spin
----
platform can be list of platforms or 'ALL'
package can be list packages or 'ALL'
yyyy_mm can be list of months, or datetime range
>>> raw2proc('spin', ['b1','b2'], ['ctd1', 'ctd2'], ['2011_11'])
>>> raw2proc('spin', ['b1','b2'], ['ctd1', 'ctd2'], 'ALL')
>>> raw2proc('spin', ['b1','b2'], ['ctd1', 'ctd2'], [datetime(2011,11,1), datetime(2012,4,1)])
>>> raw2proc('spin', ['b1','b2'], 'ALL', 'ALL')
Not a good idea but this will reprocess all the data from level0
>>> raw2proc('spin', 'ALL', 'ALL', 'ALL')
- spin(spin_list)
- wrapper to run manual() for multiple months
- uniqify(seq)
- which_raw(pi, raw_files, dts)
- Further limit file names based on configuration file timeframe
|