MakeComCamCalibrations¶

class lsst.ts.externalscripts.maintel.MakeComCamCalibrations(index=1)¶

Bases: BaseMakeCalibrations

Class for taking images, constructing, verifying, and certifying combined calibrations with LSSTComCam.

This class takes bias, darks, and flat exposures with LSSTComCam, constructs a combined calibration for each image type by calling the appropiate pipetask via OCPS, and then verifies and certifies each combined calibration. It also optionally produces defects and Photon Transfer Curves. “

Attributes Summary

`camera`
`checkpoints`	Get the checkpoints at which to pause and stop.
`detectors`	Array with detector IDs
`domain`
`group_id`	Get the group ID (a `str`), or "" if not set.
`image_in_oods`	OODS imageInOODS event.
`instrument_name`	String with instrument name for pipeline task
`n_detectors`	Number of detectors
`ocps`
`ocps_group`	Access the Remote OCPS Groups in the constructor.
`pipeline_instrument`	String with instrument name for pipeline yaml file
`state`	Get the current state.
`state_name`	Get the name of the current `state`.state.

Methods Summary

`amain`(**kwargs)	Run the script from the command line.
`analyze_report_check_verify_stats`(im_type, ...)	Report the results from `check_verification_stats`
`arun`([checkpoint])
`assert_state`(action, states)	Assert that the current state is in `states` and the script is not exiting.
`build_verification_report_summary`(...)	Helper function to print verification results.
`call_pipetask`(image_type)	Call pipetasks via the OCPS.
`certify_calib`(image_type, job_id_calib)	Certify the calibration.
`check_verification_stats`(image_type, ...[, ...])	Check verification statistics.
`checkpoint`([name])	Await this at any "nice" point your script can be paused or stopped.
`cleanup`()	Perform final cleanup, if any.
`close`([exception, cancel_start])	Shut down, clean up resources and set done_task done.
`close_tasks`()	Shut down pending tasks.
`configure`(config)	Configure the script.
`count_failed_verification_tests`(...)	Count number of tests that failed cp_verify.
`do_configure`(data)	Configure the currently loaded script.
`do_resume`(data)	Resume the currently paused script.
`do_run`(data)	Run the script and quit.
`do_setAuthList`(data)	Update the authorization list.
`do_setCheckpoints`(data)	Set or clear the checkpoints at which to pause and stop.
`do_setGroupId`(data)	Set or clear the group_id attribute.
`do_setLogLevel`(data)	Set logging level.
`do_stop`(data)	Stop the script.
`get_detectors_string`(detector_array)	Get a detector string from a detector array.
`get_exposure_id`(obsid)	Parse obsid into an exposure id.
`get_instrument_configuration`()	Get instrument configuration
`get_pipetask_parameters_bias`()	Get necessary information to run the bias generation pipetask.
`get_pipetask_parameters_dark`()	Get necessary information to run the dark generation pipetask.
`get_pipetask_parameters_defects`()	Get necessary information to run the defects generation pipetask.
`get_pipetask_parameters_flat`()	Get necessary information to run the flat generation pipetask.
`get_pipetask_parameters_ptc`()	Get necessary information to run the ptc generation pipetask.
`get_pipetask_parameters_verification_bias`(...)	Get necessary information to run the bias verification pipetask
`get_pipetask_parameters_verification_dark`(...)	Get necessary information to run the dark verification pipetask
`get_pipetask_parameters_verification_flat`(...)	Get necessary information to run the flat verification pipetask.
`get_schema`()	Return a jsonschema to validate configuration, as a `dict`.
`image_in_oods_callback`(data)	Callback function to check images are in oods
`make_from_cmd_line`(**kwargs)	Make a script from command-line arguments.
`next_supplemented_group_id`()	Return the group ID supplemented with a new subgroup.
`put_log_level`()	Output the logLevel event.
`report_gains_from_flat_pairs`(job_id_calib)	Print gains estimated form flat pairs.
`run`()
`set_exp_times_per_im_type`(image_type)	Define exp_times and n_images according to image type.
`set_metadata`(metadata)	Set estimated duration of the script.
`set_state`([state, reason, keep_old_reason, ...])	Set the script state.
`start`()	Finish construction.
`start_phase2`()	Additional work after `start` before fully started.
`take_image_type`(image_type, exp_times)	Take exposures and build exposure set.
`take_images`(image_type)	Take images with instrument.
`verify_calib`(image_type, job_id_calib)	Verify the calibration.

Attributes Documentation

camera¶

checkpoints¶

Get the checkpoints at which to pause and stop.

Returns self.evt_checkpoints.data which has these fields:

pause: checkpoints at which to pause, a regular expression
stop: checkpoints at which to stop, a regular expression

detectors¶: Array with detector IDs

domain¶

group_id¶: Get the group ID (a str), or “” if not set.

image_in_oods¶: OODS imageInOODS event.

instrument_name¶: String with instrument name for pipeline task

n_detectors¶: Number of detectors

ocps¶

ocps_group¶

Access the Remote OCPS Groups in the constructor.

The OCPS index will be 2 for LSSTComCam: OCPS:2.

pipeline_instrument¶: String with instrument name for pipeline yaml file

state¶

Get the current state.

Returns self.evt_state.data, which has these fields:

state: lsst.ts.idl.enums.Script.ScriptState
The current state.
lastCheckpoint: str
Name of most recently seen checkpoint.
reason: str
Reason for this state, if any.
numCheckpoints: int
The number of checkpoints seen.

state_name¶: Get the name of the current state.state.

Methods Documentation

async classmethod amain(**kwargs: Any) → None¶

Run the script from the command line.

Parameters:

kwargs: Keyword arguments for the script constructor. Must not include index. Ignored (other than testing for index) if the command specifies --schema.

Raises:

RuntimeError: If kwargs includes index.

Notes

The final return code will be:

0 if final state is lsst.ts.idl.enums.Script.ScriptState.DONE or lsst.ts.idl.enums.Script.ScriptState.STOPPED.
1 if final state is anything else, or if script.done_task is an exception (which would be raised by the script’s cleanup code).

Issues a RuntimeWarning if script.done_task is an exception and the final script state is anything other than Failed. This should never happen.

async analyze_report_check_verify_stats(im_type, report_check_verify_stats, job_id_verify, job_id_calib)¶

Report the results from check_verification_stats

Parameters:

im_typestr: Image or calibration type.
report_check_verify_statsdict: Dictionary returned by check_verification_stats.
job_id_verifystr: Job ID returned by OCPS during previous calibration verification pipetask call.
job_id_calibstr: Job ID returned by OCPS during previous calibration generation pipetask call. If “generate_calibrations” is False, this variable is “None”.

Notes

Suported calibrations: see self.pipetask_parameters_verification.

async arun(checkpoint=False)¶

assert_state(action: str, states: Sequence[ScriptState]) → None¶

Assert that the current state is in states and the script is not exiting.

Parameters:

actionstr: Description of what you want to do.
stateslist [lsst.ts.idl.enums.Script.ScriptState]: Allowed states.

async build_verification_report_summary(report_check_verify_stats)¶

Helper function to print verification results.

Parameters:

report_check_verify_statsdict: Dictionary with verification results:
certify_calibbool: Booolean indicating if the calibration should be certified or not.
num_stat_errorsdict`[`str][str] or None: Dictionary with the total number of tests failed per exposure and per cp_verify test type. If there are not any tests that failed, None will be returned.
failure_thresholdsdict`[`str][int] or None: Dictionary reporting the different thresholds used to decide whether a calibration should be certified or not (see Notes below). If there are not any tests that failed, None will be returned.
verify_statsdict: Statistics from cp_verify.

Returns:

final_report_stringstr: String with full report.

async call_pipetask(image_type)¶

Call pipetasks via the OCPS.

Parameters:

image_typestr: Image or calibration type.

Returns:

responsedict: Dictionary with the final OCPS status.

Notes

Suported calibrations: see self.pipetask_parameters

async certify_calib(image_type, job_id_calib)¶

Certify the calibration.

Parameters:

image_typestr: Image or calibration type.
jod_id_calibstr: Job ID returned by OCPS during previous calibration generation pipetask call.

Raises:

RuntimeErrorError in running the butler certification command.

Notes

The calibration will certified for use with a timespan that indicates its validity range.

Suported calibrations: see self.supported_calibrations_certification.

async check_verification_stats(image_type, job_id_verify, job_id_calib=None)¶

Check verification statistics.

Parameters:

image_type:: Image or calibration type.
jod_id_calibstr, optional: Job ID returned by OCPS during previous calibration generation pipetask call.
job_id_verifystr: Job ID returned by OCPS during previous calibration verification pipetask call.

Returns:

report_check_verify_statsdict: Dictionary with results:
certify_calibbool: Booolean indicating if the calibration should be certified or not.
num_stat_errorsdict`[`str][str] or None: Dictionary with the total number of tests failed per exposure and per cp_verify test type. If there are not any tests that failed, None will be returned.
failure_thresholdsdict`[`str][int] or None: Dictionary reporting the different thresholds used to decide whether a calibration should be certified or not (see Notes below). If there are not any tests that failed, None will be returned.
verify_statsdict: Statistics from cp_verify.

Notes

When generate_calibrations=False, verification will be performed using the combined calibrations in self.config.input_collections_verify_bias, self.config.input_collections_verify_dark, and/or self.config.input_collections_verify_bias, and this script won’t have generated any combined calibrations from the images taken. Therefore, job_id_calib can be None.

Suported calibrations: see self.pipetask_parameters_verification.

async checkpoint(name: str = '') → None¶

Await this at any “nice” point your script can be paused or stopped.

Parameters:

namestr, optional: Name of checkpoint; “” if it has no name.

Raises:

RuntimeError: If the state is not ScriptState.RUNNING. This likely means you called checkpoint from somewhere other than run.
RuntimeError: If _run_task is None or done. This probably means your code incorrectly set the state.

async cleanup() → None¶

Perform final cleanup, if any.

This method is called as the script state is exiting, unless the script had not yet started to run, or the script process is aborted by SIGTERM or SIGKILL.

async close(exception: Exception | None = None, cancel_start: bool = True) → None¶

Shut down, clean up resources and set done_task done.

May be called multiple times. The first call closes the Controller; subsequent calls wait until the Controller is closed.

Subclasses should override close_tasks instead of close, unless you have a good reason to do otherwise.

Parameters:

exceptionException, optional: The exception that caused stopping, if any, in which case the self.done_task exception is set to this value. Specify None for a normal exit, in which case the self.done_task result is set to None.
cancel_startbool, optional: Cancel the start task? Leave this true unless calling this from the start task.

Notes

Removes the SAL log handler, calls close_tasks to stop all background tasks, pauses briefly to allow final SAL messages to be sent, then closes the dds domain.

async close_tasks() → None¶

Shut down pending tasks. Called by close.

Perform all cleanup other than disabling logging to SAL and closing the dds domain.

async configure(config)¶

Configure the script.

Parameters:

config: `types.SimpleNamespace`: Configuration data. See get_schema for information about data structure.

async count_failed_verification_tests(verify_stats, max_number_failures_per_detector_per_test)¶

Count number of tests that failed cp_verify.

Parameters:

verify_statsdict: Statistics from cp_verify.
max_number_failures_per_detector_per_testint: Minimum number of verification tests per detector per exposure per test type that should pass to certify the combined calibration.

Returns:

certify_calibbool: Boolean assessing whether the calibration should be certified.
total_counter_failed_testsdict`[`str][str] or None.: Dictionary with the total number of tests failed per exposure and per cp_verify test type. If there are not any tests that failed, None will be returned.
thresholdsdict`[`str][int] or None: Dictionary reporting the different thresholds used to decide whether a calibration should be certified or not (see Notes below). If there are not any tests that failed, None will be returned.

Notes

For at least one type of test, if the majority of tests fail in the majority of detectors and the majority of exposures, then don’t certify the calibration.

Suported calibrations: see self.pipetask_parameters_verification.

async do_configure(data: BaseMsgType) → None¶

Configure the currently loaded script.

Parameters:

datacmd_configure.DataType: Configuration.

Raises:

base.ExpectedError: If self.state.state is not lsst.ts.idl.enums.Script.ScriptState.UNCONFIGURED.

Notes

This method does the following:

Parse the config field as yaml-encoded dict and validate it (including setting default values).
Call configure.
Set the pause and stop checkpoints.
Set the log level if data.logLevel != 0.
Call set_metadata.
Output the metadata event.
Change the script state to lsst.ts.idl.enums.Script.ScriptState.CONFIGURED.

async do_resume(data: BaseMsgType) → None¶

Resume the currently paused script.

Parameters:

datacmd_resume.DataType: Ignored.

Raises:

base.ExpectedError: If self.state.state is not lsst.ts.idl.enums.Script.ScriptState.PAUSED.

async do_run(data: BaseMsgType) → None¶

Run the script and quit.

The script must have been configured and the group ID set.

Parameters:

datacmd_run.DataType: Ignored.

Raises:

base.ExpectedError: If self.state.state is not lsst.ts.idl.enums.Script.ScriptState.CONFIGURED. If self.group_id is blank.

async do_setAuthList(data: BaseMsgType) → None¶

Update the authorization list.

Parameters:

datacmd_setAuthList.DataType: Authorization lists.

Notes

Add items if the data string starts with “+”, ignoring duplicates (both with respect to the existing items and within the data string). Remove items if the data string starts with “-”, ignoring missing items (items specified for removal that do not exist). Ignore whitespace after each comma and after the +/- prefix.

async do_setCheckpoints(data: BaseMsgType) → None¶

Set or clear the checkpoints at which to pause and stop.

This command is deprecated. Please set the checkpoints using the configure command.

Parameters:

datacmd_setCheckpoints.DataType: Names of checkpoints for pausing and stopping, each a single regular expression; “” for no checkpoints, “.*” for all.

Raises:

base.ExpectedError

If self.state.state is not one of:

lsst.ts.idl.enums.Script.ScriptState.UNCONFIGURED
lsst.ts.idl.enums.Script.ScriptState.CONFIGURED
lsst.ts.idl.enums.Script.ScriptState.RUNNING
lsst.ts.idl.enums.Script.ScriptState.PAUSED

async do_setGroupId(data: BaseMsgType) → None¶

Set or clear the group_id attribute.

The script must be in the Configured state. This command may be called multiple times. It is typically called when the script reaches the top position on the script queue.

Parameters:

datacmd_setGroupId.DataType: Group ID, or “” to clear the group ID.

Raises:

base.ExpectedError: If state.state is not lsst.ts.idl.enums.Script.ScriptState.CONFIGURED.

async do_setLogLevel(data: BaseMsgType) → None¶

Set logging level.

Parameters:

datacmd_setLogLevel.DataType: Logging level.

async do_stop(data: BaseMsgType) → None¶

Stop the script.

Parameters:

datacmd_stop.DataType: Ignored.

Notes

This is a no-op if the script is already exiting. This does not wait for _exit to run.

get_detectors_string(detector_array)¶

Get a detector string from a detector array. Convert a detector array of the form [0, 1, 2, …] into a string of the form “(0, 1, 2, …)” to be used by pipetasks.

Parameters:

detector_arrayarray: Array with the detector IDs

Returns:

detectors_stringstr: Detector IDs list in the form “(0, 1, 2..)”

static get_exposure_id(obsid)¶

Parse obsid into an exposure id. Convert string in the format ??_?_YYYYMMDD_012345 into an integer like YYYYMMDD12345.

Parameters:

obsidstr: Observation id in the format ??_?_YYYYMMDD_012345, e.g., AT_O_20220406_000007.

Returns:

int: Exposure id in the format YYYYMMDD12345, e.g., 2022040600007.

get_instrument_configuration()¶

Get instrument configuration

Returns:

instrument_configuration: dict: Dictionary with instrument configuration.

get_pipetask_parameters_bias()¶

Get necessary information to run the bias generation pipetask.

Returns:

cpBias.yamlstr: cp_pipe bias pipeline.
config_stringstr: Bias pipetask configuration for OCPS.
exposure_ids_biaslist`[`int]: List of bias exposure IDs.

get_pipetask_parameters_dark()¶

Get necessary information to run the dark generation pipetask.

Returns:

cpDark.yamlstr: cp_pipe dark pipeline.
config_stringstr: Dark pipetask configuration for OCPS.
exposure_ids_darklist`[`int]: List of dark exposure IDs.

get_pipetask_parameters_defects()¶

Get necessary information to run the defects generation pipetask.

Returns:

findDefects.yamlstr: cp_pipe defects pipeline.
config_stringstr: Defects pipetask configuration for OCPS.
exposure_ids_defectslist`[`int]: List of dark and flat exposure IDs.

get_pipetask_parameters_flat()¶

Get necessary information to run the flat generation pipetask.

Returns:

cpFlat.yamlstr: cp_pipe flat pipeline.
config_stringstr: Flat pipetask configuration for OCPS.
exposure_ids_flatlist`[`int]: List of flat exposure IDs.

get_pipetask_parameters_ptc()¶

Get necessary information to run the ptc generation pipetask.

Returns:

cpPtc.yaml or cpPtc.yaml#gainFromFlatPairsstr: cp_pipe PTC or gain from pairs pipeline.
config_stringstr: PTC or gain from pairs pipetask configuration for OCPS.
exposure_ids_ptclist`[`int]: List of flat exposure IDs for PTC or gain estimation.

get_pipetask_parameters_verification_bias(job_id_calib)¶

Get necessary information to run the bias verification pipetask

Parameters:

jod_id_calibstr: Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Returns:

pipe_yamlstr: cp_verify bias pipeline.
config_stringstr: Bias verification pipetask configuration for OCPS.
exposure_ids_biaslist`[`int]: List of bias exposure IDs.

get_pipetask_parameters_verification_dark(job_id_calib)¶

Get necessary information to run the dark verification pipetask

Parameters:

jod_id_calibstr: Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Returns:

pipe_yamlstr: cp_verify dark pipeline.
config_stringstr: Dark verification pipetask configuration for OCPS.
exposure_ids_darklist`[`int]: List of dark exposure IDs.

get_pipetask_parameters_verification_flat(job_id_calib)¶

Get necessary information to run the flat verification pipetask.

Parameters:

jod_id_calibstr: Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Returns:

pipe_yamlstr: cp_verify flat pipeline.
config_stringstr: Flat verification pipetask configuration for OCPS.
exposure_ids_flatlist`[`int]: List of flat exposure IDs.

classmethod get_schema()¶

Return a jsonschema to validate configuration, as a dict.

Please provide default values for all fields for which defaults make sense. This makes the script easier to use.

If your script has no configuration then return None, in which case the config field of the configure command must be an empty string.

async image_in_oods_callback(data)¶

Callback function to check images are in oods

Parameters:

dataevt_imageInOODS.DataType: OODS, imageInOODS event sample.

classmethod make_from_cmd_line(**kwargs: Any) → lsst.ts.salobj.base_script.BaseScript | None¶

Make a script from command-line arguments.

Return None if --schema specified.

Parameters:

kwargs: Keyword arguments for the script constructor. Must not include index.

Raises:

RuntimeError: If kwargs includes index.

next_supplemented_group_id() → str¶

Return the group ID supplemented with a new subgroup.

The returned string has this format: f”{self.group_id}#{subgroup_id}”, where subgroup_id is an integer that starts at 1 and is incremented for every call to this method.

Raises:

RuntimeError: If there is no group ID.

async put_log_level() → None¶: Output the logLevel event.

async report_gains_from_flat_pairs(job_id_calib)¶

Print gains estimated form flat pairs.

Parameters:

jod_id_calibstr: Job ID returned by the OCPS after running the “GAIN” or “PTC” pipetasks

Notes

The “PTC” and “GAIN” tasks are defined by the “cp_pipe” pipelines “cpPtc.yaml” and “cpPtc.yaml#genGainsFromFlatPairs”, respectively.

async run()¶

async set_exp_times_per_im_type(image_type)¶

Define exp_times and n_images according to image type.

Parameters:

image_typestr: Image type. One of [“BIAS”, “DARK”, “FLAT”].

set_metadata(metadata)¶: Set estimated duration of the script.

async set_state(state: lsst.ts.xml.enums.Script.ScriptState | int | None = None, reason: str | None = None, keep_old_reason: bool = False, force_output: bool = False) → None¶

Set the script state.

Parameters:

stateScriptState or int, optional: New state, or None if no change
reasonstr, optional: Reason for state change. None for no new reason.
keep_old_reasonbool: If true, keep old reason; append the reason argument after “;” if it is is a non-empty string. If false replace with reason, or “” if reason is None.
force_outputbool, optional: If true the output even if not changed.

Notes

The lastCheckpoint field is set from self.last_checkpoint, and the numCheckpoints field is set from self.num_checkpoints.

async start() → None¶: Finish construction.

async start_phase2() → None¶

Additional work after start before fully started.

Used by BaseCsc to transition to handle the initial state.

async take_image_type(image_type, exp_times)¶

Take exposures and build exposure set.

Parameters:

image_typestr: Image type. One of [“BIAS”, “DARK”, “FLAT”].
exp_timeslist: List of exposure times.

Returns:

Tuple with exposure IDs.

async take_images(image_type)¶

Take images with instrument.

Parameters:

image_typestr: Image type. One of [“BIAS”, “DARK”, “FLAT”].

Returns:

exposurestuple: Tuple with the IDs of the exposures taken.

async verify_calib(image_type, job_id_calib)¶

Verify the calibration.

Parameters:

image_typestr: Image type.
jod_id_calibstr: Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Notes

The verification step runs tests in cp_verify that check the metrics in DMTN-101.

Suported calibrations: see self.pipetask_parameters_verification.

Navigation

MakeComCamCalibrations¶