MakeComCamCalibrations

class lsst.ts.externalscripts.maintel.MakeComCamCalibrations(index=1)

Bases: BaseMakeCalibrations

Class for taking images, constructing, verifying, and certifying combined calibrations with LSSTComCam.

This class takes bias, darks, and flat exposures with LSSTComCam, constructs a combined calibration for each image type by calling the appropiate pipetask via OCPS, and then verifies and certifies each combined calibration. It also optionally produces defects and Photon Transfer Curves.

Attributes Summary

camera

checkpoints

Get the checkpoints at which to pause and stop.

detectors

Array with detector IDs

domain

group_id

Get the group ID (a str), or "" if not set.

image_in_oods

OODS imageInOODS event.

instrument_name

String with instrument name for pipeline task

n_detectors

Number of detectors

obs_id

Get the block ID (a str), or "" if not set.

ocps

ocps_group

Access the Remote OCPS Groups in the constructor.

pipeline_instrument

String with instrument name for pipeline yaml file

state

Get the current state.

state_name

Get the name of the current state.state.

Methods Summary

amain(**kwargs)

Run the script from the command line.

analyze_report_check_verify_stats(im_type, ...)

Report the results from check_verification_stats

assert_state(action, states)

Assert that the current state is in states and the script is not exiting.

build_verification_report_summary(...)

Helper function to print verification results.

call_pipetask(image_type)

Call pipetasks via the OCPS.

certify_calib(image_type, job_id_calib)

Certify the calibration.

check_verification_stats(image_type, ...[, ...])

Check verification statistics.

checkpoint([name])

Await this at any "nice" point your script can be paused or stopped.

cleanup()

Perform final cleanup, if any.

close([exception, cancel_start])

Shut down, clean up resources and set done_task done.

close_tasks()

Shut down pending tasks.

configure(config)

Configure the script.

count_failed_verification_tests(...)

Count number of tests that failed cp_verify.

do_configure(data)

Configure the currently loaded script.

do_resume(data)

Resume the currently paused script.

do_run(data)

Run the script and quit.

do_setCheckpoints(data)

Set or clear the checkpoints at which to pause and stop.

do_setGroupId(data)

Set or clear the group_id attribute.

do_setLogLevel(data)

Set logging level.

do_stop(data)

Stop the script.

get_detectors_string(detector_array)

Get a detector string from a detector array.

get_exposure_id(obsid)

Parse obsid into an exposure id.

get_instrument_configuration()

Get instrument configuration

get_obs_id()

Get obs id from camera obs id server.

get_pipetask_parameters_bias()

Get necessary information to run the bias generation pipetask.

get_pipetask_parameters_dark()

Get necessary information to run the dark generation pipetask.

get_pipetask_parameters_defects()

Get necessary information to run the defects generation pipetask.

get_pipetask_parameters_flat()

Get necessary information to run the flat generation pipetask.

get_pipetask_parameters_ptc()

Get necessary information to run the ptc generation pipetask.

get_pipetask_parameters_verification_bias(...)

Get necessary information to run the bias verification pipetask

get_pipetask_parameters_verification_dark(...)

Get necessary information to run the dark verification pipetask

get_pipetask_parameters_verification_flat(...)

Get necessary information to run the flat verification pipetask.

get_schema()

Return a jsonschema to validate configuration, as a dict.

image_in_oods_callback(data)

Callback function to check images are in oods

make_from_cmd_line(**kwargs)

Make a script from command-line arguments.

next_supplemented_group_id()

Return the group ID supplemented with a new subgroup.

process_calibration(calib_type)

Generate and certify a specific type of calibration.

process_images(im_type)

Generate and optionally verify and certify calibrations for a given image type.

process_verification(im_type, job_id_calib)

Verify and certify the generated calibration for a given image type.

program_reason()

Context manager to publish appropriate checkpoints with program and reason.

put_log_level()

Output the logLevel event.

report_gains_from_flat_pairs(job_id_calib)

Print gains estimated form flat pairs.

run()

Override base script run to encapsulate execution with appropriate checkpoints.

run_block()

save_test_case()

Save test case to the LFA.

set_exp_times_per_im_type(image_type)

Define exp_times and n_images according to image type.

set_metadata(metadata)

Set estimated duration of the script.

set_state([state, reason, keep_old_reason, ...])

Set the script state.

start()

Finish construction.

start_phase2()

Additional work after start before fully started.

take_image_type(image_type, exp_times)

Take exposures and build exposure set.

take_images(image_type)

Take images with instrument.

test_case_step([comment])

Context manager to handle test case steps.

verify_calib(image_type, job_id_calib)

Verify the calibration.

wait_for_background_tasks()

Await the completion of all background calibration and verification tasks.

Attributes Documentation

camera
checkpoints

Get the checkpoints at which to pause and stop.

Returns self.evt_checkpoints.data which has these fields:

  • pause: checkpoints at which to pause, a regular expression

  • stop: checkpoints at which to stop, a regular expression

detectors

Array with detector IDs

domain
group_id

Get the group ID (a str), or “” if not set.

image_in_oods

OODS imageInOODS event.

instrument_name

String with instrument name for pipeline task

n_detectors

Number of detectors

obs_id

Get the block ID (a str), or “” if not set.

ocps
ocps_group

Access the Remote OCPS Groups in the constructor.

The OCPS index will be 2 for LSSTComCam: OCPS:2.

pipeline_instrument

String with instrument name for pipeline yaml file

state

Get the current state.

Returns self.evt_state.data, which has these fields:

  • state: lsst.ts.idl.enums.Script.ScriptState

    The current state.

  • lastCheckpoint: str

    Name of most recently seen checkpoint.

  • reason: str

    Reason for this state, if any.

  • numCheckpoints: int

    The number of checkpoints seen.

state_name

Get the name of the current state.state.

Methods Documentation

async classmethod amain(**kwargs: Any) None

Run the script from the command line.

Parameters:
kwargs

Keyword arguments for the script constructor. Must not include index. Ignored (other than testing for index) if the command specifies --schema.

Raises:
RuntimeError

If kwargs includes index.

Notes

The final return code will be:

  • 0 if final state is lsst.ts.idl.enums.Script.ScriptState.DONE or lsst.ts.idl.enums.Script.ScriptState.STOPPED.

  • 1 if final state is anything else, or if script.done_task is an exception (which would be raised by the script’s cleanup code).

Issues a RuntimeWarning if script.done_task is an exception and the final script state is anything other than Failed. This should never happen.

async analyze_report_check_verify_stats(im_type, report_check_verify_stats, job_id_verify, job_id_calib)

Report the results from check_verification_stats

Parameters:
im_typestr

Image or calibration type.

report_check_verify_statsdict

Dictionary returned by check_verification_stats.

job_id_verifystr

Job ID returned by OCPS during previous calibration verification pipetask call.

job_id_calibstr

Job ID returned by OCPS during previous calibration generation pipetask call. If “generate_calibrations” is False, this variable is “None”.

Notes

Suported calibrations: see self.pipetask_parameters_verification.

assert_state(action: str, states: Sequence[ScriptState]) None

Assert that the current state is in states and the script is not exiting.

Parameters:
actionstr

Description of what you want to do.

stateslist [lsst.ts.idl.enums.Script.ScriptState]

Allowed states.

async build_verification_report_summary(report_check_verify_stats)

Helper function to print verification results.

Parameters:
report_check_verify_statsdict

Dictionary with verification results:

certify_calibbool

Booolean indicating if the calibration should be certified or not.

num_stat_errorsdict`[`str][str] or None

Dictionary with the total number of tests failed per exposure and per cp_verify test type. If there are not any tests that failed, None will be returned.

failure_thresholdsdict`[`str][int] or None

Dictionary reporting the different thresholds used to decide whether a calibration should be certified or not (see Notes below). If there are not any tests that failed, None will be returned.

verify_statsdict

Statistics from cp_verify.

Returns:
final_report_stringstr

String with full report.

async call_pipetask(image_type)

Call pipetasks via the OCPS.

Parameters:
image_typestr

Image or calibration type.

Returns:
responsedict

Dictionary with the final OCPS status.

Notes

Suported calibrations: see self.pipetask_parameters

async certify_calib(image_type, job_id_calib)

Certify the calibration.

Parameters:
image_typestr

Image or calibration type.

jod_id_calibstr

Job ID returned by OCPS during previous calibration generation pipetask call.

Raises:
RuntimeErrorError in running the butler certification command.

Notes

The calibration will certified for use with a timespan that indicates its validity range.

Suported calibrations: see self.supported_calibrations_certification.

async check_verification_stats(image_type, job_id_verify, job_id_calib=None)

Check verification statistics.

Parameters:
image_type:

Image or calibration type.

jod_id_calibstr, optional

Job ID returned by OCPS during previous calibration generation pipetask call.

job_id_verifystr

Job ID returned by OCPS during previous calibration verification pipetask call.

Returns:
report_check_verify_statsdict

Dictionary with results:

certify_calibbool

Booolean indicating if the calibration should be certified or not.

num_stat_errorsdict`[`str][str] or None

Dictionary with the total number of tests failed per exposure and per cp_verify test type. If there are not any tests that failed, None will be returned.

failure_thresholdsdict`[`str][int] or None

Dictionary reporting the different thresholds used to decide whether a calibration should be certified or not (see Notes below). If there are not any tests that failed, None will be returned.

verify_statsdict

Statistics from cp_verify.

Notes

When generate_calibrations=False, verification will be performed using the combined calibrations in self.config.input_collections_verify_bias, self.config.input_collections_verify_dark, and/or self.config.input_collections_verify_bias, and this script won’t have generated any combined calibrations from the images taken. Therefore, job_id_calib can be None.

Suported calibrations: see self.pipetask_parameters_verification.

async checkpoint(name: str = '') None

Await this at any “nice” point your script can be paused or stopped.

Parameters:
namestr, optional

Name of checkpoint; “” if it has no name.

Raises:
RuntimeError

If the state is not ScriptState.RUNNING. This likely means you called checkpoint from somewhere other than run.

RuntimeError

If _run_task is None or done. This probably means your code incorrectly set the state.

async cleanup() None

Perform final cleanup, if any.

This method is called as the script state is exiting, unless the script had not yet started to run, or the script process is aborted by SIGTERM or SIGKILL.

async close(exception: Exception | None = None, cancel_start: bool = True) None

Shut down, clean up resources and set done_task done.

May be called multiple times. The first call closes the Controller; subsequent calls wait until the Controller is closed.

Subclasses should override close_tasks instead of close, unless you have a good reason to do otherwise.

Parameters:
exceptionException, optional

The exception that caused stopping, if any, in which case the self.done_task exception is set to this value. Specify None for a normal exit, in which case the self.done_task result is set to None.

cancel_startbool, optional

Cancel the start task? Leave this true unless calling this from the start task.

Notes

Removes the SAL log handler, calls close_tasks to stop all background tasks, pauses briefly to allow final SAL messages to be sent, then closes the dds domain.

async close_tasks() None

Shut down pending tasks. Called by close.

Perform all cleanup other than disabling logging to SAL and closing the dds domain.

async configure(config)

Configure the script.

Parameters:
config: `types.SimpleNamespace`

Configuration data. See get_schema for information about data structure.

async count_failed_verification_tests(verify_stats, max_number_failures_per_detector_per_test)

Count number of tests that failed cp_verify.

Parameters:
verify_statsdict

Statistics from cp_verify.

max_number_failures_per_detector_per_testint

Minimum number of verification tests per detector per exposure per test type that should pass to certify the combined calibration.

Returns:
certify_calibbool

Boolean assessing whether the calibration should be certified.

total_counter_failed_testsdict`[`str][str] or None.

Dictionary with the total number of tests failed per exposure and per cp_verify test type. If there are not any tests that failed, None will be returned.

thresholdsdict`[`str][int] or None

Dictionary reporting the different thresholds used to decide whether a calibration should be certified or not (see Notes below). If there are not any tests that failed, None will be returned.

Notes

For at least one type of test, if the majority of tests fail in the majority of detectors and the majority of exposures, then don’t certify the calibration.

Suported calibrations: see self.pipetask_parameters_verification.

async do_configure(data: BaseMsgType) None

Configure the currently loaded script.

Parameters:
datacmd_configure.DataType

Configuration.

Raises:
base.ExpectedError

If self.state.state is not lsst.ts.idl.enums.Script.ScriptState.UNCONFIGURED.

Notes

This method does the following:

  • Parse the config field as yaml-encoded dict and validate it (including setting default values).

  • Call configure.

  • Set the pause and stop checkpoints.

  • Set the log level if data.logLevel != 0.

  • Call set_metadata.

  • Output the metadata event.

  • Change the script state to lsst.ts.idl.enums.Script.ScriptState.CONFIGURED.

async do_resume(data: BaseMsgType) None

Resume the currently paused script.

Parameters:
datacmd_resume.DataType

Ignored.

Raises:
base.ExpectedError

If self.state.state is not lsst.ts.idl.enums.Script.ScriptState.PAUSED.

async do_run(data: BaseMsgType) None

Run the script and quit.

The script must have been configured and the group ID set.

Parameters:
datacmd_run.DataType

Ignored.

Raises:
base.ExpectedError

If self.state.state is not lsst.ts.idl.enums.Script.ScriptState.CONFIGURED. If self.group_id is blank.

async do_setCheckpoints(data: BaseMsgType) None

Set or clear the checkpoints at which to pause and stop.

This command is deprecated. Please set the checkpoints using the configure command.

Parameters:
datacmd_setCheckpoints.DataType

Names of checkpoints for pausing and stopping, each a single regular expression; “” for no checkpoints, “.*” for all.

Raises:
base.ExpectedError

If self.state.state is not one of:

  • lsst.ts.idl.enums.Script.ScriptState.UNCONFIGURED

  • lsst.ts.idl.enums.Script.ScriptState.CONFIGURED

  • lsst.ts.idl.enums.Script.ScriptState.RUNNING

  • lsst.ts.idl.enums.Script.ScriptState.PAUSED

async do_setGroupId(data: BaseMsgType) None

Set or clear the group_id attribute.

The script must be in the Configured state. This command may be called multiple times. It is typically called when the script reaches the top position on the script queue.

Parameters:
datacmd_setGroupId.DataType

Group ID, or “” to clear the group ID.

Raises:
base.ExpectedError

If state.state is not lsst.ts.idl.enums.Script.ScriptState.CONFIGURED.

async do_setLogLevel(data: BaseMsgType) None

Set logging level.

Parameters:
datacmd_setLogLevel.DataType

Logging level.

async do_stop(data: BaseMsgType) None

Stop the script.

Parameters:
datacmd_stop.DataType

Ignored.

Notes

This is a no-op if the script is already exiting. This does not wait for _exit to run.

get_detectors_string(detector_array)

Get a detector string from a detector array. Convert a detector array of the form [0, 1, 2, …] into a string of the form “(0, 1, 2, …)” to be used by pipetasks.

Parameters:
detector_arrayarray

Array with the detector IDs

Returns:
detectors_stringstr

Detector IDs list in the form “(0, 1, 2..)”

static get_exposure_id(obsid)

Parse obsid into an exposure id. Convert string in the format ??_?_YYYYMMDD_012345 into an integer like YYYYMMDD12345.

Parameters:
obsidstr

Observation id in the format ??_?_YYYYMMDD_012345, e.g., AT_O_20220406_000007.

Returns:
int

Exposure id in the format YYYYMMDD12345, e.g., 2022040600007.

get_instrument_configuration()

Get instrument configuration

Returns:
instrument_configuration: dict

Dictionary with instrument configuration.

async get_obs_id() str | None

Get obs id from camera obs id server.

Returns:
str or None

Id generated from camera server.

get_pipetask_parameters_bias()

Get necessary information to run the bias generation pipetask.

Returns:
cpBias.yamlstr

cp_pipe bias pipeline.

config_stringstr

Bias pipetask configuration for OCPS.

exposure_ids_biaslist`[`int]

List of bias exposure IDs.

get_pipetask_parameters_dark()

Get necessary information to run the dark generation pipetask.

Returns:
cpDark.yamlstr

cp_pipe dark pipeline.

config_stringstr

Dark pipetask configuration for OCPS.

exposure_ids_darklist`[`int]

List of dark exposure IDs.

get_pipetask_parameters_defects()

Get necessary information to run the defects generation pipetask.

Returns:
findDefects.yamlstr

cp_pipe defects pipeline.

config_stringstr

Defects pipetask configuration for OCPS.

exposure_ids_defectslist`[`int]

List of dark and flat exposure IDs.

get_pipetask_parameters_flat()

Get necessary information to run the flat generation pipetask.

Returns:
cpFlat.yamlstr

cp_pipe flat pipeline.

config_stringstr

Flat pipetask configuration for OCPS.

exposure_ids_flatlist`[`int]

List of flat exposure IDs.

get_pipetask_parameters_ptc()

Get necessary information to run the ptc generation pipetask.

Returns:
cpPtc.yaml or cpPtc.yaml#cpPtcGainFromFlatPairsstr

cp_pipe PTC or gain from pairs pipeline.

config_stringstr

PTC or gain from pairs pipetask configuration for OCPS.

exposure_ids_ptclist`[`int]

List of flat exposure IDs for PTC or gain estimation.

get_pipetask_parameters_verification_bias(job_id_calib)

Get necessary information to run the bias verification pipetask

Parameters:
jod_id_calibstr

Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Returns:
pipe_yamlstr

cp_verify bias pipeline.

config_stringstr

Bias verification pipetask configuration for OCPS.

exposure_ids_biaslist`[`int]

List of bias exposure IDs.

get_pipetask_parameters_verification_dark(job_id_calib)

Get necessary information to run the dark verification pipetask

Parameters:
jod_id_calibstr

Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Returns:
pipe_yamlstr

cp_verify dark pipeline.

config_stringstr

Dark verification pipetask configuration for OCPS.

exposure_ids_darklist`[`int]

List of dark exposure IDs.

get_pipetask_parameters_verification_flat(job_id_calib)

Get necessary information to run the flat verification pipetask.

Parameters:
jod_id_calibstr

Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Returns:
pipe_yamlstr

cp_verify flat pipeline.

config_stringstr

Flat verification pipetask configuration for OCPS.

exposure_ids_flatlist`[`int]

List of flat exposure IDs.

classmethod get_schema()

Return a jsonschema to validate configuration, as a dict.

Please provide default values for all fields for which defaults make sense. This makes the script easier to use.

If your script has no configuration then return None, in which case the config field of the configure command must be an empty string.

async image_in_oods_callback(data)

Callback function to check images are in oods

Parameters:
dataevt_imageInOODS.DataType

OODS, imageInOODS event sample.

classmethod make_from_cmd_line(**kwargs: Any) lsst.ts.salobj.base_script.BaseScript | None

Make a script from command-line arguments.

Return None if --schema specified.

Parameters:
kwargs

Keyword arguments for the script constructor. Must not include index.

Raises:
RuntimeError

If kwargs includes index.

next_supplemented_group_id() str

Return the group ID supplemented with a new subgroup.

The returned string has this format: f”{self.group_id}#{subgroup_id}”, where subgroup_id is an integer that starts at 1 and is incremented for every call to this method.

Raises:
RuntimeError

If there is no group ID.

async process_calibration(calib_type)

Generate and certify a specific type of calibration.

This method handles the generation of a specific calibration type using the call_pipetask method. After generating the calibration, it automatically certifies the calibration unless the calibration type is “GAIN”, which does not require certification. Additionally, it reports the estimated gains from flat pairs for applicable calibration types.

Parameters:
calib_typestr

The type of calibration to process. Supported types are: - “DEFECTS” - “PTC” - “GAIN”

Raises:
Exception

If an error occurs during the calibration generation or certification process.

Notes

  • The “GAIN” calibration type does not require certification.

  • Gain estimation from flat pairs is performed for “GAIN” and

“PTC” calibration types.

  • Verification for calibrations is not implemented in this method and thus not required.

async process_images(im_type)

Generate and optionally verify and certify calibrations for a given image type.

Parameters:
im_typestr

The type of image or calibration to process. One of [“BIAS”, “DARK”, “FLAT”].

Raises:
Exception

If an error occurs during verification, it is logged and ignored.

Notes

  • If generate_calibrations is set to True, the method will generate combined calibrations using the call_pipetask method.

  • If do_verify is True, the method will initiate verification of the generated calibrations.

  • If do_verify is False and calibrations are generated, the calibration will be automatically certified without verification.

  • If an error occurs during verification, it is logged and ignored.

async process_verification(im_type, job_id_calib)

Verify and certify the generated calibration for a given image type.

Parameters:
im_typestr

The type of image or calibration to verify. One of [“BIAS”, “DARK”, “FLAT”].

job_id_calibstr or None

The job ID returned by OCPS during the calibration generation pipetask call. If None, the verification will use reference calibrations from input collections.

Returns:
report_check_verify_statsdict

Dictionary containing the results of the verification checks, including whether the calibration should be certified, any statistical errors, failure thresholds, and the raw verification statistics.

Raises:
Exception

Logs and handles any exceptions that occur during the verification process.

Notes

  • Verification involves running the verify_calib method to execute the verification pipetask.

  • The method analyzes verification statistics to determine if the calibration meets the certification criteria.

program_reason()

Context manager to publish appropriate checkpoints with program and reason.

async put_log_level() None

Output the logLevel event.

async report_gains_from_flat_pairs(job_id_calib)

Print gains estimated form flat pairs.

Parameters:
jod_id_calibstr

Job ID returned by the OCPS after running the “GAIN” or “PTC” pipetasks

Notes

The “PTC” and “GAIN” tasks are defined by the “cp_pipe” pipelines “cpPtc.yaml” and “cpPtc.yaml#genGainsFromFlatPairs”, respectively.

async run()

Override base script run to encapsulate execution with appropriate checkpoints.

async run_block()
async save_test_case() None

Save test case to the LFA.

async set_exp_times_per_im_type(image_type)

Define exp_times and n_images according to image type.

Parameters:
image_typestr

Image type. One of [“BIAS”, “DARK”, “FLAT”].

set_metadata(metadata)

Set estimated duration of the script.

async set_state(state: lsst.ts.xml.enums.Script.ScriptState | int | None = None, reason: str | None = None, keep_old_reason: bool = False, force_output: bool = False) None

Set the script state.

Parameters:
stateScriptState or int, optional

New state, or None if no change

reasonstr, optional

Reason for state change. None for no new reason.

keep_old_reasonbool

If true, keep old reason; append the reason argument after “;” if it is is a non-empty string. If false replace with reason, or “” if reason is None.

force_outputbool, optional

If true the output even if not changed.

Notes

The lastCheckpoint field is set from self.last_checkpoint, and the numCheckpoints field is set from self.num_checkpoints.

async start() None

Finish construction.

async start_phase2() None

Additional work after start before fully started.

Used by BaseCsc to transition to handle the initial state.

async take_image_type(image_type, exp_times)

Take exposures and build exposure set.

Parameters:
image_typestr

Image type. One of [“BIAS”, “DARK”, “FLAT”].

exp_timeslist

List of exposure times.

Returns:
Tuple with exposure IDs.
async take_images(image_type)

Take images with instrument.

Parameters:
image_typestr

Image type. One of [“BIAS”, “DARK”, “FLAT”].

Returns:
exposurestuple

Tuple with the IDs of the exposures taken.

test_case_step(comment: str | None = None) None

Context manager to handle test case steps.

async verify_calib(image_type, job_id_calib)

Verify the calibration.

Parameters:
image_typestr

Image type.

jod_id_calibstr

Job ID returned by OCPS during calibration generation pipetask call. If None, the calibrations will be sought at the input collections.

Notes

The verification step runs tests in cp_verify that check the metrics in DMTN-101.

Suported calibrations: see self.pipetask_parameters_verification.

async wait_for_background_tasks()

Await the completion of all background calibration and verification tasks.

This method waits for all background tasks (calibration and verification processes) to complete within a specified timeout. If the tasks do not complete within the timeout period, it attempts to cancel any remaining unfinished tasks.

Raises:
asyncio.TimeoutError

If the background tasks do not complete within the allotted timeout.

Exception

Logs and handles any other exceptions that occur while waiting for tasks.

Notes

  • The timeout is calculated based on the background_task_timeout configuration parameter multiplied

    by the number of background tasks.

  • Upon a timeout, the method logs a warning and cancels any tasks that are still pending.

  • After handling timeouts or exceptions, the list of background tasks is cleared.