# Pipeline Flagging Tasks
The Pipeline contains a number of tasks that flag bad data, based on e.g. online flags or user-provided flags. A
sub-set of the pipeline flagging tasks will flag data based on evaluation of their own flagging heuristics, and a
majority of this sub-set use a common Pipeline Flagging framework. The following note provides a description of the
common flagging framework, and an in-depth explanation of individual flagging tasks.
## Summary current pipeline tasks
The following pipeline tasks make use of the flagging framework and are part of (at least) one of the standard pipeline recipes:
- hif_rawflagchans: flags raw data in MS.
- [hifa/hsd]_tsysflag: flags Tsys caltable created by [hifa/hsd]_tsyscal.
- hif_lowgainflag: flags antennas in MS, based on evaluating heuristics on gain amplitude caltable created within task.
- hifa_wvrgcalflag: evaluates if applying WVR correction provides tangible improvement, flagging WVR caltable in process where needed, using WVR caltable created within task (using hifa_wvrgcal).
The following pipeline tasks perform flagging but do not make use of the common flagging framework:
- hif_correctedampflag: Flag corrected - model amplitudes based on calibrators.
- hifa_bandpassflag: uses hif_correctedampflag to flag corrected-model amplitude outliers for the bandpass calibrator.
- hifa_gfluxscaleflag: uses hif_correctedampflag to flag corrected-model amplitude outliers for the phase and flux calibrators.
- hifa_polcalflag: uses hif_correctedampflag to flag corrected-model amplitude outliers for the polarisation calibrators.
- hifa_targetflag: uses hif_correctedampflag to flag corrected-model amplitude outliers for the target source.
- hifa_flagdata: performs basic deterministic flagging on MS (e.g. shadowed antennas, online flags, autocorrelations, etc)
- hifa_flagdata: performs basic deterministic flagging on MS (e.g. shadowed antennas, online flags, autocorrelations, etc).
- hifa_flagtargets: performs deterministic flagging on science target MS (input template file).
- hifa_fluxcalflag: locates and flags line regions in solar-system flux calibrators.
## General overview of a flagging task within framework
The main flagging pipeline tasks are done in two steps:
1. Generate a "flagging view"
Some quantity of interest is taken from the MS or a caltable, depending on the flagging task, and this quantity is represented in a 1D or 2D matrix called the "flagging view". Flagging views are typically generated separately for each spw, often also separately for each polarisation. Typical examples of flagging views include:
Flagging commands are generated for each of the identified outlier data points.
If the number of data points in the flagging view are smaller than the minimum sample 'fhl_minsample' (default: 5), then no flagging is attempted.
### Task: hif_lowgainflag
This task first creates a phased-up bandpass caltable, then a gain phase caltable, and finally a gain amplitude caltable. This final gain amplitude caltable is used to identify antennas with outlier gains, for each spw. Flagging commands for outlier antennas (per spw) are applied to the entire MS.
A separate view is created for spw. Each view is a matrix with axes "time" vs. "antenna". Each point in the matrix is the absolute gain amplitude for that antenna/timestamp.
A separate view is created for spw. Each view is a matrix with axes "scan" vs. "antenna". Each point in the matrix is the absolute gain amplitude for that antenna/scan.
The views are evaluated against the "nmedian" matrix flagging rule, where data points are identified as outliers if:
1. their value is smaller than a threshold-factor * median of all non-flagged data points, where the threshold is 'fnm_lo_limit', or,
2. their value is larger than a threshold-factor * median of all non-flagged data points, where the threshold is 'fnm_hi_limit'.
Flagging commands are generated for each of the identified outlier data points.