Rejecting bad data spans and breaks#

This tutorial covers:

  • manual marking of bad spans of data,

  • automated rejection of data spans based on signal amplitude, and

  • automated detection of breaks during an experiment.

We begin as always by importing the necessary Python modules and loading some example data; to save memory we’ll use a pre-filtered and downsampled version of the example data, and we’ll also load an events array to use when converting the continuous data to epochs:

# Authors: The MNE-Python contributors.
# License: BSD-3-Clause
# Copyright the MNE-Python contributors.
import os

import numpy as np

import mne

sample_data_folder = mne.datasets.sample.data_path()
sample_data_raw_file = os.path.join(
    sample_data_folder, "MEG", "sample", "sample_audvis_filt-0-40_raw.fif"
)
raw = mne.io.read_raw_fif(sample_data_raw_file, verbose=False)
events_file = os.path.join(
    sample_data_folder, "MEG", "sample", "sample_audvis_filt-0-40_raw-eve.fif"
)
events = mne.read_events(events_file)

Annotating bad spans of data#

The tutorial Parsing events from raw data describes how Annotations can be read from embedded events in the raw recording file, and Annotating continuous data describes in detail how to interactively annotate a Raw data object. Here, we focus on best practices for annotating bad data spans so that they will be excluded from your analysis pipeline.

The reject_by_annotation parameter#

In the interactive raw.plot() window, the annotation controls can be opened by pressing a. Here, new annotation labels can be created or existing annotation labels can be selected for use.

fig = raw.plot()
fig.fake_keypress("a")  # Simulates user pressing 'a' on the keyboard.
Raw plot

You can see that you need to add a description first to start with marking spans (Push the button “Add Description” and enter the description). You can use any description you like, but annotations marking spans that should be excluded from the analysis pipeline should all begin with “BAD” or “bad” (e.g., “bad_cough”, “bad-eyes-closed”, “bad door slamming”, etc). When this practice is followed, many processing steps in MNE-Python will automatically exclude the “bad”-labelled spans of data; this behavior is controlled by a parameter reject_by_annotation that can be found in many MNE-Python functions or class constructors, including:

For example, when creating epochs from continuous data, if reject_by_annotation=True the Epochs constructor will drop any epoch that partially or fully overlaps with an annotated span that begins with “bad”.

Generating annotations programmatically#

The Overview of artifact detection tutorial introduced the artifact detection functions find_eog_events() and find_ecg_events() (although that tutorial mostly relied on their higher-level wrappers create_eog_epochs() and create_ecg_epochs()). Here, for demonstration purposes, we make use of the lower-level artifact detection function to get an events array telling us where the blinks are, then automatically add “bad_blink” annotations around them (this is not necessary when using create_eog_epochs(), it is done here just to show how annotations are added non-interactively). We’ll start the annotations 250 ms before the blink and end them 250 ms after it:

Using EOG channel: EOG 061
EOG channel index for this subject is: [375]
Filtering the data to remove DC offset to help distinguish blinks from saccades
Selecting channel EOG 061 for blink detection
Setting up band-pass filter from 1 - 10 Hz

FIR filter parameters
---------------------
Designing a two-pass forward and reverse, zero-phase, non-causal bandpass filter:
- Windowed frequency-domain design (firwin2) method
- Hann window
- Lower passband edge: 1.00
- Lower transition bandwidth: 0.50 Hz (-12 dB cutoff frequency: 0.75 Hz)
- Upper passband edge: 10.00 Hz
- Upper transition bandwidth: 0.50 Hz (-12 dB cutoff frequency: 10.25 Hz)
- Filter length: 1502 samples (10.003 s)

Now detecting blinks and generating corresponding events
Found 46 significant peaks
Number of EOG events detected: 46
General
Filename(s) sample_audvis_filt-0-40_raw.fif
MNE object type Raw
Measurement date 2002-12-03 at 19:01:10 UTC
Participant Unknown
Experimenter Unknown
Acquisition
Duration 00:04:38 (HH:MM:SS)
Sampling frequency 150.15 Hz
Time points 41,700
Channels
Magnetometers
Gradiometers and
EEG and
EOG
Stimulus
Head & sensor digitization 146 points
Filters
Highpass 0.10 Hz
Lowpass 40.00 Hz
Projections PCA-v1 (off)
PCA-v2 (off)
PCA-v3 (off)
Average EEG reference (off)


Now we can confirm that the annotations are centered on the EOG events. Since blinks are usually easiest to see in the EEG channels, we’ll only plot EEG here:

Raw plot

See the section Creating annotations programmatically for more details on creating annotations programmatically.

Detecting and annotating breaks#

Another useful function, albeit not related to artifact detection per se, is mne.preprocessing.annotate_break: It will generate annotations for segments of the data where no existing annotations (or, alternatively: events) can be found. It can therefore be used to automatically detect and mark breaks, e.g. between experimental blocks, when recording continued.

For the sake of this example, let’s assume an experiment consisting of two blocks, the first one stretching from 30 to 90, and the second from 120 to 180 seconds. We’ll mark these blocks by annotations, and then use mne.preprocessing.annotate_break to detect and annotate any breaks.

Note

We need to take raw.first_time into account, otherwise the onsets will be incorrect!

onsets = [raw.first_time + 30, raw.first_time + 180]
durations = [60, 60]
descriptions = ["block_1", "block_2"]

block_annots = mne.Annotations(
    onset=onsets,
    duration=durations,
    description=descriptions,
    orig_time=raw.info["meas_date"],
)
raw.set_annotations(raw.annotations + block_annots)  # add to existing
raw.plot()
Raw plot

Now detect break periods. We can control how far the break annotations shall expand toward both ends of each break.

break_annots = mne.preprocessing.annotate_break(
    raw=raw,
    min_break_duration=20,  # consider segments of at least 20 s duration
    t_start_after_previous=5,  # start annotation 5 s after end of previous one
    t_stop_before_next=2,  # stop annotation 2 s before beginning of next one
)

raw.set_annotations(raw.annotations + break_annots)  # add to existing
raw.plot()
Raw plot
Ignoring annotations with descriptions starting with: bad, edge

Detected 3 break periods of >= 20 s duration:
    0.0 – 28.0 s [28.0 s]
    95.0 – 178.0 s [83.0 s]
    245.0 – 277.7 s [32.7 s]
In total, 51.7% of the data (143.7 s) have been marked as a break.

You can see that 3 segments have been annotated as BAD_break:

  • the first one starting with the beginning of the recording and ending 2 seconds before the beginning of block 1 (due to t_stop_before_next=2),

  • the second one starting 5 seconds after block 1 has ended, and ending 2 seconds before the beginning of block 2 (t_start_after_previous=5, t_stop_before_next=2),

  • and the last one starting 5 seconds after block 2 has ended (t_start_after_previous=5) and continuing until the end of the recording.

You can also see that only the block_1 and block_2 annotations were considered in the detection of the break periods – the EOG annotations were simply ignored. This is because, by default, annotate_break ignores all annotations starting with 'bad'. You can control this behavior via the ignore parameter.

It is also possible to perform break period detection based on an array of events: simply pass the array via the events parameter. Existing annotations in the raw data will be ignored in this case:

# only keep some button press events (code 32) for this demonstration
events_subset = events[events[:, -1] == 32]
# drop the first and last few events
events_subset = events_subset[3:-3]

break_annots = mne.preprocessing.annotate_break(
    raw=raw,
    events=events_subset,  # passing events will ignore existing annotations
    min_break_duration=25,  # pick a longer break duration this time
)

# replace existing annotations (otherwise it becomes difficult to see any
# effects in the plot!)
raw.set_annotations(break_annots)
raw.plot(events=events_subset)
Raw plot
Detected 2 break periods of >= 25 s duration:
    0.0 – 51.4 s [51.4 s]
    185.6 – 277.7 s [92.1 s]
In total, 51.7% of the data (143.5 s) have been marked as a break.

Rejecting Epochs based on peak-to-peak channel amplitude#

Besides “bad” annotations, the mne.Epochs class constructor has another means of rejecting epochs, based on signal amplitude thresholds for each channel type. In the overview tutorial we saw an example of this: setting maximum acceptable peak-to-peak amplitudes for each channel type in an epoch, using the reject parameter. There is also a related parameter, flat, that can be used to set minimum acceptable peak-to-peak amplitudes for each channel type in an epoch:

reject_criteria = dict(
    mag=3000e-15,  # 3000 fT
    grad=3000e-13,  # 3000 fT/cm
    eeg=100e-6,  # 100 µV
    eog=200e-6,
)  # 200 µV

flat_criteria = dict(mag=1e-15, grad=1e-13, eeg=1e-6)  # 1 fT  # 1 fT/cm  # 1 µV

The values that are appropriate are dataset- and hardware-dependent, so some trial-and-error may be necessary to find the correct balance between data quality and loss of power due to too many dropped epochs. Here, we’ve set the rejection criteria to be fairly stringent, for illustration purposes.

Two additional parameters, reject_tmin and reject_tmax, are used to set the temporal window in which to calculate peak-to-peak amplitude for the purposes of epoch rejection. These default to the same tmin and tmax of the entire epoch. As one example, if you wanted to only apply the rejection thresholds to the portion of the epoch that occurs before the event marker around which the epoch is created, you could set reject_tmax=0. A summary of the causes of rejected epochs can be generated with the plot_drop_log() method:

raw.set_annotations(blink_annot)  # restore the EOG annotations
epochs = mne.Epochs(
    raw,
    events,
    tmin=-0.2,
    tmax=0.5,
    reject_tmax=0,
    reject=reject_criteria,
    flat=flat_criteria,
    reject_by_annotation=False,
    preload=True,
)
epochs.plot_drop_log()
8 of 319 epochs removed (2.5%)
Not setting metadata
319 matching events found
Setting baseline interval to [-0.19979521315838786, 0.0] s
Applying baseline correction (mode: mean)
Created an SSP operator (subspace dimension = 4)
4 projection items activated
Loading data for 319 events and 106 original time points ...
    Rejecting  epoch based on EEG : ['EEG 001', 'EEG 002', 'EEG 003', 'EEG 007']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EEG : ['EEG 001', 'EEG 003', 'EEG 007']
    Rejecting  epoch based on MAG : ['MEG 1711']
    Rejecting  epoch based on EEG : ['EEG 007']
    Rejecting  epoch based on EEG : ['EEG 003']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on EEG : ['EEG 001', 'EEG 002', 'EEG 003']
8 bad epochs dropped

Notice that we’ve passed reject_by_annotation=False above, in order to isolate the effects of the rejection thresholds. If we re-run the epoching with reject_by_annotation=True (the default) we see that the rejections due to EEG and EOG channels have disappeared (suggesting that those channel fluctuations were probably blink-related, and were subsumed by rejections based on the “bad blink” label).

epochs = mne.Epochs(
    raw,
    events,
    tmin=-0.2,
    tmax=0.5,
    reject_tmax=0,
    reject=reject_criteria,
    flat=flat_criteria,
    preload=True,
)
epochs.plot_drop_log()
39 of 319 epochs removed (12.2%)
Not setting metadata
319 matching events found
Setting baseline interval to [-0.19979521315838786, 0.0] s
Applying baseline correction (mode: mean)
Created an SSP operator (subspace dimension = 4)
4 projection items activated
Loading data for 319 events and 106 original time points ...
    Rejecting  epoch based on MAG : ['MEG 1711']
39 bad epochs dropped

More importantly, note that many more epochs are rejected (~12.2% instead of ~2.5%) when rejecting based on the blink labels, underscoring why it is usually desirable to repair artifacts rather than exclude them.

The plot_drop_log() method is a visualization of an Epochs attribute, namely epochs.drop_log, which stores empty lists for retained epochs and lists of strings for dropped epochs, with the strings indicating the reason(s) why the epoch was dropped. For example:

((np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), ('MEG 1711',), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), (), (np.str_('bad blink'),), (), (), (), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), (), ())

Finally, it should be noted that “dropped” epochs are not necessarily deleted from the Epochs object right away. Above, we forced the dropping to happen when we created the Epochs object by using the preload=True parameter. If we had not done that, the Epochs object would have been memory-mapped (not loaded into RAM), in which case the criteria for dropping epochs are stored, and the actual dropping happens when the Epochs data are finally loaded and used. There are several ways this can get triggered, such as:

You can also trigger dropping with the drop_bad() method; if reject and/or flat criteria have already been provided to the epochs constructor, drop_bad() can be used without arguments to simply delete the epochs already marked for removal (if the epochs have already been dropped, nothing further will happen):

Alternatively, if rejection thresholds were not originally given to the Epochs constructor, they can be passed to drop_bad() later instead; this can also be a way of imposing progressively more stringent rejection criteria:

stronger_reject_criteria = dict(
    mag=2000e-15,  # 2000 fT
    grad=2000e-13,  # 2000 fT/cm
    eeg=100e-6,  # 100 µV
    eog=100e-6,
)  # 100 µV

epochs.drop_bad(reject=stronger_reject_criteria)
print(epochs.drop_log)
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1441', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1541']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1441', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1221', 'MEG 1321', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1441', 'MEG 1541', 'MEG 1741', 'MEG 2141', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 1741', 'MEG 2141', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421']
    Rejecting  epoch based on MAG : ['MEG 1421', 'MEG 1431']
    Rejecting  epoch based on MAG : ['MEG 1431']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 0811', 'MEG 0911', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431']
    Rejecting  epoch based on EOG : ['EOG 061']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1321', 'MEG 1331', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 0521', 'MEG 0811', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 1421']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2131', 'MEG 2531', 'MEG 2541', 'MEG 2611', 'MEG 2621', 'MEG 2631']
    Rejecting  epoch based on MAG : ['MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 2421']
    Rejecting  epoch based on MAG : ['MEG 1321', 'MEG 1331']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1541']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621']
    Rejecting  epoch based on MAG : ['MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 2621']
65 bad epochs dropped
((np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1441', 'MEG 1541', 'MEG 2621'), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), (), ('MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (), (np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (np.str_('bad blink'),), (), ('MEG 1421', 'MEG 1431', 'MEG 2621'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (), (), (), (np.str_('bad blink'),), (), (), ('EOG 061',), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431'), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1541'), ('EOG 061',), (np.str_('bad blink'),), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1441', 'MEG 1541', 'MEG 2621', 'EOG 061'), (), (), (), ('EOG 061',), ('MEG 0111', 'MEG 0141', 'MEG 1221', 'MEG 1321', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1441', 'MEG 1541', 'MEG 1741', 'MEG 2141', 'MEG 2621'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 1741', 'MEG 2141', 'MEG 2621'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), ('EOG 061',), (), ('MEG 1711',), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1541'), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431'), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (np.str_('bad blink'),), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1421'), (), (), (), (), ('MEG 1421', 'MEG 1431'), (), ('MEG 1431',), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431'), (), (np.str_('bad blink'),), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), (), (), (), (), (), ('EOG 061',), (), (np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621'), ('EOG 061',), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), (np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 2621'), (np.str_('bad blink'),), (np.str_('bad blink'),), ('MEG 0111', 'MEG 0141', 'MEG 0811', 'MEG 0911', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431'), (np.str_('bad blink'),), ('EOG 061',), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (), (), (np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1541'), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1321', 'MEG 1331', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 0521', 'MEG 0811', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (np.str_('bad blink'),), ('MEG 1421',), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), (), ('MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621'), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621'), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2131', 'MEG 2531', 'MEG 2541', 'MEG 2611', 'MEG 2621', 'MEG 2631'), (), (), (), (), (np.str_('bad blink'),), (), (), (), (), ('MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 2621'), ('MEG 2421',), (), (np.str_('bad blink'),), (np.str_('bad blink'),), ('MEG 1321', 'MEG 1331'), (), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541'), (np.str_('bad blink'),), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1541'), (np.str_('bad blink'),), (np.str_('bad blink'),), (), (), (np.str_('bad blink'),), (), (np.str_('bad blink'),), (), (), (), (), (), (), (), (), (), (), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (), (), (), (), ('MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 2621'), (), (), ('MEG 0111', 'MEG 0141', 'MEG 1421', 'MEG 1431', 'MEG 1541', 'MEG 2621'), (), (), (), (), (np.str_('bad blink'),), (), (), ('MEG 0141', 'MEG 1411', 'MEG 1421', 'MEG 1431', 'MEG 2621'), ())

Rejecting Epochs using callables (functions)#

Sometimes it is useful to reject epochs based criteria other than peak-to-peak amplitudes. For example, we might want to reject epochs based on the maximum or minimum amplitude of a channel. In this case, the mne.Epochs.drop_bad function also accepts callables (functions) in the reject and flat parameters. This allows us to define functions to reject epochs based on our desired criteria.

Let’s begin by generating Epoch data with large artifacts in one eeg channel in order to demonstrate the versatility of this approach.

raw.crop(0, 5)
raw.del_proj()
chans = raw.info["ch_names"][-5:-1]
raw.pick(chans)
data = raw.get_data()

new_data = data
new_data[0, 180:200] *= 1e3
new_data[0, 460:580] += 1e-3
edit_raw = mne.io.RawArray(new_data, raw.info)

# Create fixed length epochs of 1 second
events = mne.make_fixed_length_events(edit_raw, id=1, duration=1.0, start=0)
epochs = mne.Epochs(edit_raw, events, tmin=0, tmax=1, baseline=None)
epochs.plot(scalings=dict(eeg=50e-5))
Raw plot
Creating RawArray with float64 data, n_channels=4, n_times=752
    Range : 0 ... 751 =      0.000 ...     5.002 secs
Ready.
Not setting metadata
5 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 5 events and 151 original time points ...
0 bad epochs dropped
Using data from preloaded Raw for 5 events and 151 original time points ...
Using data from preloaded Raw for 5 events and 151 original time points ...
Using data from preloaded Raw for 5 events and 151 original time points ...

As you can see, we have two large artifacts in the first channel. One large spike in amplitude and one large increase in amplitude.

# Let's try to reject the epoch containing the spike in amplitude based on the
# maximum amplitude of the first channel. Please note that the callable in
# ``reject`` must return a (good, reason) tuple. Where the good must be bool
# and reason must be a str, list, or tuple where each entry is a str.

epochs = mne.Epochs(
    edit_raw,
    events,
    tmin=0,
    tmax=1,
    baseline=None,
    preload=True,
)

epochs.drop_bad(
    reject=dict(eeg=lambda x: ((np.max(x, axis=1) > 1e-2).any(), "max amp"))
)
epochs.plot(scalings=dict(eeg=50e-5))
Raw plot
Not setting metadata
5 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 5 events and 151 original time points ...
0 bad epochs dropped
1 bad epochs dropped

Here, the epoch containing the spike in amplitude was rejected for having a maximum amplitude greater than 1e-2 Volts. Notice the use of the any() function to check if any of the channels exceeded the threshold. We could have also used the all() function to check if all channels exceeded the threshold.

# Next, let's try to reject the epoch containing the increase in amplitude
# using the median.

epochs = mne.Epochs(
    edit_raw,
    events,
    tmin=0,
    tmax=1,
    baseline=None,
    preload=True,
)

epochs.drop_bad(
    reject=dict(eeg=lambda x: ((np.median(x, axis=1) > 1e-4).any(), "median amp"))
)
epochs.plot(scalings=dict(eeg=50e-5))
Raw plot
Not setting metadata
5 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 5 events and 151 original time points ...
0 bad epochs dropped
1 bad epochs dropped

Finally, let’s try to reject both epochs using a combination of the maximum and median. We’ll define a custom function and use boolean operators to combine the two criteria.

def reject_criteria(x):
    max_condition = np.max(x, axis=1) > 1e-2
    median_condition = np.median(x, axis=1) > 1e-4
    return ((max_condition.any() or median_condition.any()), ["max amp", "median amp"])


epochs = mne.Epochs(
    edit_raw,
    events,
    tmin=0,
    tmax=1,
    baseline=None,
    preload=True,
)

epochs.drop_bad(reject=dict(eeg=reject_criteria))
epochs.plot(events=True)
Raw plot
Not setting metadata
5 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 5 events and 151 original time points ...
0 bad epochs dropped
2 bad epochs dropped

Note that a complementary Python module, the autoreject package, uses machine learning to find optimal rejection criteria, and is designed to integrate smoothly with MNE-Python workflows. This can be a considerable time-saver when working with heterogeneous datasets.

Total running time of the script: (0 minutes 16.432 seconds)

Gallery generated by Sphinx-Gallery