Data Quality and Caveats
We recommend and request that you contact the ECT team early in your studies so that we can help assess if there are any data issues that might affect the results of your analysis. This page has a list of some known issues but it is necessarily brief and incomplete.
Contact Harlan Spence, Geoff Reeves
Known issues in Data Releases 2 and 3
Release 2 (rel02) of the RBSP-ECT data contains known problems and 'issues' that could affect the scientific analysis of the data. In the coming weeks we will be documenting the ones that it is most important to be aware of. For example:
• None of the ECT instruments has been fully cross-calibrated against the other instruments in the suite. The largest discrepancies in electron fluxes are in the overlap region between the MagEIS High channels and REPT.
• None of the ECT instruments have applied background subtraction to any of the data
Here is a partial list of known issues for each instrument. The instrument names link to more detail about these data issues.
- Some adjustments for electron flux conversion factors in lowest ~2 channels have occurred at various intervals
- When signal to background gets low the spectrum appears to "flatten out". This primarily occurs in the highest electron chanels. Background appears to be more than just galactic cosmic rays.
- Energy thresholds have been changed in the detector logic at various times. This primarily affects inner zone populations.
- One proton channel (~60 MeV) appears to have anomalously high fluxes
MagEIS - data release 3 (rel03) notes (contact: Seth Claudepierre, firstname.lastname@example.org)
- Please see this document for a more detailed description of the following points.
- Detectors 0 and 1 (aka "pixels") on the six LOW/MED units (LOW-A,B; M35-A,B; M75-A,B) are known to be noisy and do not produce valid electron data.
- Detector 0 failed on MagEIS-HIGH-A on 2013/10/03. It measures ~1 MeV electrons.
- Background contamination in both MagEIS electrons and protons, due to inner zone protons and bremsstrahlung x-rays. We can correct for contamination in the MagEIS electron measurements. We cannot correct for contamination in the MagEIS proton measurements.
- There is light contamination in the MagEIS proton telescopes, when they view the sunlit Earth. This is corrected for in the level 3 data but not in the level 2 data.
- The MagEIS proton telescopes have noise in the lowest energy channels on both spacecraft, likely due to ion implant damage. The noise began around March 2013 and gets progressively worse as the mission progresses and also moves into higher energy channels.
- Major changes in electron instrument settings/performance in July-August 2013. Revised energy channel definitions and flux conversion factors, based on GEANT4 simulations and bow-tie analyses.
- MagEIS energy channels and pitch-angle bins are time-varying. Such changes were more frequent early in the mission and the energy channels have been stable since April 2013. Please see this document for a detailed history of energy channel values.
- MagEIS electron data that suffer from thermal oscillations have been removed and set to fill. This has occurred in several intervals during the mission and is due to improper functioning of the thermal control system.
- When the fluxes are low, counting statistics can produce significant errors and uncertainties.
- For a few reasons throughout the mission, some MagEIS spectrometers were operated in the “Bias Off” configuration for varying periods of time. With the bias is off, the data are not valid.
- The L2 and L3 files contain a data quality flags that the user should consult.
- Frequent periods when counting rates are near 1-count level. When the fluxes are low, counting statistics can produce significant errors and uncertainties.
- Operational changes in January for energies and time-of-flight tables
- Feature of pre-September 5/6, 2013 data where flux binning is messy
- HOPE uses highly time-dependent, multi-mode operations (e.g., Burst support mode) and hence the data sets can be complicated
- Low energy correction for spacecraft velocity not yet corrected
- Different energy tables and modes for apogee versus perigee
- No spacecraft potential flags or corrections (can be significant)
Issues fixed in Data Release 02
The most significant change in this data release is that Level 3 (pitch angle resolved)
files are now available for all ECT instruments.
Level 2 files have also been reprocessed and have been released with the "rel02" label.
Any routines that read the previous "rel01" files should be changed to read "rel02"
and we recommend that only release 02 data be used now.
Significant, known data issues remain in release 02 and are documented at
Please contact the instrument teams for more information regarding
proper analysis and interpretation of the data.
There was a problem with the timing on sector 0 that was in
the Level 2 files but wasn't an issue in L2 analysis. When sector was turned
into pitch angle the sector0 problem showed up as a pitch angle problem.
This has been fixed. Release02 data will improve over the initial release.
MagEIS Level 3 files are now available on line. In addition to sorting by
pitch angle the MagEIS L3 files also applies a routine to to mask sectors that
saw either the sun or the sunlit Earth and combine the data from the
different MagEIS instruments into a single L3 data product. the outcome is a single
common data set of pitch angles that the community can all use without
significant overhead of reprocessing by each investigator.
While validating the HOPE moments we found a serious problem that affects
all the pitch angles and pitch angle-derived products, rendering them essentially unusable
scientifically. The issue was a subtle incompatibility in the APL SPICE and LANL
Space Py libraries that applied incorrect rotation matrices. The resulting errors
could be large and could result in unphysically isotropic distribution functions.
HOPE Level 3 codes have been completely rewritten and the data for the full
mission have been processed.
Issues fixed in Data Release 01
This is an update on some recent changes in the processing of HOPE data that could affect studies you are doing.
An issue emerged as we were looking at cross-calibration of HOPE and MagEIS data. The upper end of HOPE and the lower end of MagEIS did not agree and were off by almost a factor of 10. Some good detective work by the HOPE team revealed that the Level1 to Level2 processing was not using the correct flux conversion factor. SIMION simulations include a factor of DeltaE/E in the geometric factor that was not properly accounted for in the flux conversion factor. With logarithmic energy spacing DeltaE/E is constant with energy step and it is nearly a factor of 7. As a result HOPE fluxes were high by a factor of 7. This problem has been fixed and all the HOPE data have been reprocessed.
The factor of 7 applies to both ion and electron channels. While it does not affect relative variations of flux it clearly does affect absolute flux values and you should make sure you are using the most recent HOPE files in any analysis you are doing.
The second issue was more subtle and appeared as the team was trying to validate pitch angle data. Like all electrostatic analyzers, HOPE steps in energy rather than taking all energies simultaneously. There was an issue with the time being associated with a particular energy step which also affects the pitch angle assigned to that measurement. Times were being truncated rather than rounded introducing an energy-dependent time (and pitch angle) offset. The error was in the Level0 to Level1 software but didn't appear until Level3 data products. Tracking this down passes a major hurdle in releasing HOPE Level3 data.
While the timing problem is not obvious until Level3 it does exist in Level1 and 2 and will affect any studies looking at detailed timing (particularly burst support data). We don't' know of any current studies that would be affected but, again, we recommend that any times at less than 1-spin resolution should be checked against the new data.