Data screening

Near the end of the - spi_sa_``UT date''.log - log file, you can find a table with (``CHI2'') values per pointing looking as follows (with a slightly different formatting, however):

---------------------------------------------------------
Contributions to CHI2 parameter by pointing exposure.

Ptg Rev Exp  ONTIME  CHI2,ML Expected Diff  Reduced Data 
-no -no -no  (secs)   value   value   /STD  CHI2,ML excl 
--------------------------------------------------------
 1 102  20  2155.2    34.6    18.0   2.77   1.92   0.00 
 2 102  21  1932.7    49.6    18.0   5.27   2.76   0.00 
 3 102  22  2200.8    44.5    18.0   4.42   2.47   0.00 
 4 102  23  2198.8    32.5    18.0   2.41   1.80   0.00 
 5 102  24  2196.8    31.7    18.0   2.28   1.76   0.00 
 6 102  25  2143.8    28.3    18.0   1.72   1.57   0.00
 7 102  26  2198.8    55.0    18.0   6.16   3.05   0.00
 8 102  27  2196.8    40.7    18.0   3.78   2.26   0.00
 9 102  28  2194.8    31.3    18.0   2.22   1.74   0.00
10 102  29  2189.8    47.6    18.0   4.93   2.64   0.00

The most important information to consider is ``Diff/STD'', which gives the difference, expressed in terms of standard deviations, between the (or ML) resulting values and the expected ones. In our case, these residuals are not reasonable for pointings 2, 3, 7, 8, and 10. When using your own data, you may find some pointings with very large residual values. DO NOT TRUST THE RESULTS if some of the involved pointings have residuals much larger than 3 standard deviations (although see Sect.  for possible exception to this rule). You must get rid of these residuals. The first thing to do is to use the AUTO filter of spiros. Launch spi_science_analysis again, un-select all tasks except spiros and enter ``AUTO'' in the pointing subset under spiros options and run the pipeline again. The AUTO filter automatically removes all data on a per pointing and per detector base that have completely unreasonable values. For most datasets, this significantly improves the results, but not in our case. The next thing to do is to work on the background. You should use different background models to improve the modelling. We will use the MCM method in the next step: launch spi_science_analysis again, open the spiros options, and select background method 5 and run the pipeline. The results are now much better:

----------------------------------------------------------
Contributions to CHI2 parameter by pointing exposure.

Ptg Rev Exp  ONTIME  CHI2,ML Expected Diff  Reduced Data 
-no -no -no  (secs)   value   value   /STD  CHI2,ML excl 
----------------------------------------------------------
 1 102  20  2155.2    18.7    15.3   0.61   1.22   0.00 
 2 102  21  1932.7    17.8    15.3   0.45   1.16   0.00 
 3 102  22  2200.8    19.8    15.3   0.81   1.29   0.00 
 4 102  23  2198.8    27.3    15.3   2.18   1.79   0.00 
 5 102  24  2196.8    25.3    15.3   1.81   1.66   0.00 
 6 102  25  2143.8    14.6    15.3  -0.13   0.95   0.00
 7 102  26  2198.8    25.7    15.3   1.88   1.68   0.00
 8 102  27  2196.8    14.2    15.3  -0.20   0.93   0.00
 9 102  28  2194.8    19.3    15.3   0.73   1.26   0.00
10 102  29  2189.8    12.3    15.3  -0.54   0.80   0.00

All residuals are now below three sigma and we do not need to further work on the extraction. However, in other datasets you may still have pointings with significant residuals. In this case you may have bright variable sources in the FoV. See Sect.  to know how to deal with such cases.

In most cases, you will encounter pointings with high residual values during your analysis, which you have to remove to obtain reliable results. To exclude such bad pointings from your analysis, note the pointing numbers of the bad pointings. Then launch spi_science_analysis again, de-select all tasks except spiros. Open the ``Spiros options'' GUI. Enter all good pointings in the pointing subset entry field. For example, if pointing 7 and 8 of a total of 20 pointings are bad, enter 1-6,9-20 as pointing subset. Now run the analysis again. You will find that this time the resulting images are much better. If you check the log file, you'll see that the values of all pointings have improved significantly. If you still encounter some pointings with bad values, you can repeat the procedure until all values are reasonable. However, it is easy to obtain results with very small residuals by removing all bad pointings from the analysis. You must be very careful when removing pointings: all really bad pointings are removed automatically by the pipeline during the pointing step and by the AUTO filter of spiros. Very often when you still encounter bad pointings, these are due to an insufficient sky model (e.g., you may have a strong variable source or a transient in the FoV).