Earthshine blog

Now that the camera is up and running again, a series of exposures have been taken to test the bias field.

Below is the old and new compared, and the difference in percent:

The bias level has changed by 1.3% or so, and the slope of the new field is different from the old – there is also some, smaller, change in curvature. The change in mean level is not so important since we scale the superbias to the current bias mean value anyway, but the change in slope is important. The relative level from up-slope to down-slope is something like 0.1% which is the adopted science goal for our project.

Further testing of the bias field will be performed as we proceed and if the above pattern is stable we shall simply start using a new superbias for currently acquired science frames. If studies show that the bias goes up and down and changes slope willy-nilly, as it were, then we have more to worry about!

We rely on subtracting ‘smooth scaled superbias’ fields from science frames, because of the 20-minute thermostat-oscillation in the bias mean, and a wish to avoid adding noise by subtraction of noisy bias fields. We generate the scaling factor for the superbias by averaging the bias frames taken just before and after the science frame. We seem to be assuming that the bias mean does not change – or does not change irregularly – during this procedure. Let us examine that assumption.

We take most bias frames as single frames, but happen to have 50-image stacks here:

2455643.4932200DARK_DARK-50ms.fits.gz
2455643.4957035DARK_DARK-500ms.fits.gz
2455643.4942722DARK_DARK-300ms.fits.gz
2455643.4949637DARK_DARK-400ms.fits.gz

We calculate the average of each subimage in these stacks and plot the results:

The camera can take many images per second in ‘kinetic mode’ so the above sequence last something like 10 seconds each. There is evidently some slight trend during that time – at the 0.05% (or 0.15 counts) level compared to the bias mean. This may not seem much, but if the scaled superbias mean is wrong by 0.15 cts it can mean an error on the DS flux of many percent – because the DS fluxes we are using are at anything from near 1 count above sky to maybe 10 counts above sky. For a DS at 1 count the 0.15 count error is 15% while it is 1.5% for the 10-count DS. This is quite serious.

It may be the reason that the halo-removing methods based on ‘subtracting a model halo’ did not work well – they relied on the bias being correctly subtracted previously. The ‘profile fitting method’ apparently worked better and this is probably because it inherently contains a fit to the sky (and thus bias, and also any insufficiently subtracted, bias).

Bias subtraction does more than subtract a mean level, of course – there is also some structure – notably a 0.2 count raised edge along the vertical sides, stretching some 10% into the field. Some row asymmetry is also evident, along with a very slight ‘ripple’ in the column direction (Henriette’s thesis has all this).

Subtracting the scaled superbias is therefore mostly a good thing, but it should be realised that important level-errors may be present causing blind reliance on the ‘complete bias removal’ to be dangerous – better to allow for an additional small pedestal in your modelling.

These are calibrated B-V images from JD2455945.17, as explained at this link.

The one on the left is generated WITH flatfielding of the B and V images used, while the one on the right is generated WITHOUT flatfielding. They are almost identical, but not quite. Here is the difference between the two:

The pattern is the expected pattern from the CCD chip. The histogram of the values is here:

So there is a difference which is distributed around 0 with S.D. 5 millimagnitudes. The effect of flattening would appear to be small – but recall that these are difference images.

Apart from various problems related to aligning images it is rather nice to work with difference images – problems tend to cancel out! However, note in the above how some of the structure on the DS in the B-V image looks like the features of the falt field – as if perhaps the flatfield we used did nothave enoughj ‘amplitude’ in the stripes.

We show here the B-V mean value of all pixels lit by sunshine. The data have been selected for being ‘good’, so the scatter is somewhat disappointing. Best one can say is that we do not contradict the published B-V=0 .92 value.

Blue and red points correspond to different parts of the lunar cycle – before and after Full Moon – waxing and waning, whatever you want to call it.

In the entry below, at:
http://iloapp.thejll.com/blog/earthshine?Home&post=269
we considered the need to remove some residuals left over when a poorly-fitted halo had been removed from the observed image. Sensing that the problem has to do with asymmetry in the solution forced by the lunar disc not being well centred, we consider now the effect of lunar position in the image on the quality of the halo fit. We estimate the quality of the halo fit from the mean value in a sky-patch near the DS.

We see here on the x-axis the value of the disc centre coordinate (i.e. column number in the image) and on the y-axis the mean value of the DS sky patch. We seem to have some scatter as well as a structure that looks like an inverted parabola, for these points near y-value 0. That is – the sky-patch mean value of the DS in images where the halo has been removed with the present EFM method depends on where the lunar disc is – the further away from x0=256 (middle of image) the Moon is, the larger a residual is left on the sky after the halo is removed with the EFM method.

We need to invent a better EFM method!

In this post:
http://iloapp.thejll.com/blog/earthshine?Home&post=268
I pointed at the unwanted structure in the sky near the lunar disc in an image that was Bias-reduced as well as had had its halo removed. Bias problems were ruled out – and left was a speculation on internal reflections in the camera system.

If we are truly left with that horrible structure on the sky we cannot just remove the halo in the present way, as the method depends on ‘fitting the sky’. If there are basins and hills in the ‘sky’ the fit will be bad – if the halo subtracted has the right shape but not the right ‘level’ then perhaps we can think of a fix: After removing the halo, but before extracting photometry from the DS disc in the image we can reference the sky adjacent to the DS and offset our disc value from there.

I tried this, by estimating the sky level on a part of the sky near the DS on an image where the halo has been subtracted, following the EFM method:

The two patches on the lunar disc are the Grimaldi and Crisium reference patches (DS and BS, respectively [although the BS value is estimated from the same pixels in the raw image]) and the large semi-rectangle on the sky next to the DS is the ‘skypatch reference area’. We calculate the average value of that patch and subtract it from the value extracted for the DS. If the error is of an offset type, rather than a slope type we have then corrected for the halo misfit.

To see the effect of this problem we estimated the DS/BS ratio in all EFM-cleaned images, for each filter, with and without referencing to the sky level:

As usual, the jpegs above are poor in quality so we also post the pdfs:

Each panel contains three columns – the rows are for filters. The first column is the DS/BS ratio as function of lunar phase (Full Moon is at phase 0). The second column is the ratio of the DS and the total RAW image flux [for reasons explained elsewhere!]. The third column shows the ‘alfa’ value derived by the EFM method.

Almost the same list of images were used for the two plots above – note the differences by inspecting the alfa plots – crosses are present in one plot but not the other one.

We see that the sequences of points for the case ‘skyptach reference level removed’ are slightly fuzzier than the sequence where the reference is not subtracted! Before we can interpret this I think we need to restrict the two plots to the same set of images – I shall do this and return …

[later]: This has now been done. For the IRCUT filter (341 data points) we show the effect of skypatch-referencing on the DS/BS ratio expressed as obs/model:

First we notice the wide span in DS/BS along both axes – this is not new: this is mainly the phase-dependence we see – this is MUCH LESS than would be seen for BBSO-method treated images, and is not the point – the point is that the spread along y for a given value of x – say near x=1 is something like 50%: That is – the effect of removing the sky-level from the DS value, in EFM-cleaned images, is to alter the DS/BS ratio by 50%. This means that the EFM-method did not do a very good job of removing the halo. It did remove a lot of the halo – but clearly not so much that a considerable effect is felt when the small remaining offset is removed.

This is important for our Science Goals: On the one hand we see the EFM method work much better than ‘BBS linear’ and ‘BBSO log’ methods in removing the effect of phase on the DS/BS ratio [shown elsewhere in this blog], on the other hand we see it does not do a sufficient job.

We should note that the BBSO linear method removes the sky level in one step (since it is a fit to the sky near the DS) but that, being linear, it does a poor job of removing the halo where it matters most – on the DS disc. The EFM method may be much better at removing a halo of approximately the right shape – but there remains a bias that is important.

The current EFM method is not ‘anchored in the sky’ – it merely seeks to minimize the residuals formed when an empirically generated halo is subtracted from the observed image – as evaluated on a masked section of the sky part of the image. There are choices made when that mask is set up, and they are:

a) The mask used above was such that sky was included on both sides of the lunar disc, set off from the disc by some 20 pixels and curtailed vertically at 0.7 radii (so that a ‘band across the lunar disc, but not including the disc’ is set up). This gave weight to both the part of the halo on the BS and the DS.

b) The mask is quite large to give access to a large number of pixels so that the effect of noise on the fitting-procedure was reduced.

The requirement in b) forces the choice in a) to be so large that any unevennesses in the sky (such as we suspect occurs due to internal reflections when the Moon is near the side of the image) has an influence on the quality of the fit, forcing the above consideration of a ‘special removal of the sky level near the DS’. To this comes the problems of allowing the BS halo to influence the fit.

We should investigate how we can improve the EFM method. Possibilities include

1) spatially weighting pixels so that ‘difficult areas’ are avoided or are given less influence on the final fit – such as the bothersome BS part of the halo.

2) use much smaller areas to fit the halo on the DS to – such as a patch near the DS patch on the lunar disc. This has to be balanced – on the one hand we get more influence of noise (which influences the BBSO method too, as it uses ‘narrow radial conical segments’ on the sky), but on the other hand we eliminate the effect of the BS halo as well as the ‘unevennesses in the sky field’.

3) One hybrid form of the EFM, tested earlier, consisted of enforcing that the model halo should be flux conserving (as now) and at the same time pass through an average point on the sky near the DS in the observed image.

We should implement a method that does not show the above sensitivity to removal of a sky reference level calculated on the already cleaned-up image. Access to a system without these internal reflections would also be nice. One significant step to take before that could be to find all images without the ‘streak’ effect that lead to the above considerations.

A strange structure has been found in some reduced images. To the right we show a dark frame taken seconds before the image on the left. The image on the left is an EFM image – i.e. the halo from the BS has been subtracted after fitting to the sky areas of the image. We clearly see the structure in the left image – a ‘band’ stretching to the left of the DS. This structure is not present in the dark image – so it is not an artefact of bias subtraction [Of course – the bias subtraction is not performed with adjacent dark frames – noise would be added that way – rather, a scaled almost noise-free superbias is subtracted. It is like a lower-noise replica of the image on the right – i.e. no structure, just mild level of noise.]. Below the two we show a plot along a vertical axis in the two images: columns 50 to 150 were averaged and shown in the plot as the black and the red lines.

There is a very clear structure in the sky of the Moon image. Given that it is not induced by bias-subtraction it must be due to the presence of the Moon itself. We speculate that:

a) it is some optical effect – reflection – from the inside of the camera or telescope. Halo-subtraction has reduced the sky level to almost zero but a little too much has been subtracted in the ‘dark band’ and a little too little has been removed outside the band. The fitting of a rather smooth ‘halo’ from the observed image could give this effect, if the structure itself is present in the image.

b) some electronic effect is causing the rows with the very bright BS in to somehow ‘jump low’ due to some effect we do not understand. NB: The readout direction is ‘down’ – not to the left!

Note the presence of what looks like a truncated halo to the left in the image, at the frame edge: this could be an internal reflection showing the right hand side of the halo being reflected inside the camera. We saw this same effect in the animated sequence of the tau Tauri occultation – as the Moon migrated to the RHS of the image frame an ‘echo’ appeared on the left.

This therefore seems to support idea a) above. If this is the case we may have an effect on solution-quality from position of Moon in the image. We should investigate if there is a position nearer the centre that eliminates the effect, and then omit images that are too close to the edge.

We inspect four flat fields taken of the Hohlraum lamp using AIR, ND1.0, ND1.3 and ND2.0 (ND0.9 was skipped as it is apparently AIR). All images were exposed so that count levels were between 10000 and 56000.

At upper left is AIR, upper right is ND1.0, then ND1.3 lower left and ND2.0 lower right.

Looking at these images there is evidently some light-leakage problem at the upper right corner, that has to do with which ND filter is used – or the lamp shifted slightly between exposures. Other experiences with the Hohlraum lamp underline how exactly positioned the telescope must be – this is difficult since small mount alignment problems apparently accumulate over time (the ‘creep problem’). The mount should therefore be freshly calibrated before use of the Hohlraum, which is not always convenient. It must furthermore be calibrated on a part of the sky near the lamp or the ‘offset problem’ will be evident.

‘offset’ and ‘creep’ are relatives, I think – something to do with the axes of the mount and the polar alignment.

A wishlist for this system might include an item about ‘proper polar axis alignment’ or perhaps an item about ‘automatic centering of telescope on flattest part of hohlraum field’.

I am testing a new method to calculate bias and flat fields. The principle is this: If several images are taken of the same scene at different exposure times then for each pixel you could plot counts against exposure time. They should lie on a straight line (if the shutter works so we can trust exposure time) with intercept at the bias value for that pixel and the slope equal to the gain. The gain and the flat field are the same thing, just scaled differently.

I did this for the V filter using AIR, N1.0, ND1.3, and ND2.0, and using the hohlraum lamp as the source. There is a ND0.9 filter setting that appears to be just AIR, so we ignore that one. For each pixel the regression of exposure time (taken as the requested exposure time – as noted in this blog the measured exposure time no longer does anything, it seems) against counts was made and the intercept and slope stored, along with uncertainties on these, were collected as well as the correlation between regressor and regressand.

More results are on the way but so far this is what we get for the flat field and its uncertainty:

On the left is the flatfield normalized to mean=1. On the right is the uncertainty in each pixel expressed as a percentage of the FF. The color bar at the bottom gives the colors for the uncertainty – i.e the uncertainties are in the range 0.19-0.28%.

The FF looks familiar – bars, spots and the dge effects. I suggest it could be compared to the ones Henriette has from analysis of a few sky sessions as well as the dome flats and some lamp flats, for the thesis.

The plot for the ‘bias’ is more puzzling:

The bias is on the left and the values are near 650-690. The uncertainty is on the right and it is in the range of 10%.

The value for the bias is odd – we are used to 394-396. The uncertainty is huge. We see the signature of ‘dust spots’ on the bias and a little of the ‘barred structure’ that we know from the FF. There seems to be some ‘cross-talk’ during the regression.

I think the large calculated bias is due to an offset in the exposure times – if the requested exposure time is larger than the actual this will cause an overestimate of the intercept – i.e. the bias. We can calculate the offset from the difference between calculated and observed bias and the gain. We have the observed gain from a large number of dark frames taken at the same time as the V images. Calculated and observed biases and the gain are available as images, and this yields the offset as an image of the same size. Taking the average of this we find 0.010 +/- 0.0008 s. That is – the exposures (at least this time) were shorter by 10ms than what we requested. The spread in the offset is due to the roughly 10% noise found in the calculated bias image, not the much more accurately determined gain image.

For this trial the requested exposure time and counts are extremely closely correlated, so the shutter was well-behaved.

The plot shows correlation between exposure times and counts for all pixels.

Does the above suggest a hybrid procedure – subtract the known bias taken
from dark frames and fit the rest, keeping the slope as the gain?

Note added later: The method depends on the source being time-invariant and spatially flat. The main interest right now is that the technique allows an estimation of the pixel.to-pixel uncertainty of the derived flat-field.
This can be compared to other methods that depend on averaging of FF exposures.

Our CCD has a number of bad pixels with lower than average sensitivity. They are a result of imperfect manufactoring, and most of them are collected in small groups. Their numbers and location are stable, and almost all of them are completely removed with even a mediocre flatfield.

But the most severe of the bad pixels, seem to have different strength in different master flatfields, and therefore they show up in diminished form in difference images between two master flatfields. The worst pixels may be 6-7% different from one master to the next, and therefore these pixels should be avoided. Luckily most of the bad pixels are close to the edge of the CCD. There is one semi-severe group in the central part of the CCD. I have seen examples of 1.7% variation in a couple of pixels from this group.

The figure is the twilight master flatfield in the IRCUT- filter from session JD 2455827, shown in power-scale to emphasize the small dark speck that are the bad pixels. The three most severe groups of bad pixels are circled in green, and the two second worst ones are circled in blue. These five groups of pixels should be avoided if possible – just to be on the safe side.

I have improved the plot showing the periodic bias level for multiple areas on the CCD. I have added arbitrary offsets to separate the different datasets from each other, and I have included a small figure showing the location of the five 16×16 areas on the CCD.

We have seen variations in the bias level (and dark current) that we do not understand. We went back to all 6000 dark frames that we have accumulated since the days in Lund until now. We plot the median bias level against date, and we plot the median bias level against chip temperature.

The top panel shows that usually the bias level is near 397 or so, but that excursions have occurred – the plot even omits one point showing a frame with median bias 1028. Near day 270 (JD 2455745 = July 2nd 2011) a very strange event occurs. In the bottom panel we see that that event is unrelated to temperature issues.

The red line, is a fit to the temperature-dependent data – the slope is about 0.067 counts per degrees F, or 0.037 counts/degree C.

The present problems with the camera, showing bias levels near 410 does not seem to strange any more, but we are unshure if the dark current has previously been as strange as it is now.

We note that during the observation of asteroid YU55 we saw what we thought was a strong flat field pattern – but it could not have been that since the dust spots are absent.

Can a flat-field like pattern be generated by high dark current?

I have plotted the mean dark value as function of exposure time from the dark frames obtained night JD2455883. The range of exposure times is 10-200 seconds and thus covers all higher exposure times we might be interested in. No dark current is observed, only the usual scatter due to the 20 minutes period. The period was clearly seen in a plot from the same data of dark count as function of time since first frame.

I think it is safe to say that we no longer have to make dark frames – only bias frames. This will save observing time, and therefore there should be time to ALWAYS obtain a bias frame before and after each science frame or flatfield. This will improve the scaling of the superbias.

The horizontal line in the plot is the mean value of the superbias. It can be seen that it is not in the middle of the scattered dark values. I have seen this before, as well as the opposite with dark values generally being higher than the mean of the supebias. However, in both cases values are within plus/minus 0.5 ADU and the scaling of the superbias means it is not a problem.

See this entry for a test of the Diffuser

http://iloapp.thejll.com/blog/earthshine?Home&post=83

There is an ‘integrating sphere’ in the dome at MLO and we can now use it for flat field observations. Preliminary results suggest that the problems with gradients are as procounced as with other methods, and that the ‘interference pattern’ seen in VE2 with other methods are different when seen using the lamp.

The ND0.9 filter was also used and flat fields attempted – but maybe the shutter is prone to sticking during long exposures and the low temperatures we had today (4-5 degrees C).

The lamp is at Altitude 4 degrees and Azimuth 291 degrees. Some experimenst should be performed to find the best alignment.

Flat field calculated using Chae’s method. It is based on 115 lunar images ‘dithered’ randomly around a central point. The dust speks near the middle are recognized from conventional FFs. The two diagonal stripes are also recognized as part of this CCDs extraordinarily strong fixed-pattern. The gradient from blue to pink and white corresponds to 2%. The image is based on B-band images from the night JD2455847.

It turns out that ‘slopes’ in this method has to do with the normalization of the source intensity – the method depends on the source being constant, but the source is not constant if it is partially on the dge of the image frame, is it? We are working on this.