AIMLESS (CCP4: Supported Program) NAME aimless - scale together multiple observations of reflections SYNOPSIS aimless HKLIN foo_in.mtz HKLOUT foo_out.mtz [Keyworded Input] References Input and Output files Release Notes DESCRIPTION Scaling options Control of flow through the program Partially recorded reflections Scaling algorithm Data from Denzo Datasets This program scales together multiple observations of reflections, and merges multiple observations into an average intensity: it is a successor program to SCALA Various scaling models can be used. The scale factor is a function of the primary beam direction, either as a smooth function of Phi (the rotation angle ROT), or expressed as BATCH (image) number (deprecated). In addition, the scale may be a function of the secondary beam direction, acting principally as an absorption correction expanded as spherical harmonics. The secondary beam correction is related to the absorption anisotropy correction described by Blessing (Ref Blessing (1995) ). The merging algorithm analyses the data for outliers, and gives detailed analyses. It generates a weighted mean of the observations of the same reflection, after rejecting the outliers. The program does several passes through the data: 1. initial estimate of the scales 2. first round scale refinement, using strong data using an I/sigma(I) cutoff 3. first round of outlier rejection 4. if both summation and profile-fitted intensity estimates are present (from Mosflm), then the cross-over point is determined between using profile-fitted for weak data and summation for strong data. 5. first analysis pass to refine the "corrections" to the standard deviation estimates 6. final round scale refinement, using strong data within limits on the normalised intensity |E|^2 7. final analysis pass to refine the "corrections" to the standard deviation estimates 8. final outlier rejections 9. a final pass to apply scales, analyse agreement & write the output file, usually with merged intensities, but alternatively as file with scaled but unmerged observations, with partials summed and outliers rejected, for each dataset Anomalous scattering is ignored during the scale determination (I+ & I- observations are treated together), but the merged file always contains I+ & I-, even if the ANOMALOUS OFF command is used. Switching ANOMALOUS ON does affect the statistics and the outlier rejection (qv) Scaling options The optimum form of the scaling will depend a great deal on how the data were collected. It is not possible to lay down definitive rules, but some of the following hints may help. For most purposes, my normal recommendation is the default scales rotation spacing 5 secondary bfactor on brotation spacing 20 Other hints:- 1. Only use the SCALE BATCH option if every image is different from every other one, i.e. off-line detectors (including film), or rapidly or discontinuously changing incident beam flux. This is rarely the case for synchrotron data. 2. If there is a discontinuity between one set of images and another (e.g. change of exposure time), then flag them as different RUNs. This will be done automatically if no runs are specified. 3. The SECONDARY correction is recommended and is the default: this provides a correction for absorption. It should always be restrained with a TIE SURFACE command (this is the default): under these conditions it is reasonably stable under most conditions. The ABSORPTION (crystal frame) correction is similar to SECONDARY (camera frame) in most cases, but may be preferable if data has been collected from multiple alignments of the same crystal. 4. Use a B-factor correction unless the data are only very low-resolution. Traditionally, the relative B-factor is a correction for radiation damage (hence it is a function of time), but it also includes some other corrections eg absorption. 5. When trying out more complex scaling options, it is a good idea to try a simple scaling first, to check that the more elaborate model gives a real improvement. 6. When scaling multiple MAD data sets they should all be scaled together in one pass, outliers rejected across all datasets, then each wavelength merged separately. This is the default if multiple datasets are present in the input file. Other options are described in greater detail under the KEYWORDS. Control of flow through the program The ONLYMERGE flag skips the scaling (usually in conjuction with RESTORE to read in previously determined scales), calculates statistics and outputs the data. Partially recorded reflections See appendix 1 The different options for the treatment of partials are set by the PARTIALS command. Partials may either be summed or scaled : in the latter case, each part is treated independently of the others. Summed partials [default]: All the parts are summed (after applying scales) to give the total intensity, provided some checks are passed. The number of reflections failing the checks is printed. You should make sure that you are not losing too many reflections in these checks. Scaled partials: In this option, each individual partial observation scaled up by the inverse FRACTIONCALC, provided that the fraction is greater than [default = 0.5]. This only works well if the calculated fractions are accurate, which is not usually the case. Scaling algorithm See appendix 2 Data from Denzo Data integrated with Denzo may be scaled and merged with Aimless as an alternative to Scalepack, or unmerged output from scalepack may be used. Both have some limitations. See appendix 3 for more details. Datasets TBD KEYWORDED INPUT - DESCRIPTION In the definitions below "[]" encloses optional items, "|" delineates alternatives. All keywords are case-insensitive, but are listed below in upper-case. Anything after "!" or "#" is treated as comment. The available keywords are: ANALYSIS, ANOMALOUS, BINS, DUMP, EXCLUDE, INITIAL, INTENSITIES, NAME, ONLYMERGE, OUTPUT, PARTIALS, REFINE, REJECT, RESOLUTION, RESTORE, RUN, SCALES, SDCORRECTION, TIE, TITLE RUN BATCH to Define a "run" : Nrun is the Run number, with an arbitrary integer label (i.e. not necessarily 1,2,3 etc). A "run" defines a set of reflections which share a set of scale factors. Typically a run will be a continuous rotation around a single axis. The definition of a run may use several RUN commands. If no RUN command is given then run assignment will be done automatically, with run breaks at discontinuities in dataset, batch number or Phi. If any RUN definitions are given, then all batches not explicitly specified will be excluded. SCALES [] Define layout of scales, ie the scaling model. Note that a layout may be defined for all runs (no RUN subkeyword), then overridden for particular runs by additional commands. Subkeys: RUN Define run to which this command applies: the run must have been previously defined. If no run is defined, it applies to all runs ROTATION | SPACING Define layout of scale factors along rotation axis (i.e. primary beam), either as number of scales or (if SPACING keyword present) as interval on rotation [default SPACING 5] BATCH Set "Batch" mode, no interpolation along rotation (primary) axis. This option is compulsory if a ROT column is not present in the input file, but otherwise the ROTATION option is preferred. WARNING: this option is not optimised and may take a very long time if you have many batches BFACTOR ON | OFF Switch Bfactors on or off. The default is ON. BROTATION | SPACING Define number of B-factors or (if SPACING keyword present) the interval on "time": usually no time is defined in the input file, and the rotation angle is used as its proxy [default SPACING 20]. SECONDARY [] Secondary beam correction expanded in spherical harmonics up to maximum order Lmax in the camera spindle frame. The number of parameters increases as (Lmax + 1)**2, so you should use the minimum order needed (eg 4 - 6, default 4). The deviation of the surface from spherical should be restrained eg with TIE SURFACE 0.001 [default] ABSORPTION [] Secondary beam correction expanded in spherical harmonics up to maximum order Lmax in the crystal frame based on POLE (qv). The number of parameters increases as (Lmax + 1)**2, so you should use the minimum order needed (eg 4 - 6, default 4). The deviation of the surface from spherical should be restrained eg with TIE SURFACE 0.001 [default]. This is not substantially different from SECONDARY in most cases, but may be preferred if data are collected from multiple settings of the same crystal, and you want to use the same absorption surface. This would only be strictly valid if the beam is larger than the crystal. POLE Define the polar axis for ABSORPTION or SURFACE as h, k or l (eg POLE L): the pole will default to either the closest axis to the spindle (if known), or l (k for monoclinic space-groups). CONSTANT One scale for each run (equivalent to ROTATION 1) TILE [CCD] Define a detector scale for each tile. Currently this implements a scale model for 3x3 tiled CCD detectors to correct for the underestimation of intensities in the corners of the tile, see Appendix 2. If the detector appears to be a 3x3 CCD (3072x3072 pixels) then this correction will be activated automatically unless the NOTILE keyword is given. The parameters are restrained using the TIE TILE parameters (qv) NOTILE Switch off the automatic TILE 3 3 correction for CCD detectors SDCORRECTION [[NO]REFINE] [INDIVIDUAL | SAME [FIXSDB] [RUN ] [FULL | PARTIAL] [] [DAMP ] [SIMILAR [ ]] || [[NO]TIE SdFac | SdB | SdAdd ] Input or set options for the "corrections" to the input standard deviations: these are modified to sd(I) corrected = SdFac * sqrt{sd(I)**2 + SdB*Ihl + (SdAdd*Ihl )**2} where Ihl is the intensity and (SdB may be omitted in the input). The default is "SDCORRECTION REFINE INDIVIDUAL", If explicit values are given, the default changes to NOREFINE. The keyword REFINE controls refinement of the correction parameters, essentially trying to make the plot of the SD of the distribution of fraction deviations (Ihl - )/sigma = 1.0 over all intensity ranges. The residual minimised is Sum( w * (1 - SD)^2) + Restraint Residual Other subkeys control what values are determined and used for each run (if more than one). TIE and SIMILAR are mutually exclusive * SAME same SD parameters for all runs, different for fulls and partials * INDIVIDUAL [default] use different SD parameters for each run, fulls and partials * FIXSDB fixes the SdB parameter in the refinement (but it seems best to let it refine, even though it has no obvious physical meaning) * DAMP set dampfactor to damp shifts in the refinement [default 0.05] * SIMILAR restrain parameters to be the same for all runs, with SDs optionally given for SdFac (sd1), SdB (sd2), and SdAdd (sd3) [defaults 0.2, 3.0, 0.04) * TIE set restraints for named parameter, "SdFac", "SdB", or "SdAdd". Each restraint is to a specified target value, with a weight = 1/(SDtarget^2). The default is to restrain SdB only, target value 0.0, SD 20. NOTIE removes all restraints, TIE without values sets the defaults. RUN Define run for which values are given the run must have been previously defined. If no run is defined, it applies to all runs. Different values may be specified for fully recorded reflections (FULL) and for partially recorded reflections (PARTIAL), or the same values may be used for both if one set is given, e.g. sdcorrection full 1.4 0.11 part 1.4 0.05 PARTIALS [[NO]CHECK] [TEST [ ] [CORRECT ] [[NO]GAP []] Set criteria for accepting complete or incomplete partials. Default is CHECK TEST 0.95 1.05 CORRECT 0.95 NOGAP After all parts have been assembled, the total observation is accepted if:- 1. the CHECK flag is set [default] and the MPART flags (if present) are all consistent (these flags indicate that a set of parts is eg 1 of 3, 2 of 3, 3 of 3) 2. if CHECK fails, then the total fraction is checked to lie between lower_limit & upper_limit [default 0.95, 1.05] 3. if this fails, then the incomplete partial is scaled up by the total fraction if it is > minimum_fraction [default 0.95] (NB Pointless has different default for a different purpose) 4. a reflection has a gap in the middle may be accepted if GAP is set, maxgap is maximum number of missing slots [not recommended: default 1 if GAP is set] INITIAL UNITY | MEAN Set initial scale factors either based on mean intensities (MEAN, default) or all set to 1.0 (UNITY) INTENSITIES [SUMMATION | PROFILE | COMBINE [] [POWER ] Set which intensity to use, of the integrated intensity (column I) or profile-fitted (column IPR), if both are present. This applies to all stages of the program, scaling & averaging. Mosflm produces two different estimates of the intensity, from summation integration and from profile fitting. Generally the profile-fitted estimate is better, but for the strongest reflections the summation value is often better. The default is to use a weighted mean, depending on the "raw" intensity ie before LP correction (COMBINE option), and to optimise automatically the switch-over point Imid, to give the best overallR [meas]. Subkeys: SUMMATION use summation integrated intensity Isum. PROFILE use profile-fitted intensity Ipr. COMBINE [] [POWER ] Use weighted mean of profile-fitted & integrated intensity, profile-fitted for weak data, summation integration value for strong. If no value is given for Imid, it will be automatically optimised I = w*Ipr + (1-w)*Isum w = 1/(1 + (Iraw/Imid)**Ipower) Ipower defaults to 3. REJECT [SCALE | MERGE] [COMBINE] [SEPARATE] [] [ALL []] [KEEP | REJECT | LARGER | SMALLER] [EMAX ] Define rejection criteria for outliers: different criteria may be set for the scaling and for the merging passes. If neither SCALE nor MERGE are specified, the same values are used for both stages. The default values are REJECT 6 ALL -8, ie test within I+ or I- sets on 6sigma, between I+ & I- with a threshold adjusted upwards from 8sigma according to the strength of the anomalous signal. The adjustment of the ALL test is not necessarily reliable. If there are multiple datasets, by default, deviation calculations include data from all datasets [COMBINE]. The SEPARATE flag means that outlier rejections are done only between observations from the same dataset. The usual case of multiple datasets is MAD data. If ANOMALOUS ON is set, then the main outlier test is done in the merging step only within the I+ & I- sets for that reflection, ie Bijvoet-related reflections are treated as independent. The ALL keyword here enables an additional test on all observations including I+ & I- observations. Observations rejected on this second check are flagged "@" in the ROGUES file. Subkeys: SEPARATE rejection & deviation calculations only between observations from the same dataset COMBINE rejection & deviation calculations are done with all datasets [default] SCALE use these values for the scaling pass MERGE use these values for the merging (FINAL) pass sdrej sd multiplier for maximum deviation from weighted mean I [default 6.0] [sdrej2] special value for reflections measured twice [default = sdrej] ALL check outliers in merging step between as well as within I+ & I- sets (not relevant if ANOMALOUS OFF). A negative value [default -8] means adjust the value upwards according to the slope of the normal probability analysis of anomalous differences (AnomPlot) sdrej+- sd multiplier for maximum deviation from weighted mean I including all I+ & I- observations (not relevant if ANOMALOUS OFF) [sdrej2+-] special value for reflections measured twice [default = sdrej+-] KEEP in merging, if two observations disagree, keep both of them [default] REJECT in merging, if two observations disagree, reject both of them LARGER in merging, if two observations disagree, reject the larger SMALLER in merging, if two observations disagree, reject the smaller EMAX maximum acceptable value for E = normalised |F| [default = 10.0 for acentrics] The test for outliers is described in Appendix 4 ANOMALOUS [OFF] [ON] OFF [default] no anomalous used, I+ & I- observations averaged together in merging ON separate anomalous observations in the final output pass, for statistics & merging: this is also selected the keyword ANOMALOUS on its own RESOLUTION [RUN ] [[LOW] ] [[HIGH] ] Set resolution limits in Angstrom, either order, optionally for individual datasets. The keywords LOW or HIGH, followed by a number, may be used to set the low or high resolution limits explicitly: an unset limit will be set as in the input HKLIN file. If a RUN is specified this limit applies only to that run: this may a previous general limit for all runs, and may be used with automatic run generation. [Default use all data] TITLE Set new title to replace the one taken from the input file. By default, the title is copied from hklin to hklout ANALYSIS CONE CCMINIMUM ISIGMINIMUM BATCHISIGMINIMUM Specify analysis parameters: CONE specifies the half-angle (degrees) for cones around each reciprocal axis, for anisotropy analysis [default 20°]. CCMINIMUM & ISIGMINIMUM specify limits for estimation of suitable resolution cut-off limits, both overall and along each reciprocal axis. These estimates are printed in the final Results summary, and give guide to possible cut-offs. BATCHISIGMINIMUM gives the cutoff for the analysis of maximum resolution by batch, on before averaging * MinimumHalfdatasetCC minimum half-dataset CC [default 0.5] * MinimumIoverSigma minimum </sd()> (=~ signal/noise) [default 2.0] * MinimumBatchIoverSigma minimum (=~ signal/noise) [default 1.0, a smaller value as I/sd is before averaging] ONLYMERGE Only do the merge step, no initial analysis, no scaling. If RESTORE is also given, the SDCORRECTION optimising will also be skipped. DUMP [] Dump all scale factors to a file after the main scaling. These can be used to restart scaling using the RESTORE option, or for rerunning the merge step. If no filename is given, the scales will be written to logical file SCALES, which may be assigned on the command line. RESTORE [] Read scales and SDcorrection parameters from a SCALES file from a previous run of Aimless (see DUMP). REFINE [CYCLES ] [BFGS | FH] [SELECT []] [PARALLEL [AUTO] | | ] Define number of refinement cycles Ncycle and method for scale refinement. BFGS use BFGS optimisation (usual method) FH use Fox-Holmes least-squares algorithm (not recommended) SELECT define selection limits for the two rounds of scaling. If unset, suitable values will be chosen automatically * IovSDmin /sd'(I) limit for selection of reflections for 1st round scaling (< 0 for automatic selection) * E2min minimum E^2 for selection of reflections for main scaling [default 0.8] * E2max maximum E^2 for selection of reflections for main scaling [default 5.0] PARALLEL use multiple processors for the scale refinement steps, if available. This produces some speed-up for very large jobs. For this option to be available, the program must be compiled and linked with the "-fopenmp" option, and the environment variable OMP_NUM_THREADS must be set to the maximum number of threads allowed by the system * number of processors to use (this will be forced to be < OMP_NUM_THREADS) * (< 1.0) fraction of OMP_NUM_THREADS to use * AUTO [default if no argument to PARALLEL] determine the number of processors to use from the number of observations in the file, currently 1 processor / 200 000 observations, up to the maximum allowed (the optimum settings for this have yet to be determined) EXCLUDE BATCH |] BATCH | ... | TO | Define a list of batches, or a range of batches, to be excluded altogether. TIE [SURFACE ] [BFACTOR ] [ZEROB ] [ROTATION ] [TILE ] Apply or remove restraints to parameters. These can be pairs of neighbouring scale factors on rotation axis (ROTATION = primary beam) to have the same value, or neighbouring Bfactors, or surface spherical harmonic parameters to zero (for SECONDARY or SURFACE corrections, to keep the correction approximately spherical), with a standard deviation as given. This may be used if scales are varying too wildly, particularly in the detector plane. The default is no restraints on scales. A tie is recommended for SECONDARY or SURFACE corrections, eg TIE SURFACE 0.001. A negative SD value indicates no tie. SURFACE: tie surface parameters to spherical surface [default is TIE SURFACE 0.001] BFACTOR: tie Bfactors along rotation ZEROB: tie all B-factors to zero ROTATION: tie parameters along rotation axis (mainly useful with BATCH mode) TILE: tie the CCD tile parameters. 5 numbers for radius r, width w, amplitude A, centre x0,y0, and Fourier coefficients OUTPUT [MTZ] [NO]MERGED [UNMERGED [SPLIT|TOGETHER]] [SCALEPACK [MERGED | UNMERGED]] Control what goes in the output file. Two types of output files may be produced, either in MTZ format or in Scalepack format: (a) MERGED (or AVERAGE), average intensity for each hkl (I+ & I-) (b) UNMERGED, unaveraged observations, but with scales applied, partials summed or scaled, and outliers rejected. Up to four types of files may be created at the same time: UNMERGED filenames are created from the HKLOUT filename (with dataset appended if there are multiple datasets) with the string "_unmerged" appended. If there are multiple datasets, by default MTZ files, merged or unmerged, are split into separate files (SPLIT). Unmerged MTZ files may optionally include all datasets if the keyword TOGETHER qualifies UNMERGED. The default is to create a merged MTZ file for each dataset. File format options: NONE no output file written MERGED or AVERAGE [default] output averaged intensities, & for each hkl UNMERGED apply scales, sum or scale partials, reject outliers, but do not average observations SCALEPACK or POLISH Write reflections to a formatted file in a format as written by "scalepack" (or my best approximation to it). If the UNMERGED option is also selected, then the output matches the scalepack "output nomerge original index", otherwise it is the "normal" scalepack output, with either I, sigI or I+ sigI+, I-, sigI-, depending on the "anomalous" flag. KEEP [OVERLOADS|BGRATIO |PKRATIO |GRADIENT |EDGE] Set options to accept observations flagged as rejected by the FLAG column from Mosflm. By default, any observation with FLAG .ne. 0 is rejected. Flagged reflections which are accepted may be marked in the ROGUES file. Subkeys: OVERLOADS Accept profile-fitted overloads BGRATIO Observations are flagged in Mosflm if the ratio of rms background deviation relative to its expected value from counting statistics is too large. This option accepts observations if bgratio < bgratio_max [default in Mosflm 3.0] PKRATIO Accept observations with peak fitting rms/sd ratio pkratio < pkratio_max [default maximum in Mosflm 3.5]. Only set for fully recorded observations GRADIENT Accept observations with background gradient < bg_gradient_max [default in Mosflm 0.03]. EDGE Accept profile-fitted observations on edge of active area of detector LINK [SURFACE] ALL | TO NOT YET DONE run_2 will use the same SURFACE (or SECONDARY) parameters as run_1. This can be useful when different runs come from the same crystal, and may stabilize the parameters. The keyword ALL will be assumed if omitted. * For SECONDARY or SURFACE parameters, the default is to link runs which come from the same dataset. They should be UNLINKed if they are different. UNLINK [SURFACE|TAILS] ALL | TO NOT YET DONE Remove links set by LINK command (or by default). The keyword ALL will be assumed if omitted BINS [RESOLUTION] INTENSITY Define number of resolution and intensity bins for analysis [default 10] SMOOTHING NOT YET DONE Set smoothing factors ("variances" of weights). A larger "variance" leads to greater smoothing Subkeys: TIME smoothing of B-factors [default 0.5] ROTATION smoothing of scale along rotation [default 1.0] PROB_LIMIT maximum values of normalized squared deviation (del**2/V) to include a scale [default set automatically, typically 3. NAME PROJECT CRYSTAL DATASET Assign or reassign project/crystal/dataset names, for output file. The names given here supersede those in the input file and redefines the single output dataset. Note that these names apply to all data: if multiple datasets are required, these must be specified in Pointless. DATASET must be present, and may optionally be given in the syntax crystal_name/dataset_name BASE [CRYSTAL ] DATASET NOT YET DONE If there are multiple datasets in the input file, define the "base" dataset for analysis of dispersive (isomorphous) differences. Differences between other datasets and the base dataset are analysed for correlation and ratios, ie for the i'th dataset (I(i) - I(base)). By default, the datasets with the shortest wavelength will be chosen as the base (or dataset 1 if wavelength is unknown). Typically, the CRYSTAL keyword may be omitted. INPUT AND OUTPUT FILES Input HKLIN The input file must be sorted on H K L M/ISYM BATCH Compulsory columns: H K L indices M/ISYM partial flag, symmetry number BATCH batch number I intensity (integrated intensity) SIGI sd(intensity) (integrated intensity) Optional columns: XDET YDET position on detector of this reflection: these may be in any units (e.g. mm or pixels), but t he range of values must be specified in the orientation data block for each batch. ROT rotation angle of this reflection ("Phi"). If this column is absent, only SCALES BATCH is va lid. IPR intensity (profile-fitted intensity) SIGIPR sd(intensity) (profile-fitted intensity) SCALE previously calculated scale factor (e.g. from previous run of Scala). This will be applied on input SIGSCALE sd(SCALE) TIME time for B-factor variation (if this is missing, ROT is used instead) MPART partial flag from Mosflm FRACTIONCALC calculated fraction, required to SCALE PARTIAL S LP Lorentz/polarization correction (already appli ed) FLAG error flag (packed bits) from Mosflm (v6.2.3 or later). By default, if this column is prese nt, observations with a non-zero FLAG will be omitted. They may be conditionally accepted using the KEEP command (qv) Bit flags: 1 BGRATIO too large 2 PKRATIO too large 4 Negative > 5*sigma 8 BG Gradient too high 16 Profile fitted overload 32 Profile fitted "edge" reflecti on BGPKRATIOS packed background & peak ratios, & background gradient, from Mosflm, to go with FLAG Output Reflection files output In all cases, separate files are written for each dataset: files are named with the base HKLOUT name with the dataset name appended, as "_dataset" (a) HKLOUT: option OUTPUT [MTZ] MERGED The output file contains columns H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) Note that there are no M/ISYM or BATCH columns. I(+) & I(-) are the means of the Bijvoet positive and negative reflections respectively and are always present even for the option ANOMALOUS OFF. (b) HKLOUTUNMERGED: option OUTPUT [MTZ] UNMERGED Unmerged data with scales applied, with no partials (i.e. partials have been summed or scaled, unmatched partials removed), & outliers rejected. Only a single scaled intensity value is written, chosen as summation, profile-fitted or combined as specified by the INTENSITIES command. Columns defining the diffraction geometry (e.g. FRACTIONCALC XDET YDET ROT TIME WIDTH LP) will be preserved in the output file. If HKLOUTUNMERGED is not specified, then the filename for the unmerged file has "_unmerged" appended to HKLOUT Output columns: H,K,L REDUCED or ORIGINAL indices (see OUTPUT options) M/ISYM Symmetry number (REDUCED), = 1 for ORIGINAL indices BATCH batch number as for input I, SIGI scaled intensity & sd(I) SCALEUSED scale factor applied SIGSCALEUSED sd(SCALE applied) NPART number of parts, = 1 for fulls, negated for scaled partials, i.e. = -1 for scaled single part partial FRACTIONCALC total fraction (if present in input file) TIME copied from input if present XDET,YDET copied from input if present ROT copied from input if present (averaged for multi-part partials) WIDTH copied from input if present LP copied from input if present (c) SCALEPACK: option OUTPUT SCALEPACK MERGED If a SCALEPACK filename is not specified then the filename will be taken from HKLOUT with the extension ".sca" (d) SCALEPACKUNMERGED: option OUTPUT SCALEPACK UNMERGED If a SCALEPACKUNMERGED filename is not specified then the filename will be taken from SCALEPACK with "_unmerged" appended and the extension ".sca" Other files XMLOUT XML output for plotting etc. It includes the NORMPLOT, ANOMPLOT, CORRELPLOT and ROGUEPLOT data, as well as the $TABLE graph data SCALES scale factors from DUMP, used by RESTORE option ROGUES list of bad agreements TILEIMAGE a detector image representing the CCD TILE correction, if activated, in ADSC image format which may be viewed with adxv The following 4 files are also represented in the XMLOUT file: NORMPLOT normal probability plot from merge stage *** this is at present written is a format for plotting program xmgr (aka [xm]grace), but can also be read by loggraph *** ANOMPLOT normal probability plot of anomalous differences (I+ - I-)/sqrt[sd(I+)**2 + sd(I-)**2] *** this is at present written is a format for plotting program xmgr (aka grace), but can also be read by loggraph *** CORRELPLOT scatter plot of pairs of anomalous differences (in multiples of RMS) from random half-datasets. One of these files is generated for each output dataset *** this is at present written is a format for plotting program xmgr (aka grace), but can also be read by loggraph *** ROGUEPLOT a plot of the position on the detector (on an ideal virtual detector with the rotation axis horizontal) of rejected outliers, with the position of the principle ice rings shown *** this is at present written is a format for plotting program xmgr (aka grace), but can also be read by loggraph *** REFERENCES 1. P.R. Evans and ,G.N. Murshudov "How good are my data and what is the resolution?" Acta Cryst. (2013). D69, 1204–1214 2. P.R.Evans "An introduction to data reduction: space-group determination, scaling and intensity statistics", Acta Cryst. D67, 282-292 (2011) 3. P.R.Evans, "Scaling and assessment of data quality", Acta Cryst. D62, 72-82 (2006). Note that definitions of R[meas] and R[pim] in this paper are missing a square-root on the (1/n-1) factor 4. W. Kabsch, J.Appl.Cryst. 21, 916-924 (1988) 5. P.R.Evans, "Data reduction", Proceedings of CCP4 Study Weekend, 1993, on Data Collection & Processing, pages 114-122 6. P.R.Evans, "Scaling of MAD Data", Proceedings of CCP4 Study Weekend, 1997, on Recent Advances in Phasing, Click here 7. R.Read, "Outlier rejection", Proceedings of CCP4 Study Weekend, 1999, on Data Collection & Processing 8. Hamilton, Rollett & Sparks, Acta Cryst. 18, 129-130 (1965) 9. Blessing, R.H., Acta Cryst. A51, 33-38 (1995) 10. Kay Diederichs & P. Andrew Karplus, "Improved R-factors for diffraction data analysis in macromolecular crystallography", Nature Structural Biology, 4, 269-275 (1997) 11. Manfred Weiss & Rolf Hilgenfeld, "On the use of the merging R factor as a quality indicator for X-ray data", J.Appl.Cryst. 30, 203-205 (1997) 12. Manfred Weiss, "Global Indicators of X-ray data quality" J.Appl.Cryst. 34, 130-135 (2001) Appendix 1: Partially recorded reflections In the input file, partials are flagged with M=1 in the M/ISYM column, and have a calculated fraction in the FRACTIONCALC column. Data from Mosflm also has a column MPART which enumerates each part (e.g. for a reflection predicted to run over 3 images, the 3 parts are labelled 301, 302, 303), allowing a check that all parts have been found: MPART = 10 for partials already summed in MOSFLM. Summed partials: All the parts are summed (after applying scales) to give the total intensity, provided some checks are passed. The parameters for the checks are set by the PARTIALS command. The number of reflections failing the checks is printed. You should make sure that you are not losing too many reflections in these checks. 1. if the CHECK option is set (the default if an MPART column is present), the MPART flags are examined. If they are consistent, the summed intensity is accepted. If they are inconsistent (quite common), the total fraction is checked (TEST). NOCHECK switches off this check. 2. if the TEST option is set (default), the summed reflection is accepted if the total fraction (the sum of the FRACTIONCALC values) lies between -> [default limits = 0.95 1.05] 3. if the CORRECT option is set, the total intensity is scaled by the inverse total fraction for total fractions between to . This works also for a single unmatched partial. This correction relies on accurate FRACTIONCALC values, so beware. 4. if the GAP option is set (not recommended), partials with a gap in are accepted, e.g. a partial over 3 parts with the middle one missing. The GAP option implies TEST & NOCHECK, & the CORRECT option may also be set. By setting the TEST & CORRECT limits, you can control summation & scaling of partials, e.g . TEST 1.2 1.2 CORRECT 0.5 will scale up all partials with a total fraction between 0.5 & 1.2 TEST 0.95 1.05 will accept summed partials 0.95->1.05, no scaling TEST 0.95 1.05 CORRECT 0.4 will accept summed partials 0.95->1.05, and scale up those with fractions between 0.4 & 0.95 Appendix 2: Scaling algorithm For each reflection h, we have a number of observations Ihl, with estimated standard deviation shl, which defines a weight whl. We need to determine the inverse scale factor ghl to put each observation on a common scale (as Ihl/ghl). This is done by minimizing Sum( whl * ( Ihl - ghl * Ih )**2 ) Ref Hamilton, Rollett & S parks where Ih is the current best estimate of the "true" intensity Ih = Sum ( whl * ghl * Ihl ) / Sum ( whl * ghl**2) Each observation is assigned to a "run", which corresponds to a set of scale factors. A run would typically consist of a continuous rotation of a crystal about a single axis. The inverse scale factor ghl is derived as follows: ghl = Thl * Chl * Shl where Thl is an optional relative B-factor contribution, Chl is a scale factor, and Shl is a anisotropic correction expressed as spherical harmonics (ie SECONDARY, ABSORPTION options). a) B-factor (optional) For each run, a relative B-factor (Bi) is determined at intervals in "time" ("time" is normally defined as rotation angle if no independent time value is available), at positions ti (t1, t2, . . tn). Then for an observation measured at time tl B = Sum[i=1,n] ( p(delt) Bi ) / Sum (p(delt)) where Bi are the B-factors at time ti delt = tl - ti p(delt) = exp ( - (delt)**2 / Vt ) Vt is "variance" of weight, & controls the smoothness of interpolation Thl = exp ( + 2 s B ) s = (sin theta / lambda)**2 b) Scale factors For each run, scale factors Cz are determined at intervals on rotation angle z. Then for an observation at position (z0), Chl(z0) = Sum(z)[p(delz)*Cxyz]/Sum(z)[p(delz)] where delz = z - z0 p(delz) = exp(-delz**2/Vz) Vz is the "variance" of the weight & controls the smoothness o f interpolation For the SCALES BATCH option, the scale along z is discontinuous: the normal option has one scale factor for each batch. c) Anisotropy factor The optional surface or anisotropy factor Shl is expressed as a sum of spherical harmonic terms as a function of the direction of (1) the secondary beam (SECONDARY correction) in the camera spindle frame, (2) the secondary beam (ABSORPTION correction) in the crystal frame, permuted to put either a*, b* or c* along the spherical polar axis 1. SECONDARY beam direction (camera frame) s = [Phi] [UB] h s2 = s - s0 s2' = [-Phi] s2 Polar coordinates: s2' = (x y z) PolarTheta = arctan(sqrt(x**2 + y**2)/z) PolarPhi = arctan(y/x) where [Phi] is the spindle rotation matri x [-Phi] is its inverse [UB] is the setting matrix h = (h k l) 2. ABSORPTION: Secondary beam direction (permuted crystal frame) s = [Phi] [UB] h s2 = s - s0 s2c' = [-Q] [-U] [-Phi] s2 Polar coordinates: s2' = (x y z) PolarTheta = arctan(sqrt(x**2 + y**2)/z) PolarPhi = arctan(y/x) where [Phi] is the spindle rotation matri x [-Phi] is its inverse [Q] is a permutation matrix to put h, k, or l along z (see POLE op tion) [U] is the orientation matrix [B] is the orthogonalization matri x h = (h k l) then Shl = 1 + Sum[l=1,lmax] Sum[m=-l,+l] Clm Ylm(PolarTheta,PolarPhi) where Ylm is the spherical harmonic funct ion for the direction given by the pola r angles Clm are the coefficients determined by the program Notes: * The initial term "1" is essentially the l = 0 term, but with a fixed coefficient. * The number of terms = (lmax + 1)**2 - 1 * Even terms (ie l even) are centrosymmetric, odd terms antisymmetric * Restraining all terms to zero (with the TIE SURFACE) reduces the anisotropic correction. This should always be done (d) Detector correction (TILES) A correction for tiled CCD detectors has been implemented to attempt to correct for the underestimation of spots falling in the corner of the detector. The present model expresses a correction factor in terms of an erfc function of the distance from the tile centre, such that the correction = 1 in the centre of the tile and falls off at the edge and corners For a spot at position x,y relative to the tile centre, normalised by the tile width in pixels such that x & y run from -1 to +1, then distance from centre (x0,y0) d = sqrt[(x-x0)^2 + (y-y0)^2] correction factor g = A f(z) + 1 - A where A is the amplitude of the correction near the edge and f(z) is a radial function of the modified "radius" z = (2/w)(d - r - w) . r defines the point at which the scale starts to decline from 1.0, and w the "width" of the fall-off Currently f(z) = 0.5 erfx(z) though other expressions have been tried Amplitude A various azimuthally with the angle phi = tan^-1(y/x) as a Fourier series, A = A0{a cos(phi) + b sin(phi) + c cos(2phi) + d sin(2phi)} Refined parameters for each tile are r, w, A0, x0, y0, and the four Fourier terms for A, a,b,c,d. By default, parameters are restrained (TIE) as follows (see TIE TILE) A0, a,b,c,d and x0,y0 are tied to 0.0 r, w, and A0 are tied to be similar over all tiles Five SD values control the strength of the restraints, respectively for r, w, A0, x0|y0, and abcd SD = 0 switches off the restraint ^ _________________________________________________ Appendix 3: Data from Denzo DENZO is often run refining the cell and orientation angles for each image independently, then postrefinement is done in Scalepack. It is essential that you do this postrefinement. Either then reintegrate the images with the cell parameters fixed, or use unmerged output from scalepack as input to Aimless. The DENZO or SCALEPACK outputs will need to be converted to a multi-record MTZ file using COMBAT (see COMBAT documentation) or POINTLESS (for Scalepack output only). Both of these options have some problems + If you take the output from Denzo into Scala, there may be problems with partially recorded reflections: it is difficult for Scala to determine reliably that it has all parts of a partial to sum together. + If you take unmerged output from scalepack into Aimless, most of the geometrical information about how the observations were collected is lost, so many of the scaling options in Aimless are not available. Only Batch scaling can be used, but simultaneous scaling of several wavelengths or derivatives may still be useful _________________________________________________ Appendix 4: Outlier algorithm The test for outliers is as follows: (1) if there are 2 observations (left), then (a) for each observation Ihl, test deviation Delta(hl) = (Ihl - ghl Iother) / sqrt[sigIhl**2 + (ghl*sdIother) **2] against sdrej2, where Iother = the other observation (b) if either |Delta(hl)| > sdrej2, then 1. in scaling, reject reflection. Or: 2. in merging, 1. keep both (default or if KEEP subkey given) or 2. reject both (subkey REJECT) or 3. reject larger (subkey LARGER) or 4. reject smaller (subkey SMALLER). (2) if there 3 or more observations left, then (a) for each observation Ihl, 1. calculate weighted mean of all other observations n-1 & its sd(n-1) 2. deviation Delta(hl) = (Ihl - ghl n-1>) / sqrt[sigIhl**2 + (ghl*sd(n-1))**2] 3. find largest deviation max|Delta(hl)| 4. count number of observations for which Delta(hl) .ge. 0 (ngt), & for which Delta(hl) .lt. 0 (nlt) (b) if max|Delta(hl)| > sdrej, then reject one observation, but which one? 1. if ngt == 1 .or. nlt == 1, then one observation is a long way from the others, and this one is rejected 2. else reject the one with the worst deviation max|Delta(hl)| (3) iterate from beginning ^ ________________________________________________________ RELEASE NOTES 0.2.17 fix bug in setting same resolution bin widths for multiple datasets when NBINS is set 0.2.16 message for std::bad_alloc, running out of memory 0.2.15 fix to XML graphs (for ccp4i2) 0.2.14 fix to correctly append to MTZ history 0.2.13 activate writing spacegroup confidence. Reflection status flags cleared before outlier checks 0.2.12 fix bug in reading multilattice files 0.2.10 small bug fix in radiation damage analysis 0.2.9 fix for Batch scaling if no phi range information 0.2.8 XML changes for I2 report. Change automatic anomalous thresholds, always output anom statistics 0.2.7 Bug fix in XML if no orientation data (ROGUEPLOT) 0.2.6 Fix to output multilattice overlaps. Added radiation damage analysis as in CHEF, for Graeme Winter 0.2.5 Fix so that BINS RESOLUTION works 0.2.4 Bug for XDS data, was omitting reflections with FRACTIONCALC (derived from IPEAK) < 0.95, leading to incompleteness 0.2.3 Now does reject and record Emax outliers properly (though work is continuing on improving this). Fixed small bug in analyseoverlaps. 0.2.2 fixed bug in Bdecay plot when batches omitted. Explicit Xrange for XML batch plots. No ROGUEPLOT if no orientation data. List overlaps in ROGUES file 0.2.1 some major reorganisations. Added XML output. SCALES TILE option. Handling of multilattice data. SDCORRECTION SIMILAR 0.1.30 allow TIE with negative sd to turn off tie, as documented. Also fixed bug in ABSORPTION 0.1.29 small change to Result table to work with Baubles arcane (and undocumented) rules for Magic Tables 0.1.28 bug fix in "sdcorrection same" 0.1.27 bug fix in minimizer which sometimes affected the case with just 2 parameters 0.1.26 Default to "scales secondary" 0.1.25 omit sigI<=0, process REJECT command properly, small bug fix in smoothed Bfactors 0.1.24 small bug fix in printing batch tables with multiple datasets 0.1.22,23 INITIAL UNITY option. In tables, print batches with no observations but not rejected batches. Put title into output file. Fix initial scale bug with 3 scales 0.1.21 corrections to ROGUEPLOT, ice rings were in wrong place (by a factor of wavelength) 0.1.20 made sdcorrection refinement more robust to low multiplicity. If anomalous off (or no anomalous detected), statistics are now printed over all I+ I- together. Reject large negative observations (default E < -5) 0.1.19 preliminary addition of spg_confience(|| status). Bug fix from valgrind (from Marcin) 0.1.18 changed tablegraph to fix compilation problem (va_start) 0.1.17 bug fix in outlier rejection, problem with large variances leading to inconsistencies in Rogues file and some over-rejection 0.1.16 made SDcorrection refinement more robust 0.1.14,15 various bug fixes (including memory leaks), fixed autorun generation, improved SD correction for large anomalous, constrain cell to lattice group, etc 0.1.12 Half-dataset CC labelled as "CC(1/2)" 0.1.11 Small bug fixes 0.1.9 autodetect anomalous. Plot Rmeas for each run 0.1.7 fix for SCALES CONSTANT from XSCALE 0.1.6 anisotropy analysis against planes in trigonal, hexagonal and tetragonal systems (inlcuding rhombohedral axes), principal anisotropic axes in monoclinic and triclinic, cone analyses weighted according to cos(AngleFromPrincipalDirection). Fixed cases where multiple datasets have different resolution limits 0.1.4,5 more fixes for multiple datasets, dump/restore. OUTPUT UNMERGED SPLIT is default 0.1.3 More "resolution run"bug fixes 0.1.2 REFINE PARALLEL option (thanks to Ronan Keegan). Fixed bug in "resolution run" options 0.1.1 fixed bugs in writing ROGUES file; introduced HKLOUTUNMERGED etc filename specifiers; cleaned up Unmerged output; added Rfull to tables 0.1.0 fixed some bugs found by cppcheck and valgrind 0.0.16 fixed small bug in INTENSITIES COMBINE optimisation 0.0.15 if run definitions are given explicitly, then unspecified batches are excluded 0.0.14 Added optimisation for INTENSITIES COMBINE, for Mosflm data. This is now the default AUTHOR Phil Evans, MRC Laboratory of Molecular Biology, Cambridge (pre@mrc-lmb.cam.ac.uk) See above for Release Notes. SEE ALSO