xTc@ sdZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl m Z dd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlZdejfdYZdefdYZdejfdYZdejfdYZdejfdYZdejfdYZ dejfdYZ!dejfdYZ"d ejfd!YZ#d"ejfd#YZ$dS($s Collect multiple analyses of experimental data quality, including signal-to-noise ratio, completeness, ice rings and other suspicious outliers, anomalous measurability, and Wilson plot. i(tdivision(tabsolute_scaling(tscaling(t data_plots(tmiller(tadptbx(tflex(tchebyshev_polynome(tchebyshev_lsq_fit(terf(t approx_equal(tSorry(t table_utilsNti_sigi_completeness_statscB s,eZdZdddddZdZRS(s{ Collects resolution-dependent statistics on I/sigma expressed as percentage of reflections above specified cutoffs. ig@g333333?g @c sjjt|_jrQjrQt|_jqQnjsct j dk s{t j d|t jj|_jdt|_jdtjdd!|_g|_fd}g}ddddd d g}x$|D]} |jj|| qWd d d ddddg} tjdddddg| ddddgdgdddgdd'gd t tdd!gd"td#t|_xtdt|jdD]i} |j| d} |j| } | | g}x&|jD]}|jd$|| qW|jj|qW||_||_||_ d%|_!||}d%}xqtdt|jdD]S} |j| d} |j| } | |kr|| |kr| d&|_!qqqWdS((Ntn_binst use_binningiic sQjj|jk}|j|jdtdd}|jS(NRt return_failg(tselecttdatatsigmastuse_binning_oft completenesstTrue(t cut_valuet tmp_millerR(t miller_array(sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytcut_completeness4s   iiii is Res. Ranges I/sigI>1 s I/sigI>2 s I/sigI>3 s I/sigI>5 s I/sigI>10s I/sigI>15ttitlesCompleteness and data strengtht column_labelssLow res.s High res.tcolumn_formatss%4.2fs%5.1fit graph_namessI/sigI by shellt graph_labelssHigh resolution of shells % of totalt graph_columnsitx_is_inverse_d_mint first_two_columns_are_resolutiongY@g@g(sHigh resolution of shells % of total("t deep_copytset_observation_typetFalsetamplitudes_flagt is_real_arraytis_xray_amplitude_arrayRt f_as_f_sqtis_xray_intensity_arraytAssertionErrorRtNonet setup_binnertlisttbinnertlimitstresolution_binsRtoverallRtoverall_binnedtcompleteness_binstappendRt table_datatrangettabletxrangetlentadd_rowt isigi_cuttcompleteness_cuttresolution_at_leasttresolution_cut(tselfRRR<R=R>RR6tcut_listRtlegendtiitatbtrowtjjt comp_datatreso((RsK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyt__init__s`           #       #  cC s|jd|jd|jr3|jdn|j|jdd|j|jd|j|jd|j f}|j |j kr|d7}n|d |j 7}|j|dS( NsCompleteness at I/sigma cutoffssg The following table lists the completeness in various resolution ranges, after applying a I/sigI cut. Miller indices for which individual I/sigI values are larger than the value specified in the top row of the table, are retained, while other intensities are discarded. The resulting completeness profiles are an indication of the strength of the data. sPlease be aware that the input data were given as amplitudes and squared for the purposes of this analysis, therefore the numbers displayed here are less reliable than the values calculated from the raw intensities. tindentisn The completeness of data for which I/sig(I)>%.2f, exceeds %.0f %% for resolution ranges lower than %.2fA. ids_ The data are cut at this resolution for the potential twin tests and intensity statistics. s As we do not want to throw away too much data, the resolution for analyzing the intensity statistics will be limited to %3.2fA. ( tshow_sub_headert show_textR&twarnt show_tableR8t show_plotR<R=R?R>(R@touttmessage((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyt _show_impl_s     (t__name__t __module__t__doc__RJRS(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyR s  =tcompleteness_enforcementcB seZddZRS(g?cC s|j|_|jjdt|jjdtdd}tj|jjjt }xW|j j D]F}|j j |j }|j||krn|j|t}qnqnW|j||_dS(Nt auto_binningRRg?(R#Rtsetup_binner_d_star_sq_stepRRRtbooltindicestsizeR%R/t range_allt selectiont iselectionRt set_selectedRt new_miller(R@Rtminimum_completenessRtselection_arraytbinR^((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJs !(RTRURJ(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRWstanalyze_resolution_limitscB s8eZdZddZdZddZdZRS(sN Check for elliptical truncation, which may be applied to the data by some processing software (or as a post-processing step). As a general rule this is not recommended since phenix.refine and related programs will handle anisotropy automatically, and users tend to apply it blindly (and even deposit the modified data). g?cC s[|j}|jjddkrtj}x~|jD]p\}}}|dkrt|dkrt|dks|dkrA|dkrA|dkrA|j|||fqAqAWt|tj r|j d|}q|j d|}n |j}|j |_ t|j |j dds0t|j\|_|_|_||_dS(NiiZiR[tepsg?(iZiZiZ(t expand_to_p1t unit_cellt parametersRt miller_indexR[R5t isinstanceRtarraytsettcustomized_copytd_mint d_min_overallR R+td_min_along_a_b_c_startd_min_atd_min_btd_min_ctd_min_max_delta(R@t miller_setRuRtall_pos_neg_indicesthtktl((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJs  H $cC st|j|j|jg}d}xOtdD]A}x8t|dD]'}t|t||||}qAWq+W|S(sD Return the maximum difference in d_min along any two axes. iii(RrRsRtR7tmaxtabs(R@R0t max_deltatitj((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytmax_d_min_deltas )cC s(|dkr|j}n|j|kS(N(R,RuR(R@Ru((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytis_elliptically_truncateds  cC s|jd|jd|j}|jd|j|j|j|j|f||jkrq|jdn |jddS(NsAnalysis of resolution limitssYour data have been examined to determine the resolution limits of the data along the reciprocal space axes (a*, b*, and c*). These are expected to vary slightly depending on unit cell parameters and overall resolution, but should never be significantly different for complete data. (This is distinct from the amount of anisotropy present in the data, which changes the effective resolution but does not actually exclude reflections.)s overall d_min = %.3f d_min along a* = %.3f d_min along b* = %.3f d_min along c* = %.3f max. difference between axes = %.3f sThe resolution limit appears to be direction-dependent, which may indicate issues with the data collection geometry, processing errors, or that elliptical truncation has been applied. We do not recommend using elliptically truncated data, as anisotropy is handled automatically by Phaser, phenix.refine, and related programs, and discarding large numbers of weak reflections may result in increased map bias and/or artifacts. You should always deposit the original, uncorrected reflections in the PDB, not the truncated data.s1Resolution limits are within expected tolerances.( RLRMRtshow_preformatted_textRpRrRsRtRu(R@RQR}((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSs   N(RTRURVRJRR,RRS(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRes   tlog_binned_completenesscB s)eZdZddddZdZRS(s? Table of completeness using log-scale resolution binning. idii2cC sg}|jd|d|d|}x|D]}|j|}|j\} } |j} |jd| j} |jd| } |jd| | fd| | fd| dfq+Wtjd d d d gd ||_ dS(Nt&n_reflections_in_lowest_resolution_bintmax_number_of_binstmin_reflections_in_bintd_maxs %.4f - %.4fs%d/%ds%.1f%%idtcolumn_headerst Resolutiont Reflectionst Completenesst table_rows( t log_binningRt d_max_minR\t complete_setRR5R t simple_tableR8(R@RRRRtrowstbinsR^t bin_arrayRRotn_reflt n_refl_expectR((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJs      cC s.|jd|jd|j|jdS(NsCompleteness (log-binning)sThe table below presents an alternative overview of data completeness, using the entire resolution range but on a logarithmic scale. This is more sensitive to missing low-resolution data (and is complementary to the separate table showing low-resolution completeness only).(RLRMROR8(R@RQ((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSs (RTRURVRJRS(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRs  tpossible_outlierscB sJeZdZdddZdZdZdZdZdZRS( su Flag specific reflections with suspicious intensities. Inspired by: Read, Acta Cryst. (1999). D55, 1759-1764 g?gư>cC s|jdkr!tdn|js<|j}n |j}|jsZt|j}tj|dt }|j d|j j |j j j}|j|_|jj|}|jj|}|j jdstddtj|j }dttj|j d} | j} |j} dtjdtj|j t| } dtjttj|j dt| } | |kp||k}| |kp| |k}|jj||_|j j||_tj|j|_|jj j||_ |j||_!| j||_"t#j$ddd d d d d dgddddddgddgddddddgg|_%x]t&|j |j|j|j!|j"D]4\}}}}}|j%j'|||||gqW|jj||_(|j j||_)tj|j)|_)|jj j||_*| j||_+| j||_,t#j$ddd d d d d dgddddddgddgddddddgg|_-x]t&|j*|j(|j)|j+|j,D]4\}}}}}|j-j'|||||gqDWdS(NsUnknown observation typet auto_kernelRig?g@RsAcentric reflectionsRt d_spacingsH K Ls|E|s p(wilson)s p(extreme)Rs%8.3fs %5i,%5i,%5is%6.2fs%9.2es%10.2eRsPossible acentric outliersR iiiisCentric reflectionssPossible centric outliers(.tobservation_typeR,R R*R)R#R+Rtkernel_normalisationRRltnormalised_millerRtepsilonst as_doubleR\Rtselect_centricR$tselect_acentrictall_geRtexpR tsqrttpowtfloatR[Rtacentric_outlier_millertacentric_outlier_e_valst d_spacingstacentric_outlier_d_spacingstacentric_outlier_p_valtacentric_outlier_extreme_valRR6tacentric_outliers_tabletzipR;tcentric_outlier_millertcentric_outlier_e_valstcentric_outlier_d_spacingstcentric_outlier_p_valtcentric_outlier_extreme_valtcentric_outliers_table(R@Rt prob_cut_ext prob_cut_wilt work_arrayt normalizert centric_cutt acentric_cuttp_acentric_singletp_centric_singlet n_centrict n_acentrictextreme_acentrictextreme_centrictacentric_outliertcentric_outliertdthkltetptextr((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJs       #  -2             #        cC s|j|jS(N(t n_outliersR(R@((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytfraction_outliersescC s&|jj}|jj}||S(N(RR\R(R@RR((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRhscC sjtj|j|j}|j|jd}tj|j|j}|j|jd}|S(Ni(Rt match_indicesR[RRtsingle_selectionR(R@Rtcentric_matchestacentric_matches((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytremove_outliersms  cC s|jd|jd|jd|jjdkrL|jdn0|j|jdd|jd|jd |jd |jjdkr|jdn0|j|jdd|jd |jd dS( NsPossible outlierss8 Inspired by: Read, Acta Cryst. (1999). D55, 1759-1764 sAcentric reflections:is None RKisP p(wilson) : 1-(1-exp[-|E|^2]) p(extreme) : 1-(1-exp[-|E|^2])^(n_acentrics) si p(wilson) is the probability that an E-value of the specified value would be observed if it were selected at random the given data set. p(extreme) is the probability that the largest |E| value is larger or equal than the observed largest |E| value. Both measures can be used for outlier detection. p(extreme) takes into account the size of the dataset. sCentric reflections:sV p(wilson) : 1-(erf[|E|/sqrt(2)]) p(extreme) : 1-(erf[|E|/sqrt(2)])^(n_acentrics) sr p(wilson) is the probability that an E-value of the specified value would be observed when it would selected at random from the given data set. p(extreme) is the probability that the largest |E| value is larger or equal than the observed largest |E| value. Both measures can be used for outlier detection. p(extreme) takes into account the size of the dataset. ( RLtshowRR\RORRRR(R@RQ((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSws$     cC s$|jdkrdgSdgSdS( NgMbP?is1There are a large number of outliers in the data.sPossible outliersis7The fraction of outliers in the data is less than 0.1%.(is1There are a large number of outliers in the data.sPossible outliers(is7The fraction of outliers in the data is less than 0.1%.sPossible outliers(R(R@((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytsummarize_issuess ( RTRURVRJRRRRSR(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyR s R   *tice_ring_checkercB s2eZdZddddZdZdZRS(s Check intensity and completeness statistics in specific resolution ranges known to have strong diffraction when crystalline ice is present. g@g?i c C s||_||_||_tjddddddddd d g |_tjd d d dddddddg |_ddddddddddg |_d|_ d|_ |d}||j d}|d|d} d} tj|j d} x|jD]} d | d} | |}|| d| }t |dd}|dkred}n||j dkrd}n||j| <| d7} |dk r d| |<|ddkrd| |ds%9.3fs%10.3fs%7.2fRRRsExpected rel. Is Data Z-scoreR(tcompleteness_abnormality_leveltintensity_levelt z_score_limitRtdoubletice_d_spacingstice_rel_intensR,tice_ring_bin_locationt mean_compt mean_z_scoreR\tinttsumtmathRtstd_compt std_z_scoretabnormality_intensitytvalue_intensitytabnormality_completenesstvalue_completenessR7twarningsR5R|R RR8(R@t bin_centerstcompleteness_datat z_scores_dataRRRttmp_low_d_star_sqttmp_high_d_star_sqttmp_steptcounttweightstice_ringt tmp_ice_ringttmp_ice_ring_binRRRRRCt icy_shellstcutoffR((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJs                   "    cC s"|jd|jd|jd|j|j|j|jf|jd|j|jdd|jd|j t }|j }x7t dD])}|j |dk r|j||j krt|j|}||kr;t}|jd |j|t|j||kr+|jd q;|jd nt|j||ksyt|j|t|d dkrt}t|j||kr|jd |j||fqqqqW|s|jdn|jdkr|jdn&|jdkr|jd|jndS(NsIce ring related problemssV The following statistics were obtained from ice-ring insensitive resolution ranges: s mean bin z_score : %4.2f ( rms deviation : %4.2f ) mean bin completeness : %4.2f ( rms deviation : %4.2f ) sr The following table shows the Wilson plot Z-scores and completeness for observed data in ice-ring sensitive areas. The expected relative intensity is the theoretical intensity of crystalline ice at the given resolution. Large z-scores and high completeness in these resolution ranges might be a reason to re-assess your data processsing if ice rings were present. RKis Abnormalities in mean intensity or completeness at resolution ranges with a relative ice ring intensity lower than %3.2f will be ignored.i sU At %3.2f A there is an lower occupancy than expected from the rest of the data set.sY At the same resolution range, the expected mean intensity does not behave as it should.so Even though the completeness is lower as expected, the mean intensity is still reasonable at this resolution.Rs At %3.2f A the z-score is more than %3.2f times the standard deviation of all z-scores, while at the same time, the occupancy does not go down. s No ice ring related problems detected. If ice rings were present, the data does not look worse at ice ring related d_spacings as compared to the rest of the data set.is| As there was only 1 ice-ring related warning, it is not clear whether or not ice ring related features are really present.sZ There were %d ice ring related warnings. This could indicate the presence of ice rings.(RLRRRRRRROR8RR%RR7RR,RR|RRRRRtgetattrR(R@RQtproblems_detectedRRCR((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSsR #     %%  cC s!|jdkrdgSdgSdS(Niis.The data appear to have one or more ice rings.sIce ring related problemss&Ice rings do not appear to be present.(is.The data appear to have one or more ice rings.sIce ring related problems(is&Ice rings do not appear to be present.sIce ring related problems(R(R@((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyR[s (RTRURVRJRSR(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRs  c Btanalyze_measurabilitycB s#eZddddZdZRS(gQ?gQ?c C s||k}||k}|j|} |j|} d|_d|_ddgg} | jdkrtj| d|_tj| | dd 3.0 min[I(+)/sigma_I(+), I(-)/sigma_I(-)] > 3.0 s The measurability provides an intuitive feeling of the quality of the data, as it is related to the number of reliable Bijvoet differences. When the data are processed properly and the standard deviations have been estimated accurately, values larger than 0.05 are encouraging. Note that this analysis relies on the correctness of the estimated standard deviations of the intensities. ( RLR,RRRRRRORRPR8(R@RQRRtunknown_cutoffs((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSsF          N(RTRUR,RJRS(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRes tdata_strength_and_completenesscB sAeZdZdddZdZdZdZdZRS(si Collect basic info about overall completeness and signal-to-noise ratios, independent of scaling. g@g333333?cC s|j}|jdt|jjd|_d|_d|_|j dk r|j dtdd}|j dt |j d!|_t j|j|_tjddd d d gd d gdd!gdddggd|j|jgdt|_nd|_|j dk r;t|d|d||_n|j|_|jdd}d|_d|_d|_|jjdkr|jdd|jdt|_|jdt}|jdd|_|j}g} g} xk|jD]]} |j| } | j d| d|j | } | dkrId} n| j | dqWtjddd ddgd dgdd"gdddggd| | gdtd t|_nyt!||_"Wnt#k rd|_"nXt$||_%dS(#NRXiRRiiRsSignal to noise ()Rs 1/resol**2s RsSignal to noiseRRR RR!R<R=Rog@Ri RRidsLow-resolution completenesssMax. resolutionsHigh resolution of shells % of totaltforce_exact_x_labels(s Resolutions (sHigh resolution of shells % of total(&R#RYRR/Rt d_star_sq_oriR,ti_sig_it i_sig_i_tableRt i_over_sig_iRR:RRRR6t data_strengthR Rtcompleteness_overalltresolution_filtertlow_resolution_completenesst#low_resolution_completeness_overallt low_res_tableR[R\R-R%Rt range_usedt bin_d_minR5Retd_min_directionalR+Rt log_binned(R@RR<R=Rt i_over_sigmattmp_miller_lowresRR/Rtcompti_binRotfrac((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJsv                          cC s|jd|jdk r/|jj|n|jdk rq|jd|jd|j|jddn|jj||jdk r|jj|ndS(NsData strength and completenesss$Low resolution completeness analysessbThe following table shows the completeness of the data to 5.0 A. Poor low-resolution completeness often leads to map distortions and other difficulties, and is typically caused by problems with the crystal orientation during data collection, overexposure of frames, interference with the beamstop, or omission of reflections by data-processing software.RKi( t show_headerR R,RRRLRORR(R@RQ((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSs  cC sC|jdkrdS|jj|jjkr5|jjS|jjSdS(N(R R,R?R>(R@((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pythigh_resolution_for_twin_tests0s   cC s|jdk r|jdSdS(Ni(R R,(R@((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyti_over_sigma_outer_shell9s cC sg}|jjr%|jdn |jdd|jkoLdknrn|jddd d fnVd|jkod knr|jddd d fn|jddd d f|jdkr|jdn,|jd kr|jdn |jd|S(Nis4The data appear to have been elliptically truncated.sAnalysis of resolution limitsis>The resolution cutoff appears to be similar in all directions.g?is2The overall completeness in low-resolution shells sis less than 90%.s$Low resolution completeness analysesg?sis at least 90%.s&Overall completeness is less than 75%.s&Overall completeness is less than 90%.s"Overall completeness is above 90%.(is4The data appear to have been elliptically truncated.sAnalysis of resolution limits(is>The resolution cutoff appears to be similar in all directions.sAnalysis of resolution limits(is&Overall completeness is less than 75%.N(is&Overall completeness is less than 90%.N(is"Overall completeness is above 90%.N(RRR5RR R,(R@tissues((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyR>s*     (RTRURVRJRSRRR(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRs D  t anomalouscB seZddZdZRS(c C sy|jst|j}|jdt|jjd|_d|_ |dk rj|j |_ nd|_ |j dk ru|j dtdd}t j |jdt|jd!}tjt|jjdd|j|}tt|jjddt j|jt j|j|j}|j|j}td |jd |d |d ||_ ndS( NRXiRRiii iRRRR(tanomalous_flagR+R#RYRR/RRR,t cc_anom_tableRRRRRR:RRR\RtminR{tcoefstfR(R@Rt merging_statsRRRtsmooth_meas_approxt meas_smooth((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJYs8        cC s|jd|jdk re|jd|jd|jrR|j|jqe|j|jn|jdk r|jj |ndS(NsAnomalous signals"Half-dataset anomalous correlationse This statistic (also called CC_anom) is calculated from unmerged data, which have been split into two half-datasets of approximately the same size and merged to obtain unique (anomalous) intensities, and their anomalous differences compared. Resolution shells with CC_anom above 0.3 contain substantial anomalous signal useful for heavy atom location.( RR!R,RLRMt gui_outputRPRORR(R@RQ((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSys   N(RTRUR,RJRS(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRXs twilson_scalingcB sJeZdZdd ddddZdZdZdZd ZRS( s Calculates isotropic and anisotropic scale factors, Wilson plot, and various derived analyses such as ice rings and outliers. t eigen_miniig@c%C s||dkr|dks"t|j}|j} | jdt| jjd|_|dkrwd}n|dkrd}n||dkrt dn|j j } t j d|d|| |d|| |} | |_| j|_| j|_|jj} |j j } t jd|d|| |d|| |} | j}| |_| j|_| j|_| j|_t| dd|_| j}|d|d |dd }|d}|d |kr|d }n|d|kr|d}n|d kr*| jd}nN|d krKtj| j}n-|d krrt dk sit|}nd}| | | dddg}t!j"|j#t!j$|}t j%|d| j|_&t j%|j&d||_&|j&j'||_&|j&j(dk}|j&j)d|j&j(j*|d|_&|j&j'||_&t j+|j&dt}|j,j|_,|j-dt.j/d@dt.j/dA}t j%d|d|jd|j}|j'||jdt|jjd}dtj0|j1j(tj0|j1j(||j2d <|j3dtdd}|j4dtdd}| j5|t j+|dt}|j,j}|jdt|j} | jdt| jjd|_||_6tj7|j8j9|j6|_:tj;|j(d t<|j(d !|_=t j>| |}|j?|_@tj;|jjA}|d |j2d !}|j=|j=|d|_B|jB|jC|jC7_Btj/|jB|_BtjD|j=|j@|jB|_Etj;|j(d t<|j(d !|_4tF||_G|jGjH||_Id|_J|j6j2d krtj0|j6dkrtKd|j6d |j4d!|jE|_JntLjMd"d#d$d%d&d'd(gd)d#gd*dBgd-dd dd.ggdtN|j6tN|j:tN|j=tN|j@gd/t|_O|jE|k}|jPt|_QtLjMd"d0d$d1d2d3d4gd5d6d7d7d8gd)d9gd-dd dd.gg|_RxtS|j6j2D]r} || r|j6| d:}!|jE| }"|j4| }#|j=| |j@| }$|jRjT|!|"|#|$gqqWtLjMd"d;d$d%d<d=gd)d>gd*dCgd-dd dggdtN|j6tN|jEtN|j4gd/t|_UdS(DNiRXisNo scatterers availableRt n_residuestn_basestoverall_b_cartig@R*t eigen_meant user_b_isoiggd~QJRRRg?gMb?RogGz?tcache_0tp_scaletu_starRRgư>g{Gz?RRRRsIntensity plotsRs 1/resol**2s smooth approximations via binnings expectedRRRsR iR!s"Mean intensity by shell (outliers)Rtz_scoreRs /Rs%9.3fs%7.2fs%10.3fsWorrisome resolution shellsgsZ scores and completenesstZ_scoresFractional completenesss"Data sanity and completeness checks"Z score or fractional completenessg@_@gt\;0?(s Resolutions(s Resolutions"Z score or fractional completeness(VR+tinfoR#RYRR/RRR,R t space_grouptorder_zRtml_iso_absolute_scalingtiso_scale_and_bR1t iso_p_scaletb_wilsont iso_b_wilsont scat_infotml_aniso_absolute_scalingtb_carttaniso_scale_and_bt aniso_p_scaleR2t aniso_u_start aniso_b_cartRR-t eigen_valuesRtmeantremove_aniso_b_isoRtu_cart_as_u_starRhtb_as_utanisotropic_correctiontno_aniso_arrayR$RRlR`RRRRRR"RR\t$mean_of_intensity_divided_by_epsilonRt scat_dataRRRR$tmean_I_normalisationRR:tmean_I_obs_datatexpected_intensitytmean_intensitytmean_I_obs_theoryt counts_giventmean_I_obs_sigmatsigma_intensityR|tz_scoresRtoutliersRtmiller_array_filteredt ice_ringsRRR6R.t wilson_tableRt n_worrisometoutlier_shell_tableR7R;t zscore_table(%R@RR+tremove_aniso_final_bt use_b_isot n_copies_solcR,t z_score_cutR5RR7R9R=R@R?tb_cart_observedtb_trace_averaget b_trace_mintb_usetb_cart_aniso_removedtu_star_aniso_removedtsel_bigt normalistiontabsolute_millerRtmean_observed_intensityRt normalisationRttheorytcountst worrisomeRCtd_spaceR3Rtratio((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRJs("                             !       +     #     -            #   *cC s`|jd|jd|jdkrO|j|jdd|jdn |jddS(NsMean intensity analysessl Inspired by: Morris et al. (2004). J. Synch. Rad.11, 56-59. The following resolution shells are worrisome:iRKisg Possible reasons for the presence of the reported unexpected low or elevated mean intensity in a given resolution bin are : - missing overloaded or weak reflections - suboptimal data processing - satellite (ice) crystals - NCS - translational pseudo symmetry (detected elsewhere) - outliers (detected elsewhere) - ice rings (detected elsewhere) - other problems Note that the presence of abnormalities in a certain region of reciprocal space might confuse the data validation algorithm throughout a large region of reciprocal space, even though the data are acceptable in those areas. s *** None ***(RLRRZROR[(R@RQ((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pytshow_worrisome_shellsWs  cC s|jd|jjd||jjd||jd|jd|j|j|j||j j||j dk r|j j|ndS(Ns$Absolute scaling and Wilson analysisRQs Wilson plots The Wilson plot shows the falloff in intensity as a function in resolution; this is used to calculate the overall B-factor ("Wilson B-factor") for the data shown above. The expected plot is calculated based on analysis of macromolecule structures in the PDB, and the distinctive appearance is due to the non-random arrangement of atoms in the crystal. Some variation is natural, but major deviations from the expected plot may indicate pathological data (including ice rings, detector problems, or processing errors).( RR9RR@RLRMRPRYRqRVRXR,(R@RQ((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRSqs   cC stg}|jdk r.|j|jjn|j|jj|j|jj|j|jj|S(N(RXR,textendRRVR9R@(R@R((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyRscC s[d|_d|_d|_d|_d|j_d|j_d|j_d|j_|j S(sB Pickling function with storage efficiency optimizations. N( R,RWRJRRR9RR@R=t__dict__(R@((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyt __getstate__s        N( RTRURVR,RJRqRSRRt(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyR)s   (%RVt __future__Rt mmtbx.scalingRtmmtbxRtiotbxRtcctbxRRtcctbx.array_familyRt scitbx.mathRRR tlibtbx.test_utilsR t libtbx.utilsR tlibtbxR Rtxtriage_analysisR tobjectRWReRRRRRRR)(((sK/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/data_statistics.pyts0 kO%n2