ó §™xTc@ sôddlmZddlmZddlmZddlmZddlm Z ddl m Z ddlZddlm Z m Z dd lmZdd lmZdd l mZddlZddlZdd lmZd efd„ƒYZdS(iÿÿÿÿ(tdivision(tmiller(tscaling(tsigmaa_estimation(tflex(tSorryN(tabsolute_scalingt outlier_plots(terf(t table_utils(tnull_out(tStringIOtoutlier_managercB s€eZd d„Zded„Zded„Zdded„Zded d„Zd d „Z d „Z d „Z d d „Z RS(cC s”||_|jdkr'tj|_n|dkrBtƒ|_n||_|jjƒdkrotdƒ‚n|jjƒj |jƒ|_ |j j ƒs±|j j ƒ|_ n|jj ƒsÕ|jjƒ|_n||_tj|j dtƒ}|j jd|jjƒ|jjƒjƒjƒƒ|_tj|jjƒƒdksTt‚|jjƒj |jƒ|_|jjƒj |jƒ|_dS(NtsilentsUnknown observation typet auto_kerneltdatai( touttNonetsyststdoutR t miller_obstobservation_typeRt deep_copytset_observation_typetwork_obstis_xray_intensity_arrayt f_as_f_sqtis_xray_amplitude_arrayt f_sq_as_ft r_free_flagsRtkernel_normalisationtTruetcustomized_copytnormalised_millerRtepsilonst as_doublet norma_workRtmintAssertionErrortselect_centrict centric_worktselect_acentrict acentric_work(tselfRRRt normalizer((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pyt__init__s2      "$gíµ ÷Æ°>c C sqddtj|jjƒ ƒ}dttj|jjƒdƒƒ}tj||kƒ}tj||kƒ}|jj d|jj ƒj |jj ƒƒd|j |ƒƒ}|jj d|jj ƒj |jj ƒƒd|j |ƒƒ}|j |j ƒ}|j |j ƒ}d|} |j| ||ƒ} |jJ|j| IJ|jJ|sW|S|j j|jƒƒSdS(Ngð?g@tindicesRs Outlier rejection based on basic Wilson statistics. -------------------------------------------------- See Read, Acta Cryst. (1999). D55, 1759-1764. for details. Reflections whose normalized intensity have an associated p-value lower than %s are flagged as possible outliers. (RtexpR*RRtsqrtR(tboolRR R.t concatenatet common_setRtmake_log_wilsonRtselect( R+tp_basic_wilsont return_datatp_acentric_singletp_centric_singletacentric_selectiontcentric_selectiont all_flagst all_p_valuest log_string((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pytbasic_wilson_outliers=s,!&    gš™™™™™¹?c C s¿|jjƒjƒ}|jjƒjƒ}dtjdtj|jjƒ ƒt|ƒƒ}dtjttj |jjƒdƒƒt|ƒƒ}tj ||kƒ}tj ||kƒ}|j j d|jj ƒj|jj ƒƒd|j|ƒƒ} |j j d|jj ƒj|jj ƒƒd|j|ƒƒ} | j|jƒ} | j|jƒ} d|} |j| | | ƒ} |jJ|j| IJ|jJ|s¥| S|jj| jƒƒSdS(Ngð?g@R.RsF Outlier rejection based on extreme value Wilson statistics. ----------------------------------------------------------- Reflections whose normalized intensity have an associated p-value lower than %s are flagged as possible outliers. The p-value is obtained using extreme value distributions of the Wilson distribution. (R*RtsizeR(RtpowR/tfloatRR0R1RR R.R2R3RR4RR5( R+tp_extreme_wilsonR7t n_acentrict n_centrictextreme_acentrictextreme_centricR:R;R<R=R>((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pytextreme_wilson_outliersls405    g{®Gáz„?g$@cC s0|dkst‚tjd|ƒ }tj|jjƒ|kƒ}||tjd}tj|jjƒ|kƒ}|jjƒjƒ}tj|jj ƒjƒ|kƒ} g} x—t ||| |ƒD]€\} } } }| r?|r| r| j t ƒq| j t ƒn|sL| r,| j t ƒq<| j t ƒqLqÌ| j t ƒqÌWtj| ƒ} |jjd| ƒ}d|||f}|jƒ}|jjƒ|_|j|jƒ}|jƒj|jjƒƒsÛt‚|jj|jƒƒj|jƒ}|j||ƒ}|j|IJ|s(|S|SdS(Ng333333Ó?gð?g@RsZ Possible outliers due to beamstop shadow ---------------------------------------- Reflection with normalized intensities lower than %4.3e (acentric) or %4.3e (centric) and an associated d-spacing lower than %3.1f are considered potential outliers. The rationale is that these reflection could potentially be in the shadow of the beamstop. (R&tmathtlogRR1R$Rtpit centric_flagst d_spacingstziptappendtFalseRR t map_to_asuRR3R.tall_eqR5Rtmake_log_beam_stopR(R+tleveltd_minR7tz_lim_act z_select_actz_lim_ct z_select_ctcentrictd_selectt tmp_finaltzactzctdstcftfinal_selectiont log_messageR((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pytbeamstop_shadow_outlierssP       $  cC s¹|jdk st‚|jjƒjtƒdkrX|jjd|jjƒƒ|_ntjd|j d|d|jdddd d d ƒ}|j d |j ƒ|j ƒ}t |jƒ}|j}tjd |jƒddd|jƒdtj|jƒjƒdƒd|jƒjƒd|jƒdd|jƒ|jƒƒ} | jƒ} | jƒ} | jƒ} | jƒ} | jƒ}d}tj||kƒ}|}|j||ƒ}ttj|dƒƒ}dtj|t |jƒƒƒ}tj||kƒ}|j j!d|ƒ}|j j!d|ƒ}|j j!d|ƒ}d}|j"ƒ|j"ƒ|j"ƒ|j#ƒj$|j j#ƒƒs‹t‚|j#ƒj$|j j#ƒƒs¯t‚|j#ƒj$|j j#ƒƒsÓt‚|j%||||||||ƒ}t&ƒ}||IJ|J|dIJ|dIJ|J|dIJtj'|jƒdƒ}|j d|ddƒ|j |j(ƒIJ|s{|S|j#ƒj$|j j#ƒƒsŸt‚|j j)|jƒƒSdS(NiRRt miller_calcRtkernel_width_free_reflectionsiÈtn_sampling_pointsitn_chebyshev_termsi Rtf_obst sigma_obstf_calctepsilongð?RZtalphatbetag»½×Ùß|Û=g@s Model based outlier rejection. ------------------------------ Calculated amplitudes and estimated values of alpha and beta are used to compute the log-likelihood of the observed amplitude. The method is inspired by Read, Acta Cryst. (1999). D55, 1759-1764. Outliers are rejected on the basis of the assumption that a scaled log likelihood differnce 2(log[P(Fobs)]-log[P(Fmode)])/Q" is distributed according to a Chi-square distribution (Q" is equal to the second derivative of the log likelihood function of the mode of the distribution). The outlier threshold of the p-value relates to the p-value of the extreme value distribution of the chi-square distribution. s9The histoghram of scaled (LL-gain) values is shown below.sA Note: scaled (LL-gain) is approximately Chi-square distributed.s scaled(LL-gain) Frequencyitftformat_cutoffss%7.3f(*RRR&RtcountRtarrayRtsigmaa_estimatorRtshowRtsigmaatabstnormalized_obstnormalized_calcRtlikelihood_ratio_outlier_testRtdoubleR@RLtposterior_modetlog_likelihoodtposterior_mode_log_likelihoodtposterior_mode_snd_dertstandardized_likelihoodR1t set_selectedRR0RARBR RQR.RRtmake_log_modelR t histogramtgetvalueR5(R+tf_modelRTR7tplot_outRrRttobs_normt calc_normtf_model_outlier_objecttmodestliktp_likts_dertll_gaintepstzerostp_valuestflagsRbttmp_logthisto((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pytmodel_based_outliersôs†                "   $$$       $cC s\|jjd|jjƒ|ƒj|jƒ|_|dk rX|jjd|ƒ|_ndS(NRRU(RR RRRtresolution_filter(R+t scale_factorRU((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pytapply_scale_to_original_dataas   c C ssd}|j|jƒ}|j|jƒ}|jj|jƒƒ}|j|jƒƒ}g}d}x°t|jƒ|jƒ|jƒjƒ|jƒƒD]{\} } } } | dkrét| ƒdtj | ƒt| ƒd| g} nt| ƒdt| ƒd g} |j | ƒq™Wt |ƒdkrat j |g|d dd td td dddƒ}n|d|}|S(sx produces a 'nice' table of outliers and their reason for being an outlier using basic or extreme wilson statistics tIndextE_obstCentricsp-valuesNo outliers were found.is%5.3fs%5.3es inftcommentst has_headert separate_rowstprefixs| tpostfixs |s (R—R˜R™sp-values0.000N(R3R$R5RRNR.RLtstrRIR0ROtlenR tformatRRRP(R+RbRRtheadertp_valstroguestrowsttablethkltetctptthis_rowtfinal((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pyR4hs0  2 c C s™|jjƒ|_|jjƒ|_|jƒ}|jj|jƒƒ}|jj|jƒƒ}d}d}g}|jƒjƒdkrƒ|jƒjƒdkrzxœt|jƒ|jƒjƒ|jƒ|jƒ|j ƒjƒƒD]X\}} } } } t |ƒd | d | d t j | ƒt | ƒg} |j | ƒqçWtj|g|d dd td tddddƒ}qƒd}n|d|d}|S(NR—s d-spacingtF_obssE-valueRZsNo outliers were foundiiôs%4.2fs%6.1fRšR›RœRs| Ržs |s"Over 500 outliers have been found.s s (sIndexs d-spacingR­sE-valuescentric(R$RQRR5RR@RNR.RMRLRŸRIR0ROR R¡RRRP(R+RbRRtevalsR¢R¦R¥R§tdtfobsR¨R©R«R¬((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pyRS‰s<   (   c C s d$} d } g} |j|jƒƒ} |j|jƒƒ} |j|jƒƒ}|j|jƒƒ}|j|jƒƒ}|jjƒj|jƒƒ}| jƒjƒd krŽ| jƒjƒd kr…| jƒ}| jƒdkr | jƒjƒd d }nxAt | jƒ| jƒjƒ| jƒ|jƒ|jƒ| jƒ|jƒ|jƒ|| j ƒjƒj ƒƒ D]Ð\ }}}}}}}}}}t |ƒd |d|d|d|d|d|t |ƒg}| j |ƒ|dk rztjd|d|d|d|dd||d|d|d|dt |ƒƒ qzqzWtj| g| dddtdtddd d!ƒ} qŽd"} n|d#| }|S(%NR—s d-spacingR˜tE_modeltScoresp-valueRtRZsNo outliers were foundiiôg$@s%4.2fs%6.3fs%5.2fs%5.3es%4.3fR°tsigmatfcalcRlRmgð?RkRt plot_titleRšR›RœRs| Ržs |sQMore then 500 outliers were found. This is very suspicious. Check data or limits.s (sIndexs d-spacingsE_obsR±R²sp-valuessigmaascentric(R5RRRLR.R@tsigmasRRMRNR"R#RŸRORtplotitR R¡RRP(R+RbRRŒRte_obste_calcRtR„R¢R¦R¥R¤tp_arraytll_arraytec_arraytsa_arrayRLR¶R§R¯teotectllgRªtsaR©tsR¨R«R¬((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pyR€¶sl         =      N( t__name__t __module__RR-RPR?RHRcR“R–R4RSR€(((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pyR s * . 0 V i  ! 5(t __future__RtcctbxRtmmtbxRt mmtbx.scalingRtcctbx.array_familyRt libtbx.utilsRRRt scitbx.mathRtlibtbxR R RRItlibtbx.str_utilsR tobjectR (((sM/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/scaling/outlier_rejection.pyts