ʙxTc @`sdZddlmZmZddlmZddlmZddlm Z ddl Z ddl Z ddl Z ddl mZmZddlmZdd lmZdd lmZddlZ dd lmZdd lmZdd lmZddlmZddl Z!ddl"m#Z#y(ddl$Z%ddl&Z&ddl'Z'Wn#e(k rkdZ&dZ'dZ%nXe j*j+ddde j,j-Z.dZ/ia0ia1ddddddddddddd g Z2dd!Z3d"Z4e5e5e5e5e5e5de5e5e5e6e5d# Z7e5d$d%Z8e6e6e6d&Z9dd'Z:dd(Z;dd)Z<de5e6d*Z=ddd+Z>d,Z?d-Z@d.efd/YZAd0e!jBjCjDfd1YZDdS(2st A set of functions to act upon ChemicalEnvironment and ScatteringEnvironment and produce a single class and vector of features for use with a classifier. This module relies on a SVM classifier generated by the module within phenix_dev.ion_identification.nader_ml. See that module's description for more details. See Also -------- phenix_dev.ion_identification.nader_ml i(tdivisiontabsolute_import(tIterable(tStringIO(tc_doubleN(tAutot+slots_getstate_setstate_default_initializer(tload(tmake_sub_header(tSorry(tsasaki(tcommon_residue_names_water(tN_SUPPORTED_ENVIRONMENTS(tSUPPORTED_GEOMETRY_NAMES(tutilst relative_pathschem_data/classifiersttesttmerged_high_restHOHtMNtZNtFEtNItCAtNAtMGtCLtCUtCOtCDtKcC`s tdk st| s%|tkr.t}n|tkrtjjt dj |}tjjt dj |}ytj |t|>sRVN(R R RRRGtfromiterRX(R=tgeometry_namesRb((RcsE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRK#s    c`s>|dkrtt}ntjfd|DdtS(sa Creates a vector for the identities of the ions surrounding a site. Returns a vector with a count of coordinating nitrogens, oxygens, sulfurs, and chlorines. Parameters ---------- chem_env : mmtbx.ions.environment.ChemicalEnvironment A object containing information about the chemical environment at a site. environments : list of int, optional A list of environments to check for. If unset, takes values from mmtbx.ions.environment.N_SUPPORTED_ENVIRONMENTS. Returns ------- numpy.array of float A vector containing quantitative properties for classification. c3`s|]}j|VqdS(N(t chemistry(RaRb(R=(sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pys XsRVN(R trangeR RGRdRX(R=t environments((R=sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRLAs cC`s|dkr4gtD]}|tkr|^q}ng}x<|D]4}|j|jd|dtjjj|qAWt |S(s  Calculate the BVS and VECSUM values for a variety of ion identities. Parameters ---------- chem_env : mmtbx.ions.environment.ChemicalEnvironment A object containing information about the chemical environment at a site. elements : list of str, optional List of elements to include when calculating BVS and VECSUM values. If unset, takes the list from mmtbx.ions.ALLOWED_IONS. Returns ------- numpy.array of float A vector containing quantitative properties for classification. telementtchargeN( R t ALLOWED_IONSR<R]t get_valencetmmtbxtionstservert get_charget _flatten_list(R=RDRbtretRi((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRM[s ( c`s|dkr4gtD]}|tkr|^q}njdksRjdkr{|rktjt|StjdSn|rjdn j|rtj fd|Dt }nt g}|S(s Creates a vector of the anomalous features of a site. These can either include the f'' / f''_expected for a variety of ion identities or the exact anomalous peak height. Parameters ---------- scatter_env : mmtbx.ions.environment.ScatteringEnvironment An object containing information about the scattering environment at a site. elements : list of str, optional List of elements to include when calculating f''_expected values. If unset, takes the list from mmtbx.ions.ALLOWED_IONS. ratios : bool, optional If False, instead of calculating ratios, just return a vector of the wavelength, f', and f''. anom_peak : bool, optional Whether to use the actual height of the anomalous map instead of the calculated f'' values. Returns ------- numpy.array of float A vector containing quantitative properties for classification. iic3`s4|]*}tj|jjjVqdS(N(R ttablet at_angstromt wavelengthtfdp(RaRi(theightRO(sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pys sN( R RkR<tfppRuRGtzerostlent anom_densityRdRXRq(RORDRERFRbRr((RwROsE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRNys ( cC`st|\}}}}|dks0|dkr4dSt|||}tj||d|d}t|t|kst||}tjt |d|j j tj kd} |j } t| } tjj|| | | | } g|jD]} t| ^q}t|| }|jdd|dk rx2|D]*}|tkrJtdj|qJqJWgg}} x=|D]5\}}||kr|j|| j|qqWt| }g| D]} | |^q} t|| }n|S(s Uses the trained classifier to predict the ions that most likely fit a given list of features about the site. Parameters ---------- chem_env : mmtbx.ions.environment.ChemicalEnvironment A object containing information about the chemical environment at a site. scatter_env : mmtbx.ions.environment.ScatteringEnvironment, optional An object containing information about the scattering environment at a site. elements : list of str, optional A list of elements to include within the prediction. Must be a subset of mmtbx.ions.svm.ALLOWED_IONS. Note: Water is not added to elements by default. svm_name : str, optional The SVM to use for prediction. By default, the SVM trained on heavy atoms and calcium in the presence of anomalous data is used Returns ------- list of tuple of str, float or None A sorted list of classes and the predicted probabilities associated with each or None if the trained classifier cannot be loaded. iitisKerneltkeycS`s |d S(Ni((tx((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pytssUnsupported element '{}'N(R5R RURtscale_toRzR!tsvmtgen_svm_nodearraytlisttparamt kernel_typet PRECOMPUTEDt get_nr_classRtlibsvmtsvm_predict_probabilityt get_labelsRktziptsortR R(R]tsum(R=RORDR.t classifierR2R3R4tvectortxitnr_classtprob_estimatestprobsRbtlabelstlstRitclassestprobttotal((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyt predict_ions< !   #       c`s2fdtjd|DdtS(sE Turn a tree main out of lists into one flat numpy array. Converts all Nones into zeros and integers into floats in the process. Parameters ---------- lst : list or list of list or list of list of list or ... A list to be flattened Returns ------- numpy.array of float A flat list of values. c3`sXxQ|D]I}t|trKt|t rKx!|D] }|Vq9Wq|VqWdS(sO Returns a generator for each element in the flattened version of lst. N(t isinstanceRt basestring(Rtitemtsub(t_flatten(sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRs   cs`s-|]#}|dk r!t|ndVqdS(gN(R RX(RaRb((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pys sRV(RGRdRX(R((RsE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRqs sn svm .expert_level = 3 { svm_name = Auto .type = str .help = "Name of SVM classifier to use" filtered_outputs = True .type = bool .help = "Apply an filter to the outputs of the SVM, ensuring they are chemically sane" min_score = 0.2 .type = float min_score_above = 0.1 .type = float min_fraction_of_next = 2.0 .type = float } tsvm_predictioncB`sMeZdZdddddddgZejdd Zejdd ZRS( s\ Contains information about a SVM's prediction of a site's identity. Attributes ---------- i_seq : int pdb_id_str : str atom_info_str : str map_stats : group_args atom_types : list of str scores : list of float Probabilities associated with each element listed by atom_types. final_choice : mmtbx.ions.metal_parameters ti_seqt pdb_id_strt atom_info_strt map_statst atom_typestscorest final_choicetcC`sx+|jjD]}|||jIJqW||dIJtt|j|jdd}x,|D]$\}}||d||fIJqgW|jdk r||d|jIJndS(s Shows information about a SVM's prediction of a site's identity. Parameters ---------- out : file, optional prefix : str, optional s SVM scores:R}cS`s |d S(Ni((R~((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRIss %4s : %.3fsFinal choice: %sN( Rt splitlinestrstriptsortedRRRRR (tselftouttprefixtlineRtelemtscore((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pytshow=s $c C`s|j}|dkr$d}d}nCx@t|j|jD])\}}||jkr:d|}Pq:q:W||d|j|j||jj|jj fIJdS(s Shows a brief description of a SVM's prediction of a site's identity, for use in output as a table. Parameters ---------- out : file, optional prefix : str, optional s----s%5.3fs%s %4s %5s %5.2f %5.2fN( RR RRRRiRRttwo_fofctfofc(RRRRt best_scoret atom_typeR((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyt show_briefOs   "  (t__name__t __module__t__doc__t __slots__tsyststdoutRR(((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyR+s  tmanagercB`s,eZeedZejeedZRS(cC`stjjj||}|jd|jj}|tjjjkrIdS|t k}|rjtjj }n9t |t st |t r|jddj}ng|D]}|jj^q}|dgkrg}n|rd}n |jdddlm}m} | ||j|} |d|d |d |jd |d |jd |d|jd|} t| | d|d|jjj} | dk rd} | jdd| d\}}|dkrZt| dkrtjjj|} qZ| d\}}||jjjkrZ|||jjj krZtjjj|} qZnt!}|j"ddd|t#d|d|j$|j%d|j&d|j'|dg| D]}|d^qdg| D]}|d^qd| }|SdS(s Analyzes a single water site using a SVM to decide whether to re-assign it as an ion. Parameters ---------- i_seq : int debug : bool, optional candidates : list of str, optional Returns ------- svm_prediction or None tparamst,t tXRi(tScatteringEnvironmenttChemicalEnvironmentRRR\tmFoR^smFo-DFcR{RTRDR.R}cS`s |d S(Ni((R~((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRsitidentityRRRRRRRN((RmRntidentifytAtomPropertiest get_atom_typeRtwatert WATER_POORR Rt DEFAULT_IONSRtstrtunicodetreplacetsplitR9R:R]tmmtbx.ions.environmentRRt nearby_atomstget_map_gaussian_fitRRR.RRzRotget_metal_parameterst min_scoretmin_fraction_of_nextRtshow_propertiesRt pdb_atomstid_strtgetvalueR(RRtdebugt candidatest atom_propstexpected_atom_typetauto_candidatesRbRRR=ROt predictionsRt best_guessRt next_guesst next_scoret atom_info_outtpredtresult((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyt analyze_watergsj  %          c C`sT|j}|dt|IJ|dIJt|dkr@dSg}xH|D]@}|jd|d|d|}|dk rM|j|qMqMWg}xV|D]N} |r| jd|d d |dIJn| jdk r|j| qqWt|dkr|dIJ|d IJn7td d|x$|D]} | jd|d d q0W|S( s Uses a SVM to analyze all of a model's water sites and decide whether to re-assign them as ions. Parameters ---------- out : file, optional debug : bool, optional candidates : list of str, optional Returns ------- list of svm_prediction s %d waters to analyzeRiNRRRRRs s1 No waters could be classified as possible ions.sPredicted ions( t_extract_watersRzRR R]RRRR( RRRRtwatersRRt predictiontfilteredR((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pytanalyze_waterss6           (RRtTrueRRRRR(((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyRfsJ(ERt __future__RRt collectionsRt cStringIORtctypesRR+R$RtlibtbxRRtlibtbx.easy_pickleRtlibtbx.str_utilsRt libtbx.utilsR tlibtbx.load_envt cctbx.eltbxR t iotbx.pdbR R<RR tmmtbx.ions.geometryR tmmtbx.ions.identifyRmtmmtbx.ions.svmRtnumpyRGRRt ImportErrorR tenvtfind_in_repositoriesR%tisdirR'R"R#R-RkR5R?RtFalseRURIRJRKRLRMRNRRqt svm_phil_strRRnRR(((sE/public2/ftp/pub/ccp4/6.5/ccp4-6.5/lib/py2/mmtbx/ions/svm/__init__.pyt sh          - =   I  '    1K 3;