Current File : //usr/lib/python2.7/site-packages/salt/utils/gitfs.pyc
�
���^c @@ s� d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
Z
d d l Z d d l Z d d l
Z
d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l! Z d d l" Z d d l# Z d d l$ m% Z& d d l' m( Z( d d l) m* Z+ d d l, m- Z- m. Z. m/ Z/ d d l0 m1 Z1 d d
l" m2 Z3 d d l4 m5 Z5 e& d Z6 d
f Z7 f Z8 d Z9 d f Z: d d d d d d f Z; d d d f Z< d Z= d Z> d Z? e j@ eA � ZB y+ d d lC ZC d d lD ZD e3 eC jE � ZF Wn eG k
r�eH ZF n Xy� d d lI ZI eI jJ � � eI jK d � d d lL ZL Wd QXe3 eL jE � ZM e3 eL jN � ZN eM e3 d � k rfy d d lO ZL d d lP ZL Wn eQ k
rMn Xd � ZR eR eL jS _T n y eL jU jV ZV Wn eW k
r�eG ZV n XWn? eG k
r�ZX eH ZM eH ZN eY eX eQ � r�eB jZ d � n n Xe3 d � Z[ e3 d! � Z\ e3 d" � Z] d# � Z^ d$ � Z_ d% e` f d& � � YZa d' ea f d( � � YZb d) ea f d* � � YZc i ec d 6eb d+ 6Zd d, e` f d- � � YZe d. ee f d/ � � YZf d0 ee f d1 � � YZg d2 ee f d3 � � YZh d S(4 uJ
Classes which provide the shared base for GitFS, git_pillar, and winrepo
i ( t absolute_importt print_functiont unicode_literalsN( t datetime( t DEFAULT_MASTER_OPTS( t OrderedDict( t
os_is_running( t FileserverConfigErrort GitLockErrort get_error_message( t tagify( t LooseVersion( t sixu gitfs_ref_typesu nameid u pygit2u useru passwordu pubkeyu privkeyu
passphraseu
insecure_authu
mountpointu rootu refug GitPython is installed, you may wish to set %s_provider to 'gitpython' to use GitPython for %s support.ub pygit2 is installed, you may wish to set %s_provider to 'pygit2' to use pygit2 for for %s support.u� Cache path %s (corresponding remote: %s) exists but is not a valid git repository. You will need to manually delete this directory on the master to continue to use this %s remote.u ignoreu 0.26.3c C@ s# | s
d St j j | � j d � S( Nu utf-8( t Nonet pygit2t ffit stringt decode( t ptr( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt __maybe_stringy s u Failed to import pygit2u 0.3u 0.20.3u 0.20.0c @ sl i
t d 6t d 6t d 6d d 6d d 6d d 6d d 6d d 6d d
6t d 6� � f d � } | � k r� | | � } | d k r� t j | � Sn � | } | d k r't | t j t f � s� t j | � } n t | t j � rg | j d
� D] } | j � ^ q� Sg | D] } t j | � ^ qSy | | � SWn0 t
k
rg} t j d | | � t j | � SXd S( uO
Force params to be strings unless they should remain a different type
u
ssl_verifyu
insecure_authu disable_saltenv_mappingu
stringlistu
env_whitelistu
env_blacklistu saltenv_whitelistu saltenv_blacklistu refspecsu ref_typesu update_intervalc @ sv xo � D]a } y! | j d | � r- | } Pn Wq t k
rg | j d t j | � � rh | } Pqh q Xq Wd } | S( Nu _( t endswitht TypeErrorR t text_typeR
( t keyt itemt ret( t non_string_params( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _find_global� s
u ,uG Failed to enforce type for key=%s with val=%s, falling back to a stringN(
t boolt intR
R R t
isinstancet string_typest listt splitt stript Exceptiont logt error( R t valR t expectedt xt exc( ( R s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
enforce_types� s<
&
c C@ s t d j | � � � d S( u7
Fatal configuration issue, raise an exception
u Failed to load {0}N( R t format( t role( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt failhard� s t GitProviderc B@ s. e Z d Z d d � Z d � Z d d � Z e d � � Z d � Z d � Z d d � Z
d
� Z d � Z d e
d � Z d
� Z e j d d d d � � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z e d � � Z d � Z d � Z! d � Z" RS( u�
Base class for gitfs/git_pillar provider classes. Should never be used
directly.
self.provider should be set in the sub-class' __init__ function before
invoking the parent class' __init__.
u gitfsc
@ s[ | | _ | | _ t j j j | j j d j | j � g � d t d t d t j
d d � �| _ t j
| � | _ xZ t j | j � D]F \ } } d | k r� t j j j | j | d � | j | d <q� q� Wg | D] }
|
| k r� |
^ q� } | rt j d d j | � � n y | | � Wn' t k
rPt | � t | � � n Xt | t � r�t t | � � | _ | j � t j j j | | j d t d t d t j
d t �} | s�t j d
| j | j � t | j � n t }
x� � f d � | D� D]� } t }
| t k r�| j! t" k r�d j | j | | j d j t" � | j! � } | j d
k rr| d 7} n t j | � qd j | j | | j d j � � � } | j d
k r�| d 7} n t j | � qW|
r�t | j � n | j j# | � n | | _ | j � d | j k r1d | j d <n | j d k r�d | j k r�| j j$ d d � d | j d <| j d j% � j& d � r�| j d d | j d <q�n d | j k r�t j j j | j d � j' d � | j d <n
d | j d <d | j k ri | j d <nd xa t j | j d � D]I \ } } d | k r*| j d | } t j j j | d � | d <q*q*WxO t j | j � D]; \ } } | t( k r�t) | | � r�t* | | | � q�q�WxE t( D]= } | d k r t* | d | | j | � n | j+ | � q�Wt) | d � s�d j | j � } y t, | } Wn. t- k
rpt j d | � t | j � n Xt* | d | � t j. d | | j | j � n xa d1 D]Y } t/ | | d � } | r�t j j1 j2 d# d$ j | � � t* | d% j | � | � q�q�Wt3 | d& � t) | d' � r�g | j4 D] }
|
j% � ^ q*| _4 g | j4 D] }
|
t5 k rO|
^ qO} | r�t j d( | j | j d j | � d j t5 � � t | j � q�n t | j t j6 � s�t j d) | j | j � t | j � n t/ t7 | j j d* d+ � � } t j8 r@| | j j9 d, � � j: � | _; n | | j � j: � | _; t/ | d | j; � | _< t j j= j | | j< � | _> t j j= j | d- | j< � | _? t@ j= jA | j> � s�t@ jB | j> � n y | jC � | _D Wnj tE k
rV} d. j | j | j | � } t | tF � r3| d/ 7} n t j | d0 t �t | j � n Xd S(2 Nu {0}_saltenvt strictt recurset key_cbt val_cbc S@ s
t j | � S( N( R R ( R( t y( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt <lambda>� s u
mountpointui The following parameter names are restricted to per-remote use only: %s. This is a bug, please report it.u , u� Invalid per-remote configuration for %s remote '%s'. If no per-remote parameters are being specified, there may be a trailing colon after the URL, which should be removed. Check the master configuration file.c 3@ s! | ] } | � k r | Vq d S( N( ( t .0R( ( t valid_per_remote_params( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr> s u� {0} authentication parameter '{1}' (from remote '{2}') is only supported by the following provider(s): {3}. Current {0}_provider is '{4}'.u gitfsuL See the GitFS Walkthrough in the Salt documentation for further information.uU Invalid {0} configuration parameter '{1}' in remote '{2}'. Valid parameters are: {3}.uM See the GitFS Walkthrough in the Salt documentation for further information.u rootu u winrepou nameu /i i����u .giti����u saltenvu refu _u refspecsu {0}_refspecsu@ The '%s' option has no default value in salt/config/__init__.py.ut The 'refspecs' option was not explicitly defined as a configurable parameter. Falling back to %s for %s remote '%s'.u
env_whitelistu
env_blacklistu Neonu� The gitfs_{0} config option (and {0} per-remote config option) have been renamed to gitfs_salt{0} (and salt{0}). Please update your configuration.u salt{0}u confu ref_typesuX The following ref_types for %s remote '%s' are invalid: %s. The supported values are: %suZ Invalid %s remote '%s'. Remotes must be strings, you may need to enclose the URL in quotesu hash_typeu md5u utf-8u linksu9 Exception caught while initializing {0} remote '{1}': {2}u Perhaps git is not available.t exc_info( u
env_whitelistu
env_blacklist(G t optsR, t saltt utilst datat repack_dictlistt getR+ t TrueR R t global_saltenvt copyt deepcopyt conft iteritemst urlt strip_protoR$ t criticalt joinR R R t dictt nextt itert idt get_urlR* R- t Falset AUTH_PARAMSt providert AUTH_PROVIDERSt updatet rsplitt lowerR R" t PER_SALTENV_PARAMSt hasattrt setattrt add_conf_overlayt _DEFAULT_MASTER_OPTSt KeyErrort debugt getattrR
t versionst
warn_untilt delattrt ref_typest VALID_REF_TYPESR t hashlibt PY3t encodet hexdigestt hasht cachedir_basenamet patht cachedirt linkdirt ost isdirt makedirst init_remotet newR# t GitPython( t selfR8 t remotet per_remote_defaultst per_remote_onlyt override_paramst
cache_rootR, t saltenvt saltenv_confR( t per_remote_collisionst per_remote_conft per_remote_errorst paramt msgt saltenv_ptrR R& t default_refspecsR t invalid_ref_typest hash_typeR) ( ( R6 s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt __init__� s& ! &
/
# $
% $
c
@ s' � f d � } d � j k } d � j k } t � } t j j j � j � r^ | j d � n x� t j j j | � D]� } | j
d � r� | d } n | j d d � \ } } | d k r� | r� | j d � } | d
r� | d
n | d } | | | � qt | d k rt | rt | | | � qt qt W| S(
u�
Return the names of remote refs (stripped of the remote name) and tags
which are map to the branches and tags.
c @ sb | � j k r& | j � j | � n8 | � j k rE | j d � n � j s^ | j | � n d S( uC
Add the appropriate saltenv(s) to the set
u baseN( t saltenv_revmapRQ t baset addt disable_saltenv_mapping( t env_sett rname( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
_check_ref� s u branchu tagu baseu refs/i u /i u remotesi i u tags(
R_ t setR9 R: t stringutilst is_hexR� R� R; R t
startswithR! t partition(
Rp t refsR� t use_branchest use_tagsR t reft rtypeR� t parted( ( Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _get_envs_from_ref_paths� s"
u updatec C@ s t j j j | j | d � S( Nu .lk( R9 R: Rg RG t gitdir( Rp t lock_type( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _get_lock_file� s c @ s& d � f d � } t | � | � d S( uV
Programmatically determine config value based on the desired saltenv
u basec @ s� � f d � } � j d k r5 | t � d � � � S� j j | i � � � d k r.� � � f d � } | | � } yK � j } | r� | | k r� t j d | | � j � j | | � n | SWn t k
r� n X| d k r� � j S� j
r!| d k r t j d � j � j | � n | S| p*| Sn � � k rH| � � � S| � j k r� � j | k r| � j | � � S| t � d � � � Sd S( Nc @ s � d k r | j t j � S| S( Nu rootu
mountpoint( u rootu
mountpoint( t rstripRj t sep( R( ( t name( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s u gitfsu _u refc @ sM � � k r � � S| � j k rE � � j | k rE � j | � Sd Sd S( N( R? R
( t tgt_env( R� Rw Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _get_per_saltenv s u� The per-saltenv configuration has mapped the '%s' branch/tag to saltenv '%s' for %s remote '%s', but this remote has all_saltenvs set to '%s'. The per-saltenv mapping will be ignored in favor of '%s'.u baseuW saltenv mapping is diabled for %s remote '%s' and saltenv '%s' is not explicitly mapped(
R, R[ Rv R= t all_saltenvsR$ RZ RK t AttributeErrorR� R� R
R? ( Rp R� t strip_sepR� t per_saltenv_reft all_saltenvs_ref( R� ( Rw Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _getconf� s@
N( RV ( t clsR� R� ( ( R� s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRW � s 9c C@ si t j j j | j | j � � j t j � } t j j | � rC | St
j d | j � | j | j
� d S( u�
Check if the relative root path exists in the checked-out copy of the
remote. Return the full path to that relative root if it does exist,
otherwise return None.
u7 Root path '%s' not present in %s remote '%s', skipping.N( R9 R: Rg RG Rh t rootR� Rj R� Rk R$ R% R, RK R
( Rp t root_dir( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
check_root3 s -c C@ sg g } d } t j j � } d | d <d | d <t j t j | � d t j j j
� d t j j | j
� d | d t j d t j �} | j � d
} t j r� | j t � } n | j d
k r� t j d | j | j | | � n� d } xO t j j j | d
� D]5 } | j | � r| j | t | � j � � qqW| rct j d | j d j | � � n | S( uV
Remove stale refs so that they are no longer seen as fileserver envs
u git remote prune origint Ct LANGUAGEt LC_ALLt close_fdst cwdt envt stdoutt stderri uO Failed to prune stale branches for %s remote '%s'. Output from '%s' follows:
%su * [pruned] u
u& %s pruned the following stale refs: %su , (! Rj t environR@ t
subprocesst Popent shlexR! R9 R: t platformt
is_windowsRg t dirnameR� t PIPEt STDOUTt communicateR Rb R t __salt_system_encoding__t
returncodeR$ t warningR, RK t itertoolsR� t appendt lenR" RZ RG ( Rp t cleanedt cmd_strR� t cmdt outputt markert line( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt clean_stale_refsE s8
$c @ s � j d | � � � � f d � } g } g } y t j � � Wn~ t k
r� } | j t j k rh q� | j t j k r� y t j � � Wq� t k
r� } | | | � q� Xq� | | | � n6 Xd j | � j
� j � } t j
| � | j | � | | f S( u!
Clear update.lk
R� c @ s6 d j � j � | � } t j | � | j | � d S( Nu0 Unable to remove update lock for {0} ({1}): {2} ( R+ RD R$ RZ R� ( t errlistR) R| ( t lock_fileRp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
_add_errorv s
u% Removed {0} lock for {1} remote '{2}'( R� Rj t removet OSErrort errnot ENOENTt EISDIRt shutilt rmtreeR+ R, RK R$ RZ R� ( Rp R� R� t successt failedR) R| ( ( R� Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
clear_lockp s,
c C@ s t j j | j d � } t j j j � } | j | � sL t j
d | � n�t } d } y | j | d � } Wn3 t j j j
k
r� | j | � t } d } n Xt j d | j | j | | j � | | j k r| j | d | j � t j d | j | j | j � t } n y" t | j | d d t �� } Wn t j j j k
rWg } n Xt | j � } t j d | j | j | | � | | k r�| j | d | j � t j d
| j | j | � t } n y | j d d � } WnL t j j j
k
r| j d � d } n t j j j k
r1d } n Xt j | j � j � } t j d
| j | j | | � | | k r�| j d d | � t j d | j | j | � t } n | r t j j j | d � �0 }
| j |
� t j d | j | j | � Wd QXn d S( u�
For the config options which need to be maintained in the git config,
ensure that the git config file is configured as desired.
u configu& Failed to read from git config file %su remote "origin"u urlu6 Current fetch URL for %s remote '%s': %s (desired: %s)u& Fetch URL for %s remote '%s' set to %su fetcht as_listu5 Current refspecs for %s remote '%s': %s (desired: %s)u% Refspecs for %s remote '%s' set to %su httpu sslVerifyu; Current http.sslVerify for %s remote '%s': %s (desired: %s)u+ http.sslVerify for %s remote '%s' set to %su wu/ Config updates for %s remote '%s' written to %sN(! Rj Rg RG R� R9 R: t configparsert GitConfigParsert readR$ R% RM R= t NoSectionErrort add_sectionR> R
RZ R, RK RD R� t sortedt
NoOptionErrort refspecst set_multivarR R t
ssl_verifyRS t filest fopent write( Rp t
git_configRB t conf_changedt remote_sectionRD R� t desired_refspecsR� t desired_ssl_verifyt fp_( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt enforce_git_config� sv
c C@ s� y@ | j d d � �( t j d | j | j � | j � SWd QXWnW t k
r� } | j t j k r� t j d | j | j | j
d d � | j � n t SXd S( u�
Fetch the repo. If the local copy was updated, return True. If the
local copy was already up-to-date, return False.
This function requires that a _fetch() function be implemented in a
sub-class.
R� u updateu Fetching %s remote '%s'Nu1 Update lock file is present for %s remote '%s', skipping. If this warning persists, it is possible that the update process was interrupted, but the lock could also have been manually set. Removing %s or running 'salt-run cache.clear_git_lock %s type=update' will allow updates to continue for this remote.( t gen_lockR$ RZ R, RK t _fetchR R� t EEXISTR� R� RM ( Rp R) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt fetch� s
c C@ s� yw t j | j | � t j t j Bt j B� } t j | d � �3 t j | t j j
j t j
t j � � � � Wd QXWn)t t f k
r�} | j t j k rVt j j j | j | � d � �L } y+ t t j j
j | j � � j � � } Wn t k
r d } n XWd QX| j d } | j d | � } | j | r�d j | | | | j | j � } | r�| d j | � 7} t | � s�| d 7} q�n t j | � | r�| � n d S| r�t | � r�t j d
| | j | | � | r�� n d S| rt j d | | j | | � n | j! � \ }
} |
rC| j" d d d
| � S| rO� n d Sq�d j | | j | j | � | � } t j# | d t$ �t% | j | � � n Xd j | | j | j � } t j& | � | S( uO
Place a lock file if (and only if) it does not already exist.
u wbNu ri u _global_lockR� uD {0} is enabled and {1} lockfile {2} is present for {3} remote '{4}'.u Process {0} obtained the locku� but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master.u Process %d has a %s %s lock (%s)uY Process %d has a %s %s lock (%s), but this process is not running. Cleaning up lock file.u updateR- u* Unable to set {0} lock for {1} ({2}): {3} R7 u! Set {0} lock for {1} remote '{2}'(' Rj t openR� t O_CREATt O_EXCLt O_WRONLYt fdopenR� R9 R: R� t to_bytesR R t getpidR� t IOErrorR� R� R� R� R t
to_unicodet readlineR� t
ValueErrorR, R8 R+ RK t
pid_existsR$ R� R� t _lockR% R> R RZ ( Rp R� R- t fh_R) t fd_t pidt global_lock_keyR� R| R� t fail( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� sv 8$+
c C@ sn g } g } y | j d d � } Wn# t k
rG } | j | j � n X| d k rd | j | � n | | f S( uG
Place an lock file and report on the success/failure. This is an
interface to be used by the fileserver runner, so it is hard-coded to
perform an update lock. We aren't using the gen_lock()
contextmanager here because the lock is meant to stay and not be
automatically removed.
R� u updateN( R� R R� t strerrorR
( Rp R� R� t resultR) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt locka s i g �?c c@ s� t | t j � s0 t t j d j | � � � n y t | � } Wn t k
rY d } n X| d k ro d } n t | t j t
f � s� | d k r� d } n | | k r� | } n t } z� t j � } x� t
r|y& | j d | d t
� t
} d VPWq� t t t f k
rx} | s.t j � | | k rFt | j | j � � qyt j d | | j | j | � t j | � q� q� Xq� WWd | r�| j d | � n Xd S( u4
Set and automatically clear a lock
u Invalid lock_type '{0}'i g �?R� R- NuF A %s lock is already present for %s remote '%s', sleeping %f second(s)( R R R R R� t EINVALR+ R R� t
integer_typest floatRM t timeR> R� R� R� R� R$ RZ R, RK t sleepR� ( Rp R� t timeoutt
poll_intervalt lock_sett
time_startR) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� t sD
c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( t NotImplementedError( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRm � s c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( R
( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt checkout� s c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt dir_list� s c C@ s% t j j j | d | j d | j �S( uo
Check if an environment is exposed by comparing it against a whitelist
and blacklist.
t whitelistt blacklist( R9 R: R� t check_whitelist_blacklistt saltenv_whitelistt saltenv_blacklist( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt env_is_exposed� s c C@ s
t � � d S( ub
Provider-specific code for fetching, must be implemented in a
sub-class.
N( R
( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( R
( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt envs� s c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt file_list� s c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( R
( Rp Rg R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt find_file� s c C@ s� | j d k r� | j d k r� y | j SWn t k
r< n X| j j d � pd | j j d � pd d } | d k rz | j St j | � S| j S( u0
Resolve dynamically-set branch
u
git_pillaru __env__u pillarenvu saltenvu base( R, t branchR� R� R8 R= R� R R ( Rp t target( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt get_checkout_target� s
c C@ s� | j | � s d S| j | � } | d k r2 d Sx| | j D]q } y" d j | � } t | | � } Wn* t k
r� t j d | j j
| � q< X| | � } | d k r< | Sq< Wd S( uD
Return a tree object for the specified environment
u get_tree_from_{0}u! %s class is missing function '%s'N( R R
R� R_ R+ R[ R� R$ R% t __class__t __name__( Rp R� t tgt_reft ref_typet func_namet funct candidate( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt get_tree� s"
c C@ sw | j d k rg y% | j j d d � \ | _ | _ Wqs t k
rc | j d | _ | j | _ qs Xn | j | _ d S( uW
Examine self.id and assign self.url (and self.branch, for git_pillar)
u
git_pillaru winrepoi u branchN( u
git_pillaru winrepo( R, RK R! R
R RD R� RB ( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRL s %
c C@ s4 y | j SWn"t k
r/g | _ y | j j d � } Wn' t k
rc t j d | j j � n� Xx� t | d � D]q \ } } y | | d g } Wn t k
r� g } n X| j j
t j j
j | j | | d � | g f � qu Wy* | j j d | j | d g g f � Wn t k
r'n X| j SXd S( uq
Return the expected result of an os.walk on the linkdir, based on the
mountpoint value.
u /u- %s class is missing a '_mountpoint' attributei����i i N( t
_linkdir_walkR� t _mountpointR! R$ R% R R t enumeratet
IndexErrorR� R9 R: Rg RG Ri t insert( Rp t partst idxR t dirs( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt linkdir_walk s0
*
c C@ s d S( uX
Only needed in pygit2, included in the base class for simplicty of use
N( ( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt setup_callbacksB s c C@ s
d | _ t S( uS
Override this function in a sub-class to implement auth checking.
N( R
t credentialsR> ( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt verify_authH s c C@ s
t � � d S( uA
This function must be overridden in a sub-class
N( R
( Rp t blobt dest( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
write_fileO s (# R t
__module__t __doc__R� R� R� t classmethodRW R� R� R� R� R� RM R� R t
contextlibt contextmanagerR� Rm R R R R� R R R R R RL t propertyR) R* R, R/ ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR. � s: � & @ + ) Z Q 5 $ Ro c B@ s} e Z d Z d d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
d
� Z d � Z d � Z
d
� Z RS( u
Interface to GitPython
u gitfsc C@ s5 d | _ t t | � j | | | | | | | � d S( Nu gitpython( RO t superRo R� ( Rp R8 Rq Rr Rs Rt Ru R, ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� Z s c
C@ s� | j � } y | j j d � j } Wn t k
r> d } n XxTd | d | f d | d | f f D]*\ } } y | j j | � j } Wn t k
r� qh n X| | k r� | j � SyL | j d d � �4 | j j j | � t
j d | j | j
| � Wd QXWn� t k
rw} | j t j k rQt | j d j | j | j
� � � q�t
j d | j | j | j
� d Sn t k
r�qh n X| j � SWt
j d
| | j | j
� d S( u
Checkout the configured branch/tag. We catch an "Exception" class here
instead of a specific exception class because the exceptions raised by
GitPython when running these functions vary in different versions of
GitPython.
u HEADu origin/u tags/R� u checkoutu) %s remote '%s' has been checked out to %sNu) Checkout lock exists for {0} remote '{1}'u? Error %d encountered obtaining checkout lock for %s remote '%s'uD Failed to checkout %s from %s remote '%s': remote ref does not exist( R t repot rev_parset hexshaR# R
R� R� t gitR R$ RZ R, RK R R� R� R+ R% ( Rp R t head_shat rev_parse_targett checkout_reft
target_shaR) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR b sR
!
c C@ s� t } t j | j � s9 t j j | j � | _ t } nS y t j | j � | _ Wn7 t j j
k
r� t j t
| j | j | j � | SXt j j j | j j d � | _ | j � | S( u�
Initialize/attach to a remote using GitPython. Return a boolean
which will let the calling function know whether or not a new repo was
initialized by this function.
u .git( RM Rj t listdirRh R: t Repot initR7 R> R) t InvalidGitRepositoryErrorR$ R% t
_INVALID_REPORD R, R9 R: Rg RG t working_dirR� R� ( Rp Rn ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRm � s !
c @ s t � } � j � � } | s" | S� j � � rr y | � j � � } Wn t k
r\ | SX� � f d � } n d � } � � f d � } xE | j � D]7 } t | t j � r� | j | | | j � � � q� q� W� j
� � r� | j � j
� � � n | S( uT
Get list of directories for the target environment using GitPython
c @ s t j j | � j � � � S( N( Rj Rg t relpathR� ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s c S@ s | S( N( ( Rg ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s c @ s% t j j j � j � � | d t �S( Nt
use_posixpath( R9 R: Rg RG t
mountpointR> ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s ( R� R R� RY t traverseR R: t TreeR� Rg RG ( Rp R� R t treeRE t add_mountpointR- ( ( Rp R� s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s$
#c C@ s/ g | j j D] } | j ^ q
} | j | � S( un
Check the refs and return a list of the ones which can be used as salt
environments.
( R7 R� Rg R� ( Rp R( t ref_paths( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s "c C@ s& | j j d } y | j � } Wn t k
r? | j � } n Xt } x� | D]� } | j d k r� t j d | j | j
| j | j j d | j
j d � t } qM | j | j | j f k rM t j d | j | j | j k r� d n d | j
| j � t } qM qM W| j � } | s | r"t Sd S( u�
Fetch the repo. If the local copy was updated, return True. If the
local copy was already up-to-date, return False.
i u1 %s has updated '%s' for remote '%s' from %s to %si u* %s has fetched new %s '%s' for remote '%s'u tagu headN( R7 t remotesR� t AssertionErrorRM t
old_commitR
R$ RZ R, R� RK R9 t commitR> t flagst NEW_TAGt NEW_HEADR� ( Rp t origint
fetch_resultst new_objst fetchinfoR� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s6
c @ sU t � } i } � j � � } | s. | | f S� j � � r� y | � j � � } Wn t k
rn | | f SX� � f d � } n d � } � � f d � } x� | j � D]� } t | t j � s� q� n | | | j � � } | j | � t
j | j � r� t
j � } | j | � | j d � | j � }
| j � |
| | <q� q� W| | f S( uJ
Get file list for the target environment using GitPython
c @ s t j j | � j � � � S( N( Rj Rg RE R� ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 s c S@ s | S( N( ( Rg ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 s c @ s% t j j j � j � � | d t �S( NRF ( R9 R: Rg RG RG R> ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 s i ( R� R R� RY RH R R: t BlobRg R� t statt S_ISLNKt modeR t StringIOt stream_datat seekR� t close( Rp R� R� t symlinksRJ RE RK t file_blobt file_patht streamt link_tgt( ( Rp R� s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR s4
c C@ sA | j | � } | s d Sd } d } x� t r| d 7} | t k rN d } Pn y� | | } t j | j � r� t j � } | j | � | j
d � | j � } | j � t
j j j t j j | � | d t �} n" | } t | t j � r� d } n PWq( t k
rd } Pq( Xq( Wt | t j � r=| | j | j f Sd S( uF
Find the specified file in the specified environment
i i RF N( NNN( NNN( R R
R> t SYMLINK_RECURSE_DEPTHRY RZ R[ R R\ R] R^ R� R_ R9 R: Rg RG Rj R� R R: RI RY RX R9 ( Rp Rg R� RJ R- t depthRa Rc Rd ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR & s<
!
c C@ s? y& t j | j d j | � � j j SWn t k
r: d SXd S( uh
Return a git.Tree object matching a head ref fetched into
refs/remotes/origin/
u refs/remotes/origin/{0}N( R: t RemoteReferenceR7 R+ RP RJ R� R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt get_tree_from_branchQ s
c C@ s? y& t j | j d j | � � j j SWn t k
r: d SXd S( uU
Return a git.Tree object matching a tag ref fetched into refs/tags/
u
refs/tags/{0}N( R: t TagReferenceR7 R+ RP RJ R� R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt get_tree_from_tag] s
c C@ s< y | j j | � j SWn t j j t f k
r7 d SXd S( u9
Return a git.Tree object matching a SHA
N( R7 R8 RJ t gitdbR) t ODBErrorR� R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt get_tree_from_shah s c C@ s2 t j j j | d � � } | j | � Wd QXd S( uO
Using the blob object, write the file to the destination path
u wb+N( R9 R: R� R� R] ( Rp R- R. R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR/ q s ( R R0 R1 R� R Rm R R R� R R Rh Rj Rm R/ ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRo V s C & " + t Pygit2c B@ s� e Z d Z d d � Z d � Z d � Z d d � Z d � Z d � Z d � Z
d � Z d
� Z d � Z
d � Z d
� Z d � Z d � Z d � Z d � Z RS( u
Interface to Pygit2
u gitfsc C@ s5 d | _ t t | � j | | | | | | | � d S( Nu pygit2( RO R6 Rn R� ( Rp R8 Rq Rr Rs Rt Ru R, ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� } s c C@ s- y | j � SWn t k
r( | j � SXd S( uY
Compatibility function for pygit2.Reference objects. Older versions of
pygit2 use .get_object() to return the object to which the reference
points, while newer versions use .peel(). In pygit2 0.27.4,
.get_object() was removed. This function will try .peel() first and
fall back to .get_object().
N( t peelR� t
get_object( Rp t obj( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRo � s
c @ s� � j � } d | } d | } d | } y � j j d � } Wn+ t k
rm t j d � j � j � d SXy � j | � j
} WnA t k
r� t j d � j � j � d St k
r� d } n X� j j
� } t � � f d � } yx| | k rs� j � j j | � � j � | | k r;� j j | � � n y" � j � j j | � � j
} Wn4 t k
r�t j d | � j � j d t �d SX| | k ri| j }
t |
t j � rP|
| k rP|
| k rP|
j d � d
} | s t j d |
� j � j � d Sd | } | | k r+| } n � j j |
� j j | � j � n | | d t �sid Sn � j � S| | k rc� j j | � }
t |
t j � s�t j d
| � qcy |
j j
} WnU t k
r y
|
j
} Wq t k
rt j d | � j � j d t �d SXn Xt j d | | � | | k rV| | d t �sVd Sn � j � Sn WnL t k
rz� n9 t k
r�} t j d | � j � j | d t �d SXt j d | � j � j � d S( u4
Checkout the configured branch/tag
u refs/heads/u refs/remotes/origin/u
refs/tags/u HEADu" HEAD not present in %s remote '%s'u, Unable to get SHA of HEAD for %s remote '%s'c @ s� yP � j d d � �4 � j j | � | rE � j j � t j � n Wd QXt SWnk t k
r� } | j t j k r� t | j d j
� j � j � � � q� t
j d | j � j � j � n Xt S( uP
DRY function for checking out either a branch or a tag
R� u checkoutNu) Checkout lock exists for {0} remote '{1}'u? Error %d encountered obtaining checkout lock for %s remote '%s'( R� R7 R t resetR t GIT_RESET_HARDR> R R� R� R+ R, RK R$ R% RM ( R= R R) ( t oidRp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _perform_checkout� s$ u5 pygit2 was unable to get SHA for %s in %s remote '%s'R7 i����uM pygit2 was unable to resolve branch name from HEAD ref '%s' in %s remote '%s'R u. %s does not correspond to pygit2.Commit objectuT Unable to resolve %s from %s remote '%s' to either an annotated or non-annotated tagu SHA of tag %s: %su- Failed to checkout %s from %s remote '%s': %suD Failed to checkout %s from %s remote '%s': remote ref does not existN( R R7 t lookup_referenceRY R$ R� R, RK R
Ro t hexR� R% t listall_referencesR> t create_referenceR R R R R� R� t revparse_singleR t CommitRZ RM R R# ( Rp R t local_reft
remote_reft tag_reft
local_headR; R� Ru R> t head_reft branch_namet remote_headt tag_objt tag_shaR) ( ( Rt Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s�
"
c C@ sa y t j r g SWn t k
r$ n X| j d k rN t j d t | j � g St t
| � j � S( uX
Clean stale local refs so they don't appear as fileserver environments
u� The installed version of pygit2 (%s) does not support detecting stale refs for authenticated remotes, saltenvs will not reflect branches/tags removed from remote '%s'N( R t GIT_FETCH_PRUNER� R+ R
R$ RZ t PYGIT2_VERSIONRK R6 Rn R� ( Rp t
local_refs( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� J s
c C@ s) t j j d � } | t j j t j <t } t j | j � s[ t j
| j � | _ t } nM y t j
| j � | _ Wn1 t k
r� t j t | j | j | j � | SXt j j j | j j d � | _ | j � t j j | j d � } t j j | � r%t t d � k r%| j j j | � n | S( u�
Initialize/attach to a remote using pygit2. Return a boolean which
will let the calling function know whether or not a new repo was
initialized by this function.
u ~u .gitu configu 0.28.0( Rj Rg t
expanduserR t settingst search_patht GIT_CONFIG_LEVEL_GLOBALRM R? Rh t init_repositoryR7 R> t
RepositoryRY R$ R% RC RD R, R9 R: RG t workdirR� R� t existsR� t
_LooseVersiont configt add_file( Rp t homeRn R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRm _ s"
!
$c @ sI � � f d � � t � } � j � � } | s4 | S� j � � r� y' | � j � � j } � j | } Wn t k
r~ | SXt | t j � s� | S� � f d � } n d � } g } t | � r� � | | � j � � � n � � f d � } x' | D] } | j
| | | � � � q� W� j � � rE| j
� j � � � n | S( uS
Get a list of directories for the target environment using pygit2
c @ s� x� t | � D]� } | j � j k r+ q
n � j | j } t | t j � sS q
n | j t j j j
| | j d t �� t
| � r
� | | t j j j
| | j d t �� q
q
Wd S( u�
Traverse through a pygit2 Tree object recursively, accumulating all
the empty directories within it in the "blobs" list
RF N( RJ Rt R7 R R RI R� R9 R: Rg RG R� R> R� ( RJ t blobst prefixt entryR- ( t _traverseRp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s " c @ s t j j | � j � � � S( N( Rj Rg RE R� ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s c S@ s | S( N( ( Rg ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s c @ s% t j j j � j � � | d t �S( NRF ( R9 R: Rg RG RG R> ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s ( R� R R� Rt R7 RY R R RI R� R� RG ( Rp R� R RJ Rt RE R� RK R- ( ( R� Rp R� s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR ~ s0
c C@ s | j j � } | j | � S( un
Check the refs and return a list of the ones which can be used as salt
environments.
( R7 Rx R� ( Rp RL ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s c
C@ s | j j d } | j j � } i } | j d k rD | j | d <n | j d k rb | j | _ n y t j | d <Wn t k
r� n Xy | j | � } Wn� t
k
rU} t | � j � } d | k rt
| j t j � rt j d | j | j d t �nP d | k r/t j d | j | j d t �n" t j d | j | j | d t �t SXy | d
} Wn t t f k
r�| j } n X| d k r�t j d | j | | j � n t j d | j | j � | j j � } | j d
| � } | s| | k s| rt Sd S( u�
Fetch the repo. If the local copy was updated, return True. If the
local copy was already up-to-date, return False.
i u callbacksu pruneu unsupported url protocolu� Unable to fetch SSH-based %s remote '%s'. You may need to add ssh:// to the repo string or libgit2 must be compiled with libssh2 to support SSH authentication.R7 u+ authentication required but no callback setuH %s remote '%s' requires authentication, but no authentication configuredu* Error occurred fetching %s remote '%s': %su received_objectsu& %s received %s objects for remote '%s'u %s remote '%s' is up-to-dateR� N( R7 RM Rx t remotecallbacksR
R+ R R� R� R� t GitErrorR RS R t KeypairR$ R% R, RK R> RM R t received_objectsRZ R� (
Rp RT t refs_pret fetch_kwargsRU R) t exc_strR� t refs_postR� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � sX
c @ s� � � f d � � t � } i } � j � � } | s@ | | f S� j � � r� y' | � j � � j } � j | } Wn t k
r� | | f SXt | t j � s� | | f S� � f d � } n d � } i } t | � r� � | | � j � � � n � � f d � } x3 | j
d g � D] } | j | | | � � � q Wx? t j
| j
d i � � D]" \ } }
|
| | | | � � <q]W| | f S( uG
Get file list for the target environment using pygit2
c @ s xt | � D] } | j � j k r+ q
n � j | j } t | t j � r� t j j j | | j
d t �} | j d g � j
| � t j | | j
j � r� j | | j
j j } | | j d i � | <qq
t | t j � r
� | | t j j j | | j
d t �� q
q
Wd S( u�
Traverse through a pygit2 Tree object recursively, accumulating all
the file paths and symlink info in the "blobs" dict
RF u filesu symlinksN( RJ Rt R7 R R RX R9 R: Rg RG R� R> t
setdefaultR� RY RZ t filemodeR; RI ( RJ R� R� R� Rq t repo_pathRd ( R� Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s c @ s t j j | � j � � � S( N( Rj Rg RE R� ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 % s c S@ s | S( N( ( Rg ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 ' s c @ s% t j j j � j � � | d t �S( NRF ( R9 R: Rg RG RG R> ( Rg ( Rp R� ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 + s u filesu symlinks( R� R R� Rt R7 RY R R RI R� R= R� R RC ( Rp R� R� R` RJ Rt RE R� RK R� Rd ( ( R� Rp R� s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s2
(c C@ s+ | j | � } | s d Sd } d } d } x� t r| d 7} | t k rT d } Pn y� | | } | j } t j | � r� | j | j j } t
j j j
t j j | � | d t �} n, | j | j } t | t j � r� d } n PWq. t k
r d } Pq. Xq. Wt | t j � r'| | j | f Sd S( uF
Find the specified file in the specified environment
i i RF N( NNN( NNN( R R
R> Re R� RY RZ R7 Rt R; R9 R: Rg RG Rj R� R R RI RY RX Rw ( Rp Rg R� RJ R- R[ Rf R� Rd ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR 3 s8
!
c C@ sB y) | j | j j d j | � � � j SWn t k
r= d SXd S( uk
Return a pygit2.Tree object matching a head ref fetched into
refs/remotes/origin/
u refs/remotes/origin/{0}N( Ro R7 Rv R+ RJ RY R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRh [ s
c C@ sB y) | j | j j d j | � � � j SWn t k
r= d SXd S( uX
Return a pygit2.Tree object matching a tag ref fetched into refs/tags/
u
refs/tags/{0}N( Ro R7 Rv R+ RJ RY R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRj f s
c C@ s< y | j j | � j SWn t t t t f k
r7 d SXd S( u<
Return a pygit2.Tree object matching a SHA
N( R7 Rz RJ RY R R� R� R
( Rp R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRm p s c C@ sw t t d � k rH t j d | j � | _ | j ss d � | j _ qs n+ d | _ | j ss t j
d j t � � n d S( u8
Assign attributes for pygit2 callbacks
u 0.23.2R+ c _@ s t S( N( R> ( t argst kwargs( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR4 � s u� pygit2 does not support disabling the SSL certificate check in versions prior to 0.23.2 (installed: {0}). Fetches for self-signed certificates will fail.N( R� R� R t RemoteCallbacksR+ R� R� t certificate_checkR
t warningst warnR+ ( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR* y s c @ sN d � _ t j j � j � r" t St � f d � t D� � sB t S� f d � } � f d � } � j j d � \ } } } | s� d } � j } n | j
� } | d k r� t Sd | k r$d } | j d
� d } | | k r t j
d � j � j � t � j � n | � _ t � f d
� | D� � r�g d D] } t � | d � ^ q5} xz d d f D]l \ }
} | |
} | d k r`y& t j j | � s�| | | � n Wq�t k
r�| | | � q�Xq`q`Wt j | � � _ t Sg | D]$ } t t � | d � � s�| ^ q�}
| |
� n&d | k r!d } t � f d � | D� � } t � f d � | D� � } | ryt S| r�| d k r�� j r�t j
d � j � j � j � t � j � n t j � j � j � � _ t Sg | D]$ } t t � | d � � s�| ^ q�}
| |
� n) t j
d � j � j | � t � j � d S( u�
Check the username and password/keypair info for validity. If valid,
set a 'credentials' attribute consisting of the appropriate Pygit2
credentials object. Return False if a required auth param is not
present. Return True if the required auth parameters are present (or
auth is not configured), otherwise failhard if there is a problem with
authenticaion.
c 3@ s! | ] } t � | d � Vq d S( N( R[ R
( R5 R( ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>� s c @ s6 t j d � j � j d j | � � t � j � d S( uU
Helper function to log errors about missing auth parameters
uP Incomplete authentication information for %s remote '%s'. Missing parameters: %su , N( R$ RF R, RK RG R- ( t missing( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _incomplete_auth� s c @ s0 t j d | | � j � j � t � j � d S( uN
Helper function to log errors about missing key file
u SSH %s (%s) for %s remote '%s' could not be found, path may be incorrect. Note that it may be necessary to clear git_pillar locks to proceed once this is resolved and the master has been started back up. A warning will be logged if this is the case, with instructions.N( R$ RF R, RK R- ( t key_typeRg ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _key_does_not_exist� s u ://u sshu gitu fileu pubkeyu privkeyu @i uJ Keypair specified for %s remote '%s', but remote URL is missing a usernamec 3@ s' | ] } t t � | d � � Vq d S( N( R R[ R
( R5 R( ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>� s u useru
passphrasei i u httpu passwordc 3@ s' | ] } t t � | d � � Vq d S( N( R R[ R
( R5 R( ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>� s c 3@ s' | ] } t t � | d � � Vq d S( N( R R[ R
( R5 R( ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>� s u Invalid configuration for %s remote '%s'. Authentication is disabled by default on http remotes. Either set %s_insecure_auth to True in the master configuration file, set a per-remote config option named 'insecure_auth' to True, or use https or ssh-based authentication.uE Invalid configuration for %s remote '%s'. Unsupported transport '%s'.N( u gitu file( u pubkeyu privkey( u useru pubkeyu privkeyu
passphrase( i u pubkey( i u privkey( u useru password( R
R+ Rj Rg t isabsRD R> t anyRN R� RS R! R$ RF R, RK R- t usert allR[ t isfileR R R� R t
insecure_autht UserPasst password( Rp R� R� t transportt _t addresst required_paramsR� R( t keypair_paramsR' R� t key_patht missing_autht password_okt no_password_auth( ( Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR, � s| "
$
$
c C@ s5 t j j j | d � � } | j | j � Wd QXd S( uO
Using the blob object, write the file to the destination path
u wb+N( R9 R: R� R� R� R; ( Rp R- R. R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR/ s N( R R0 R1 R� Ro R R
R� Rm R R R� R R Rh Rj Rm R* R, R/ ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRn y s"
� 3 A 9 (
{u gitpythont GitBasec B@ s� e Z d Z d d e e d d e d � Z d e e d � Z d � Z d � Z
d d d � Z d d � Z d d � Z
d d � Z d
� Z d � Z e d � Z e d
� Z d � Z d � Z RS( u)
Base class for gitfs/git_pillar
c C@ s) | | _ | d k r | n t | _ | j � | d k rM | | _ | _ nC t j j j
| j d | j � | _ t j j j
| j d � | _ t j j j
| j d � | _ t j j j
| j d � | _
t j j j
| j d d | j � | _ | r%| j | d k r| n g | | | � n d S( u
IMPORTANT: If specifying a cache_root, understand that this is also
where the remotes will be cloned. A non-default cache_root is only
really designed right now for winrepo, as its repos need to be checked
out into the winrepo locations and not within the cachedir.
As of the 2018.3 release cycle, the classes used to interface with
Pygit2 and GitPython can be overridden by passing the git_providers
argument when spawning a class instance. This allows for one to write
classes which inherit from salt.utils.gitfs.Pygit2 or
salt.utils.gitfs.GitPython, and then direct one of the GitBase
subclasses (GitFS, GitPillar, WinRepo) to use the custom class. For
example:
.. code-block:: Python
import salt.utils.gitfs
from salt.fileserver.gitfs import PER_REMOTE_OVERRIDES, PER_REMOTE_ONLY
class CustomPygit2(salt.utils.gitfs.Pygit2):
def fetch_remotes(self):
...
Alternate fetch behavior here
...
git_providers = {
'pygit2': CustomPygit2,
'gitpython': salt.utils.gitfs.GitPython,
}
gitfs = salt.utils.gitfs.GitFS(
__opts__,
__opts__['gitfs_remotes'],
per_remote_overrides=PER_REMOTE_OVERRIDES,
per_remote_only=PER_REMOTE_ONLY,
git_providers=git_providers)
gitfs.fetch_remotes()
u cachediru remotesu envs.pu hashu
file_listsN( R8 R
t
GIT_PROVIDERSt
git_providerst verify_providerRu t remote_rootR9 R: Rg RG R, t env_cachet
hash_cachedirt file_list_cachedirt init_remotes( Rp R8 RM t per_remote_overridesRs t global_onlyR� Ru R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� s& *
c C@ s8 t j | � } g t D]7 } | j d j | j | � r d j | j | � ^ q } | j t k ro | t 7} nc | r� d j | j | j d j t � � } | j d k r� | d 7} n t j
| � t | j � n i } t | � }
|
j
t | � � xl |
D]d } d j | j | � } | | j k rHt j
d | � t | j � n t | | j | � | | <q� Wg | _ x�| D]�}
| j | j | j |
| | | | j | j � } t | d � rv| j � | j � | j d d k r�| j r�| j � n i | _ x� t j | j � D]� \ } } d
| k r| d
} | j j | g � j | � | d k r�| j | j d
� | | j k r�t j d | j | j | j � | | _ q�q�qqWg } x' t j! | j � D] } | j" | � q�Wx[ t j | j# � D]G \ } } | | k r�d
| k r�| j j | d
g � j | � q�q�W| j j | � qvqvWi } x0 | j D]% } | j | j$ g � j | j � qlWg | D]" } t% | | � d
k r�| ^ q�} | rxA | D]6 } t j
d | j d j | | � � t | j � q�Wn t& d � | j D� � r4| j' � n d S( u$
Initialize remotes
u {0}_{1}u� {0} authentication was configured, but the '{1}' {0}_provider does not support authentication. The providers for which authentication is supported in {0} are: {2}.u , u gitfsuM See the GitFS Walkthrough in the Salt documentation for further information.uN Key '%s' not present in global configuration. This is a bug, please report it.u repou __roleu minionu refu baseu� The 'base' environment has been defined in the 'saltenv' param for %s remote %s and will override the branch/tag specified by %s_base (or a per-remote 'base' parameter).i uq The following %s remotes have conflicting cachedirs: %s. Resolve this using a per-remote parameter called 'name'.c s@ s | ] } | j Vq d S( N( Rn ( R5 R( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>� s N(( R@ RA RN R8 R+ R, RO RP RG R$ RF R- R� RQ R* RM R� Ru RU R, R* Rn R� R� R RC Rv R� R� t popR� R� RK t
itervaluest extendR? Rh R� R� t write_remote_map( Rp RM R� Rs R� Rt R( t global_auth_paramsR| Rr t
global_valuesR{ R Rq t repo_objRv Rw R� t all_envst env_namesRB t cachedir_mapR7 t
collisionsR� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� \ s�
7
#/
c C@ s� y t j | j � } Wn t k
r/ g } n Xx9 | j D]. } y | j | j � Wq: t k
rg q: Xq: Wg } x] | D]U } | d k r� qy n t j j
j | j | � } t j
j | � ry | j
| � qy qy Wg } | rUxt | D]i } y t j | � Wn9 t k
r7} t j d | j | | � | j
| � q� Xt j d | j | � q� Wn x | D] } | j | � q\Wt | � }
|
r�| j � n |
S( uK
Remove cache directories for remotes no longer configured
u hashu refsu. Unable to remove old %s remote cachedir %s: %su %s removed old cachedir %s( u hashu refs( Rj R? Ru R� RM R� Rf R� R9 R: Rg RG Rk R� R� R� R$ R% R, RZ R R� ( Rp t cachedir_lsR7 t to_removeR Rg R� t rdirR) t fdirR ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt clear_old_remotes� sB
c C@ s| g } xo | j | j f D][ } t j j | � r y t j | � Wqt t k
rp } | j d j | | � � qt Xq q W| S( u(
Completely clear cache
u Unable to delete {0}: {1}(
Ru R� Rj Rg R� R� R� R� R� R+ ( Rp t errorsR� R) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt clear_cache s u updatec C@ s� g } g } x� | j D]� } | r| y t j | j | � s@ w n Wq| t k
rx t j | j t j | � � sy q qy q| Xn | j d | � \ } } | j | � | j | � q W| | f S( u1
Clear update.lk for all remotes
R� ( RM t fnmatchRD R R R R� R� ( Rp Rq R� t clearedR� R7 R� R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� s
c C@ s� | d k r g } n( t | t � s= t j d | � g } n t } x� | j D]� } t | d d � } | s� | j | f | k rM y | j � r� t
} n Wq� t k
r� } t j d | j | j | d t
�q� XqM qM W| S( u�
Fetch all remotes and return a boolean to let the calling function know
whether or not any remotes were updated in the process of fetching
uL Invalid 'remotes' argument (%s) for fetch_remotes. Must be a list of stringsu nameu2 Exception caught while fetching %s remote '%s': %sR7 N(
R
R R R$ R% RM RM R[ RK R� R> R# R, ( Rp RM t changedR7 R� R) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
fetch_remotes' s(
c C@ s� g } g } x� | j D]� } | r| y t j | j | � s@ w n Wq| t k
rx t j | j t j | � � sy q qy q| Xn | j � \ } } | j | � | j | � q W| | f S( u$
Place an update.lk
( RM R� RD R R R R R� ( Rp Rq t lockedR� R7 R� R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR J s
c
C@ s� i t d 6d d 6} | j � | d <| j d | � rC t | d <n | j d d k } | d t k s| t j j | j � r� t j j | j � } t j j
| � s� t j | � n t } n | r9| j d t � } t
j j | j � } t
j j j | j d � �0 } | j | j | � � t j d | j � Wd
QXn | j j d t � r�t
j j j d | j d
| j d d | j d t �} | j | t d d g d d �� n y t
j j | j | j � Wn t t! f k
r�n Xd
S( uE
.. versionchanged:: 2018.3.0
The remotes argument was added. This being a list of remote URLs,
it will only update matching remotes. This actually matches on
repo.id
Execute a git fetch on all of the repos and perform maintenance on the
fileserver cache.
u changedu gitfsu backendRM u __roleu miniont ignore_cacheu wb+u Wrote env cache data to %sNu fileserver_eventsu masteru sock_diru transportR8 t listenu updateR� u
fileserver(" RM R� R� R> R8 Rj Rg R� R� R� R� Rl R R9 t payloadt SerialR: R� R� R� t dumpsR$ t traceR= t eventt get_eventt
fire_eventR
t
fileservert reap_fileserver_cache_dirR� R R� R� ( Rp RM R; t refresh_env_cachet env_cachedirt new_envst serialR� R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRQ ` sB
&
c C@ s d � | j D� S( ut
Returns a dictionary mapping remote IDs to their intervals, designed to
be used for variable update intervals in salt.master.FileserverUpdate.
A remote's ID is defined here as a tuple of the GitPython/Pygit2
object's "id" and "name" attributes, with None being assumed as the
"name" value if the attribute is not present.
c S@ s1 i | ]' } | j | j t | d d � f � q S( u nameN( t update_intervalRK R[ R
( R5 R7 ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys
<dictcomp>� s ( RM ( Rp ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt update_intervals� s c C@ s� d j | j � | j k r: | j d j | j � | _ n(| j j d j | j � � } | s� | j d t � r| d | _ qb| j d t � rbd | _ qbn� y | j � } Wn& t k
r� t
j | � j � } n X| | j k rt
j d | j | d j | j � � t | j � nH | d k r>| j � r>d | _ n$ | d k rb| j � rbd | _ n t | d � s�t
j d | j � t | j � n d
S( u1
Determine which provider to use
u verified_{0}_provideru {0}_providert quietu pygit2u gitpythonu/ Invalid %s_provider '%s'. Valid choices are: %su , u provideru, No suitable %s provider module is installed.N( R+ R, R8 RO R= t
verify_pygit2R> t verify_gitpythonRS R� R R R� R$ RF RG R- RU ( Rp t desired_provider( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s6
c @ s � f d � } t s< | s8 t j d � j � | � n t Sd � j k rO t Sg } t t k r� | j d j � j t t � � n t j
j j d � s� | j d j � j � � n | r� x | D] } t j | � q� W| s� | � n t Sd � j
d j � j � <t j d � j � t S( uX
Check if GitPython is available and at a compatible version (>= 0.3.0)
c @ s5 t r1 d � j k r1 t j t � j � j � n d S( Nu pygit2( R� R� R$ R% t _RECOMMEND_PYGIT2R, ( ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
_recommend� s uA %s is configured but could not be loaded, is GitPython installed?u gitpythonuW {0} is configured, but the GitPython version is earlier than {1}. Version {2} detected.u gituQ The git command line utility is required when using the 'gitpython' {0}_provider.u verified_{0}_provideru gitpython %s_provider enabled( t GITPYTHON_VERSIONR$ R% R, RM R� t GITPYTHON_MINVERR� R+ R9 R: Rg t whichR8 RZ R> ( Rp R� R� R� R% ( ( Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s<
c @ s^ � f d � } t s< | s8 t j d � j � | � n t Sd � j k rO t Sg } t t k r� | j d j � j t t � � n t t
k r� | j d j � j t
t � � n t t d t � r� t
j j j d � r� | j d j � j � � n | r.x | D] } t j | � qW| s*| � n t Sd � j d j � j � <t j d
� j � t S( u�
Check if pygit2/libgit2 are available and at a compatible version.
Pygit2 must be at least 0.20.3 and libgit2 must be at least 0.20.0.
c @ s5 t r1 d � j k r1 t j t � j � j � n d S( Nu gitpython( R� R� R$ R% t _RECOMMEND_GITPYTHONR, ( ( Rp ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � s uK %s is configured but could not be loaded, are pygit2 and libgit2 installed?u pygit2uT {0} is configured, but the pygit2 version is earlier than {1}. Version {2} detected.uU {0} is configured, but the libgit2 version is earlier than {1}. Version {2} detected.u GIT_FETCH_PRUNEu gituN The git command line utility is required when using the 'pygit2' {0}_provider.u verified_{0}_provideru pygit2 %s_provider enabled( R� R$ R% R, RM R� t
PYGIT2_MINVERR� R+ t LIBGIT2_VERSIONt LIBGIT2_MINVERR[ R R9 R: Rg R� R8 RZ R> ( Rp R� R� R� R% ( ( Rp s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� � sJ
c
C@ s� t j j j | j d � } y� t j j j | d � �z } t j � j d � } | j
d j | j | � � x? | j
D]4 } | j
t j j j d j | j | j � � � qt WWd QXWn t k
r� n Xt j d | j | � d S( u*
Write the remote_map.txt
u remote_map.txtu w+u %d %b %Y %H:%M:%S.%fu # {0}_remote map as of {1}
u
{0} = {1}
Nu Wrote new %s remote map to %s( R9 R: Rg RG Ru R� R� R t nowt strftimeR� R+ R, RM R� t to_strRf RK R� R$ t info( Rp t
remote_mapR� t timestampR7 ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� 2
s"
c C@ s� t j � } x� t j � | d k r� y | j � SWq t k
r� } | j t j k rj t j d � q q� t j d | j | j | j | d t
�Pq Xq Wt j d | j | j | j � d S( uj
Common code for git_pillar/winrepo to handle locking and checking out
of a repo.
i g�������?uI Error %d encountered while obtaining checkout lock for %s remote '%s': %sR7 u� Timed out waiting for checkout lock to be released for %s remote '%s'. If this error persists, run 'salt-run cache.clear_git_lock %s type=checkout' to clear it.N( R R R R� R� R R$ R% R, RK R> R
( Rp R7 R R) ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt do_checkoutO
s(
N( ( ( R R0 R1 R
t PER_REMOTE_ONLYt GLOBAL_ONLYR> R� R� R� R� R� R� R RQ R� R� RM R� R� R� R ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� s& = { ) # 8 ' . 9 t GitFSc B@ s� e Z d Z d Z e j � Z d d e d d e d � Z
d e d d e d � Z d � Z e
d � Z d d � Z d � Z d � Z d
� Z d � Z d � Z d
� Z RS( u>
Functionality specific to the git fileserver backend
u gitfsc
C@ s� | r t j j j � n d } | s4 | | j k r� t j | � } t t | � j
| | d k rg | n g d | d | d | d k r� | n t d | d | �| s� t j
d � n# t j
d t j � � | | j | <| St j
d t j � � | j | S(
u�
If we are not initializing remotes (such as in cases where we just want
to load the config so that we can run clear_cache), then just return a
new __init__'ed object. Otherwise, check the instance map and re-use an
instance if one exists for the current process. Weak references are
used to ensure that we garbage collect instances for threads which have
exited.
R� Rs R� Ru R� u/ Created gitfs object with uninitialized remotesu# Created gitfs object for process %su$ Re-using gitfs object for process %sN( t tornadot ioloopt IOLoopt currentR
t instance_mapt objectt __new__R6 R R� R� R$ RZ Rj R� (
R� R8 RM R� Rs R� Ru R� t io_loopRq ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR x
s&
c C@ s d S( N( ( Rp R8 RM R� Rs R� Ru R� ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR� �
s c C@ s | j | d � S( u@
Return a list of all directories on the master
u dirs( t _file_lists( Rp t load( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR �
s c C@ s� | s4 t j j | j | j � } | d k r4 | Sn t � } xv | j D]k } | j � } x' t j
| j � D] } | j | � ql W| j g | D] } | j
| � r� | ^ q� � qG Wt | � S( uH
Return a list of refs that can be used as environments
N( R9 R� t check_env_cacheR8 R� R
R� RM R R R� R� RQ R R� ( Rp R� t cache_matchR R7 t repo_envst env_listR( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR �
s 3u basec K@ s i d d 6d d 6} t j j | � sN t j j j | � rR | | j � k rR | St j j j | j d | | � } t j j j | j
| d j | � � } t j j j | j
| d j | � � } t j j j | j
| d j | � � } t j j | � } t j j | � }
t j j
| � s`y t j | � Wq`t k
r\t j | � t j | � q`Xn t j j
|
� s�y t j |
� Wq�t k
r�t j |
� t j |
� q�Xn xd| j D]Y} | j | � r�| j | j | � t j � r�q�n | t | j | � � j t j � } | j | � rQt j j j | j | � | � } n | j | | � \ }
} } |
d
k r~q�n d � } t j j | | � ym t j j j | d � �O } t j j j | j � � } | | k r | | d <| | d <| | | � SWd
QXWn. t k
r7} | j! t! j" k r8| � q8n Xt j j j | d � � Wd
QXx< t# j# | � D]+ } y t j | � Wqgt$ k
r�qgXqgW| j% |
| � t j j j | d � � } | j& | � Wd
QXy t j | � Wn t k
r�n X| | d <| | d <| | | � SW| S(
u�
Find the first file to match the path and ref, read the file out of git
and send the path to the newly cached file
u u pathu relu refsu
{0}.hash.*u {0}.hash.blob_sha1u {0}.lkc S@ s | d k r | g | d <n | S( u!
Add a the mode to the return dict. In other fileserver backends
we stat the file to get its mode, and add the stat result
(passed through list() for better serialization) to the 'stat'
key in the return dict. However, since we aren't using the
stat result for anything but the mode at this time, we can
avoid unnecessary work by just manually creating the list and
not running an os.stat() on all files in the repo.
u statN( R
( t fndR[ ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt _add_file_stat�
s
u rNu wu w+(' Rj Rg R� R9 R: R� R� R RG Ru R� R+ R� Rk Rl R� R� RM RG R� R� R� t lstripR� R R
R� t wait_lockR� R� R� R� R� R� R� t globR# R/ R� ( Rp Rg R� R� R R. t hashes_globt blobshadestt lk_fnt destdirt hashdirR7 R� R- t blob_hexshat blob_modeR R� t shaR) t filename( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR �
s�
(!
%$
c
@ s� d � k r � j d � n i d d 6d d 6} t d d d g � } t � f d � | D� � s� t j d d
j | j � � � � | S| d s� | S| d | d <� j d d � } t j
j | d � } t j
j j | d
� �� } | j � d � | j | j d � } | rDt j rDt j
j j | � rD| j t � } n | ru| rut j
j j | | � } | | d <n | | d <Wd QX| S( uG
Return a chunk from a file based on the data received
u envu u datau destu pathu locu saltenvc 3@ s | ] } | � k Vq d S( N( ( R5 R( ( R ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>+ s u< Not all of the required keys present in payload. Missing: %su , u relu gzipu rbu file_buffer_sizeN( R� R� R� R$ RZ RG t
differenceR= R
Rj Rg t normpathR9 R: R� R� R^ R� R8 R Rb t is_binaryR R� t gzip_utilt compress( Rp R R R t required_load_keyst gzipt fpathR� R; ( ( R s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt
serve_file s2
%
c @ s� d � k r � j d � n t � f d � d
D� � s< d Si | j d d 6} | d } | d } t j j j | j � d d j | | j d � � } y9 t j j
j | d � � } | j � | d
<Wd QX| SWn. t
k
r} | j t j k r| � qn Xy t j t j j | � � Wn. t k
rP} | j t j k rQ| � qQn Xt j j j | | j d � | d
<t j j
j | d � � } | j | d
� Wd QX| S( uT
Return a file hash, the hash type is set in the master config file
u envc 3@ s | ] } | � k Vq d S( N( ( R5 R( ( R ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>I s u pathu saltenvu u hash_typeu relu {0}.hash.{1}u rbu hsumNu w+( u pathu saltenv( u N( R� R� R
R8 R9 R: Rg RG R� R+ R� R� R� R� R� R� Rj Rl R� R� R� t hashutilst get_hashR� ( Rp R R R RE Rg t hashdestR� R) ( ( R s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt file_hashA s6
#c C@ sU d | k r | j d � n t j j | j � ss y t j | j � Wqs t j k
ro t j d | j � g SXn t j j j
| j d j | d j t j j
d � � � } t j j j
| j d j | d j t j j
d � � � } t j j | j | | | � \ } } } | d
k r | S| rAi t � d 6i d 6t � d 6} t j j j | d � st| d | j � k r�xm | j D]_ } | j | d � \ }
} | d j |
� | d j | � | d j | j | d � � q~Wn t | d � | d <t | d � | d <| r1t j j | j | | | � n | j | g � S| d k rQi Sg S( uL
Return a dict containing the file lists for files and dirs
u envu Unable to make cachedir %su {0}.pu saltenvu _|-u .{0}.wu filesu symlinksu dirsN( R� Rj Rg Rk R� Rl R% R$ R9 R: RG R+ t replaceR� R� t check_file_list_cacheR8 R
R� R� R� R RM R RQ R R� t write_file_list_cacheR= ( Rp R t formt
list_cachet w_lockR t
refresh_cachet
save_cacheR R7 t
repo_filest
repo_symlinks( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR e sD ( ( !% c C@ s | j | d � S( ub
Return a list of all files on the file server in a specified
environment
u files( R ( Rp R ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s c C@ s g S( uF
Return a list of all empty directories on the master
( ( Rp R ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt file_list_emptydirs� s c C@ s� d | k r | j d � n t j j j | d � rP | d | j � k rP i Sd | k rr | d j d � } n d } | j | d � } t g t j
| � D]' \ } } | j | � r� | | f ^ q� � S( uQ
Return a dict of all symlinks based on a given path in the repo
u envu saltenvu prefixu /u u symlinks( R� R9 R: R� R� R R" R RH R RC R� ( Rp R R� R` R R& ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt symlink_list� s N( ( ( R R0 R1 R, t weakreft WeakKeyDictionaryR R
R R> R R� R RM R R R, R0 R R R; R< ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR q
s$ # d ! $ 3 t GitPillarc B@ s& e Z d Z d Z d � Z d � Z RS( u;
Functionality specific to the git external pillar
u
git_pillarc C@ s; t � | _ g | _ x | j D]} | j | � } | d k r | j d k r� t | d � r� | j j d � p� | j j d � p� d } nW | j
r� | j
} nB | j | j k r� d } n' | j � } | | j k r� d n | } | j
r#| j | � r0| | j | j <| j j | j � q0q3| | j | <q q Wd S( uQ
Checkout the targeted branches/tags from the git_pillar remotes
u __env__u all_saltenvsu pillarenvu saltenvu baseN( R t pillar_dirst pillar_linked_dirsRM R R
R RU R8 R= R� R� R R" t link_mountpointRi R� ( Rp R7 Rh R� t tgt( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR � s( c
C@ s} t j j j | j | j � } t j j j | j | j � � j t j
� } t } t } y�| j d d d d � ��t
t j | j d t �� } | | j k r� t j d | j � t j d | � t j d | j � t } nIt d � | d
D� � st j d | � t } nt j j j | � s0t } n� y t j j j | � } Wn' t k
rrt j d | � t } n� X| | k r(t j d
| | | � yU t j j j � r�| j d � r�t j j | � r�t j | � n
t j | � Wn, t k
r} t j d | | j � � n Xt } t } n | r�t } y t j | j � Wn t k
r[n Xy3 t j j! | � } t j" | � t j d | � Wq�t k
r�} t j# d | | j � � t SXn | r)y+ t j$ | | � t j d | | � t SWq)t k
r%} t j# d | | | j � � t SXn Wd QXWnF t% k
rxt j# d | j& | j' | j& | j& | j( d d � � t SXt S( uv
Ensure that the mountpoint is present in the correct location and
points at the correct path
R� u
mountpointR i
t followlinksu2 Results of walking %s differ from expected resultsu Walk results: %su Expected results: %sc s@ s? | ]5 } t j j j | d � o6 t j j | d � Vq d S( i N( R9 R: Rg t islinkRj Rk ( R5 R( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pys <genexpr>� s i����u- Linkdir parents of %s are not all directoriesu Failed to read destination of %su= Destination of %s (%s) does not match the expected value (%s)u \\u; Failed to remove existing git_pillar mountpoint link %s: %su# Successfully made linkdir parent %su- Failed to os.makedirs() linkdir parent %s: %su% Successfully linked %s to cachedir %su- Failed to create symlink to %s at path %s: %sNu� Timed out setting mountpoint lock for %s remote '%s'. If this error persists, it may be because an earlier %s checkout was interrupted. The lock can be cleared by running 'salt-run cache.clear_git_lock %s type=mountpoint', or by manually removing %s.() R9 R: Rg RG Ri R" Rh R� R� Rj R� RM R� R t walkR) R$ RZ R> R� RE t readlinkR# R� R� R� Rk R� R� R� t exceptiont __str__R� R� Rl R% t symlinkR R, RK R� (
Rp R7 t
lcachelinkt
lcachedestt wipe_linkdirt create_linkt walk_resultst ldestR) t ldirname( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRB � s� -
( R R0 R1 R, R RB ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR? � s t WinRepoc B@ s# e Z d Z d Z i Z d � Z RS( u6
Functionality specific to the winrepo runner
u winrepoc C@ sO i | _ x? | j D]4 } | j | � } | d k r | | j | j <q q Wd S( uN
Checkout the targeted branches/tags from the winrepo remotes
N( t winrepo_dirsRM R R
RK ( Rp R7 Rh ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyR Z s
( R R0 R1 R, RS R ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyRR Q s (i R1 t
__future__R R R R@ R3 R� R� R Ra t loggingRj R� R� RY R� R t tornado.ioloopR R= R t salt.utils.configparserR9 t salt.utils.datat salt.utils.filest salt.utils.gzip_utilt salt.utils.hashutilst salt.utils.itertoolst salt.utils.patht salt.utils.platformt salt.utils.stringutilst salt.utils.urlt salt.utils.usert salt.utils.versionst salt.fileservert salt.configR RX t salt.utils.odictR t salt.utils.processR R� t salt.exceptionsR R R t salt.utils.eventR
R R� t salt.extR R` R R Re RP RN RT R� R� RC t getLoggerR R$ R: Rk t __version__R� R# R
R� t catch_warningst simplefilterR R� R� t
pygit2.ffit
pygit2.remotet ImportErrorR Rq t maybe_stringR� R� R� R) R RH R� R� R� R* R- R
R. Ro Rn R� R� R R? RR ( ( ( s4 /usr/lib/python2.7/site-packages/salt/utils/gitfs.pyt <module> s�
6 � � � �� $� � �
� � Z� K�
Mini Shell