Current File : //lib/python2.7/site-packages/salt/fileserver/svnfs.pyo
�
���^c @@ s4 d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
Z
d d l Z d d l m Z d d l
m Z d" Z d d l m Z e Z y" d d l Z e Z e j � Z Wn e k
r� n Xd d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l! Z d d l" Z d d l# m$ Z$ e j% e& � Z' d
Z( d � Z) d � Z* d � Z+ d � Z, d � Z- d � Z. d d � Z0 d d � Z1 d � Z2 d � Z3 e d � Z4 d � Z5 d d � Z6 d � Z7 d � Z8 d � Z9 d � Z: d � Z; d! � Z< d S(# u
Subversion Fileserver Backend
After enabling this backend, branches and tags in a remote subversion
repository are exposed to salt as different environments. To enable this
backend, add ``svnfs`` to the :conf_master:`fileserver_backend` option in the
Master config file.
.. code-block:: yaml
fileserver_backend:
- svnfs
.. note::
``svn`` also works here. Prior to the 2018.3.0 release, *only* ``svn``
would work.
This backend assumes a standard svn layout with directories for ``branches``,
``tags``, and ``trunk``, at the repository root.
:depends: - subversion
- pysvn
.. versionchanged:: 2014.7.0
The paths to the trunk, branches, and tags have been made configurable, via
the config options :conf_master:`svnfs_trunk`,
:conf_master:`svnfs_branches`, and :conf_master:`svnfs_tags`.
:conf_master:`svnfs_mountpoint` was also added. Finally, support for
per-remote configuration parameters was added. See the
:conf_master:`documentation <svnfs_remotes>` for more information.
i ( t absolute_importt print_functiont unicode_literalsN( t datetime( t FileserverConfigErroru
mountpointu rootu trunku branchesu tags( t six( t tagifyu svnc C@ s� t t d k r t St s+ t j d � t Sg } xD d D]< } t j j t | � r8 | j d j
| t | � � q8 q8 W| r� x | D] } t j | � q� Wt j d � t St S( u.
Only load if subversion is available
u fileserver_backenduf Subversion fileserver backend is enabled in configuration but could not be loaded, is pysvn installed?u svnfs_trunku svnfs_branchesu
svnfs_tagsuL Master configuration parameter '{0}' (value: {1}) cannot be an absolute pathu- Subversion fileserver backed will be disabled( u svnfs_trunku svnfs_branchesu
svnfs_tags( t __virtualname__t __opts__t Falset HAS_SVNt logt errort ost patht isabst appendt format( t errorst paramR ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt __virtual__N s"
c C@ s{ y& t t j t j | d � � � } WnC t j j t t t
f k
rk } t j d | d | d | � n X| d j
Sd S( u%
Returns revision ID of repo
u repouC Error retrieving revision ID for svnfs remote %s (cachedir: %s): %su urlu revisionN( t dictR t iteritemst CLIENTt infot pysvnt _pysvnt ClientErrort TypeErrort KeyErrort AttributeErrorR R t numbert None( t repot repo_infot exc( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _revg s &
c C@ s t d � � d S( uB
Fatal fileserver configuration issue, raise an exception
u% Failed to load svn fileserver backendN( R ( ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _failhardy s c C@ s t j j t d d � } t } g } i } x. t D]& } t j t d j | � � | | <q2 Wx�t d D]�} t j
| � } t | t � rst
t | � � } t g t j t j j j | | � � D]! \ } } | t j | � f ^ q� � }
|
s
t j d | � t � n t } x= d � |
D� D]+ } t j d | | d j t � � t } q$W| rct � n | j |
� n | } t | t j � s�t j d | � t � n y! t j j j | d
� | d
<Wn t k
r�n Xt t t j d d � � } | | � j � }
t j j | |
� } t j j! | � s>t j" | � n t j# | � s�y' t$ j% | | � | j& | � t } Wq�t' j( j) k
r�} t j d
| | � t � q�XnG y t$ j* | � Wn3 t' j( j) k
r�} t j d | | � t � n X| j i | d 6| d 6|
d 6| d 6t j j | d � d 6� | j& | � qg W| rt j j t d d � } y� t j j+ j, | d � �s } t- j. � j/ d � } | j0 d j | � � x> | D]6 } | j0 t j j1 j2 d j | d | d � � � q�WWd QXWn t3 k
rqXt j4 d | � n | S( uL
Return the list of svn remotes and their configuration information
u cachediru svnfsu svnfs_{0}u
svnfs_remotesu� Invalid per-remote configuration for remote %s. If no per-remote parameters are being specified, there may be a trailing colon after the URL, which should be removed. Check the master configuration file.c s@ s! | ] } | t k r | Vq d S( N( t PER_REMOTE_OVERRIDES( t .0t x( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pys <genexpr>� s u| Invalid configuration parameter '%s' for remote %s. Valid parameters are: %s. See the documentation for further information.u , u[ Invalid svnfs remote %s. Remotes must be strings, you may need to enclose the URL in quotesu
mountpointu hash_typeu md5u* Failed to initialize svnfs remote '%s': %su� Cache path %s (corresponding remote: %s) exists but is not a valid subversion checkout. You will need to manually delete this directory on the master to continue to use this svnfs remote.u repou urlu hashu update.lku lockfileu svnfs/remote_map.txtu w+u %d %b %Y %H:%M:%S.%fu # svnfs_remote map as of {0}
u
{0} = {1}
Nu Wrote new svnfs_remote map to %s(5 R
R t joinR R R& R t text_typeR t copyt deepcopyt
isinstanceR t nextt iterR t saltt utilst datat repack_dictlistR R R% t Truet updatet string_typest urlt strip_protoR t getattrt hashlibt gett hexdigestt isdirt makedirst listdirR t checkoutR R R R t statust filest fopenR t nowt strftimet writet stringutilst to_strt OSErrorR ( t bp_t
new_remotet repost per_remote_defaultsR t remotet repo_conft repo_urlt keyt valt per_remote_conft per_remote_errorst hash_typet repo_hasht rp_R# t
remote_mapt fp_t timestamp( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt init� s�
$J
&
c C@ s� t j j t d d � } y t j | � } Wn t k
rE g } n Xt � } x7 | D]/ } y | j | d � WqV t k
r� qV XqV Wg } xW | D]O } | d k r� q� n t j j | | � } t j j | � r� | j
| � q� q� Wg } | r`xh | D]] } y t j | � Wn3 t k
rH} t
j d | | � | j
| � q� Xt
j d | � q� Wn x | D] }
| j |
� qgWt | � | f S( uC
Remove cache directories for remotes no longer configured
u cachediru svnfsu hashu refsu1 Unable to remove old svnfs remote cachedir %s: %su svnfs removed old cachedir %s( u hashu refs( R
R R) R R? RI R[ t removet
ValueErrorR= R t shutilt rmtreeR R t debugt bool( RJ t cachedir_lsRL R! t to_removet itemR t failedt rdirR# t fdir( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _clear_old_remotes� s@
c C@ s� t j j t d d � } t j j t d d � } g } xi | | f D][ } t j j | � rE y t j | � Wq� t k
r� } | j d j | | � � q� XqE qE W| S( u&
Completely clear svnfs cache
u cachediru svnfsu file_lists/svnfsu Unable to delete {0}: {1}(
R
R R) R t existsR^ R_ RI R R ( t fsb_cachedirt
list_cachedirR Rf R# ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt clear_cache( s $c C@ s� d � } t | t � r" | | � Sg } g } x� t � D]� } | r� y | | d k r] w8 n Wq� t k
r� t j | � | d k r� q8 q� q� Xn | | � \ } } | j | � | j | � q8 W| | f S( u�
Clear update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c S@ s� d � } g } g } t j j | d � r� y t j | d � Wns t k
r� } | j t j k r� y t j | d � Wq� t k
r� } | | | | � q� Xq� | | | | � q� Xd j | d � } t
j | � | j | � n | | f S( Nc S@ s; d j | d | d | � } t j | � | j | � d S( Nu0 Unable to remove update lock for {0} ({1}): {2} u urlu lockfile( R R R` R ( t errlistR! R# t msg( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt
_add_errorA s
u lockfileu Removed lock for {0}u url(
R
R Ri R\ RI t errnot EISDIRR^ R_ R R R` R ( R! Ro t successRe R# Rn ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _do_clear_lock@ s"
u url( R- R R[ R R R* t extend( RN Rs t clearedR R! Rr Re ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt
clear_lock8 s"
c C@ s� d � } t | t � r" | | � Sg } g } x� t � D]� } | r� y t j | d | � sc w8 n Wq� t k
r� t j | d t j | � � s� q8 q� q� Xn | | � \ } } | j | � | j | � q8 W| | f S( u�
Place an update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c S@ s� g } g } t j j | d � s� y6 t j j j | d d � � } | j d � Wd QXWnP t t f k
r� } d j
| d | d | � } t j | � | j
| � q� Xd j
| d � } t j | � | j
| � n | | f S( Nu lockfileu w+u u- Unable to set update lock for {0} ({1}): {2} u urlu Set lock for {0}( R
R Ri R0 R1 RB RC RF t IOErrorRI R R R` R ( R! Rr Re RY R# Rn ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _do_lockx s
u url( R- R R[ t fnmatchR R R* Rt ( RN Rx t lockedR R! Rr Re ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt lockp s"
c
C@ s� i t d 6d d 6} t � \ | d <} x7| D]/} t j j | d � rk t j d | d | d � q. n t | � \ } } | r� t j d | d � q. n t j d | d � t
| � } y t j | d � Wn7 t
j j k
r } t j d
| d | d | � n Xt
| � } t d � | | f D� � r:q. n | | k rSt | d <n t | � q. Wt j j t d d
� } | j d t � t k s�t j j | � rDt j j | � } t j j | � s�t j | � n t d t � }
t j j t � } t j j j | d � �- } | j | j! |
� � t j" d | � Wd QXn t j d t � r�t j j# j$ d t d t d d t d t �}
|
j% | t& d d g d d �� n y* t j' j( t j j t d d � t) � Wn t* t+ f k
r�n Xd S( u3
Execute an svn update on all of the repos
u changedu svnfsu backendu lockfileu� Update lockfile is present for svnfs remote %s, skipping. If this warning persists, it is possible that the update process was interrupted. Removing %s or running 'salt-run fileserver.clear_lock svnfs' will allow updates to continue for this remote.u urlu8 Unable to set update lock for svnfs remote %s, skipping.u svnfs is fetching from %su repou1 Error updating svnfs remote %s (cachedir: %s): %su cachedirc s@ s | ] } | d k Vq d S( N( R ( R' R( ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pys <genexpr>� s u svnfs/envs.pt ignore_cacheu wb+u Wrote env cache data to %sNu fileserver_eventsu masteru sock_diru transportt optst listenu updatet prefixu
fileserveru
svnfs/hash(, R Rh R
R Ri R t warningR{ R R` R$ R R5 R R R t anyR4 Rv R) R R; t isfilet dirnameR> t envsR0 t payloadt SerialR1 RB RC RF t dumpst tracet eventt get_eventt
fire_eventR t
fileservert reap_fileserver_cache_dirt find_fileRw RI ( R2 RL R! t _R t old_revR# t new_revt env_cachet env_cachedirt new_envst serialRY R� ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyR5 � sj
+
% c C@ s� t d r- t j j j d d � t d } n
t d } t d rd t j j j d d � t d } n
t d } t j j j | d | d | �S(
uc
Check if an environment is exposed by comparing it against a whitelist and
blacklist.
u svnfs_env_whitelistu Neonut The svnfs_env_whitelist config option has been renamed to svnfs_saltenv_whitelist. Please update your configuration.u svnfs_saltenv_whitelistu svnfs_env_blacklistut The svnfs_env_blacklist config option has been renamed to svnfs_saltenv_blacklist. Please update your configuration.u svnfs_saltenv_blacklistt whitelistt blacklist( R R0 R1 t versionst
warn_untilRG t check_whitelist_blacklist( t envR� R� ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _env_is_exposed� s
c C@ s� | sG t j j t d d � } t j j t | � } | d k rG | Sn t � } x1t � D]&} t j j | d | d � } t j j
| � r� | j d � n t j
d | d | d � t j j | d | d � } t j j
| � r| j t j | � � n t j
d | d | d � t j j | d | d
� } t j j
| � re| j t j | � � qZ t j
d | d
| d � qZ Wg t | � D] } t | � r�| ^ q�S(
u@
Return a list of refs that can be used as environments
u cachediru svnfs/envs.pu repou trunku baseud svnfs trunk path '%s' does not exist in repo %s, no base environment will be provided by this remoteu urlu branchesu2 svnfs branches path '%s' does not exist in repo %su tagsu. svnfs tags path '%s' does not exist in repo %sN( R
R R) R R0 R� t check_env_cacheR t setR[ R= t addR R R5 R? t sortedR� ( R| R� t cache_matcht retR! t trunkt branchest tagsR( ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyR� s4 c C@ s� | d k rF t j j | d | d � } t j j | � r? | Sd Sn t j j | d | d � } t j j | � r� | t j | � k r� t j j | | � St j j | d | d � } t j j | � r� | t j | � k r� t j j | | � Sd S( u�
Return the root of the directory corresponding to the desired environment,
or None if the environment was not found.
u baseu repou trunku branchesu tagsN( R
R R) R= R R? ( R! t saltenvR� R� R� ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _env_root5 s ' 'u basec K@ sY i d d 6d d 6} t j j | � s5 | t � k r9 | Sxt � D]} t | | � } | d k rj qC n | d r� | j | d t j j � r� qC n | t | d � j
t j j � } | d r� t j j | d | � } n t j j | | � } t j j | � rC | | d <| | d <y t
t j | � � | d <Wn t k
rLn X| SqC W| S( u�
Find the first file to match the path and ref. This operates similarly to
the roots file sever but with assumptions of the directory structure
based on svn standard practices.
u u pathu relu
mountpointu rootu statN( R
R R R� R[ R� R t
startswitht sept lent lstripR) R� t listt statt Exception( R t tgt_envt kwargst fndR! t env_roott repo_patht full( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyR� O s0
!
#
c
@ sO d � k r � j d � n i d d 6d d 6} t � f d � d D� � sP | S| d s^ | S| d | d <� j d
d
� } t j j | d � } t j j j
| d � �� } | j � d � | j t
d � } | r
t j r
t j j j | � r
| j t � } n | r;| r;t j j j | | � } | | d
<n | | d <Wd
QX| S( u?
Return a chunk from a file based on the data received
u envu u datau destc 3@ s | ] } | � k Vq d S( N( ( R' R( ( t load( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pys <genexpr>� s u pathu locu saltenvu relu gzipu rbu file_buffer_sizeN( u pathu locu saltenv( t popt allR; R R
R t normpathR0 R1 RB RC t seekt readR R t PY3t is_binaryt decodet __salt_system_encoding__t gzip_utilt compress( R� R� R� t gzipt fpathRY R2 ( ( R� s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt
serve_file~ s*
%
c @ s� d � k r � j d � n t � f d � d D� � s< d S� d } | d k r[ d } n i } | d } | d } | s� t j j | � r� | St d | d <t j j t d
d d | d
j | t d � � } t j j | � rJt j
j j | d � �H } | j
� j d � \ } } t j j | � | k rA| | d <| SWd QXn t j
j j | t d � | d <t j j | � }
t j j |
� s�t j |
� n t j
j j | d � �0 } | j d j | d t j j | � � � Wd QX| S( uL
Return a file hash, the hash type is set in the master config file
u envc 3@ s | ] } | � k Vq d S( N( ( R' R( ( R� ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pys <genexpr>� s u pathu saltenvu u baseu trunku relu hash_typeu cachediru svnfsu hashu {0}.hash.{1}u rbu :u hsumNu wu {0}:{1}( u pathu saltenv( R� R� R
R R� R R) R Ri R0 R1 RB RC R� t splitt getmtimet hashutilst get_hashR� R> RF ( R� R� R� R� t relpathR t
cache_pathRY t hsumt mtimet cache_dir( ( R� s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt file_hash� s@
/c C@ s= d | k r | j d � n d | k s; | d t � k r? g St j j t d d � } t j j | � s� y t j | � Wq� t j k
r� t j
d | � g SXn t j j | d j | d � � } t j j | d j | d � � } t j
j t | | | � \ } } } | d k r | S| r9i t � d 6t � d 6t � d
6} xut � D]j} t | | d � }
|
d k r}qRn | d r�t j j |
| d � j t j j � }
t j j |
� s�qRq�n x� t j j j |
� D]� \ } } }
t j j | |
� } t j j | d | � } | d
k r_| d j | � | r_|
r_| d
j | � q_n xV |
D]N } t j j t j j | | � |
� } | d j t j j | d | � � qfWq�WqRW| d r�| d j | d � n x" | D] } t | | � | | <q�W| r)t j
j t | | | � n | j | g � Sg S( uY
Return a dict containing the file lists for files, dirs, emptydirs and symlinks
u envu saltenvu cachediru file_lists/svnfsu Unable to make cachedir %su {0}.pu .{0}.wu filesu dirsu
empty_dirsu rootu
mountpointu .N( R� R� R
R R) R R= R> R R t criticalR R0 R� t check_file_list_cacheR R� R[ R� t rstripR� R1 t os_walkR� R� R� t write_file_list_cacheR; ( R� t formRk t
list_cachet w_lockR� t
refresh_cachet
save_cacheR� R! R� t roott dirsRB R� t
dir_rel_fnt fnamet rel_fnRQ ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt _file_lists� sf ""
( %
0
c C@ s
t | d � S( uV
Return a list of all files on the file server in a specified
environment
u files( R� ( R� ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt file_list s c C@ s
t | d � S( u>
Return a list of all empty directories on the master
u
empty_dirs( R� ( R� ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt file_list_emptydirs s c C@ s
t | d � S( u8
Return a list of all directories on the master
u dirs( R� ( R� ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt dir_list' s ( u
mountpointu rootu trunku branchesu tags(= t __doc__t
__future__R R R R+ Rp Ry R: t loggingR
R^ R t salt.exceptionsR R& t salt.extR R R
R R4 t ClientR t ImportErrort salt.utils.dataR0 t salt.utils.filest salt.utils.gzip_utilt salt.utils.hashutilst salt.utils.patht salt.utils.stringutilst salt.utils.urlt salt.utils.versionst salt.fileservert salt.utils.eventR t getLoggert __name__R R R R$ R% R[ Rh Rl R Rv R{ R5 R� R� R� R� R� R� R� R� R� R� ( ( ( s9 /usr/lib/python2.7/site-packages/salt/fileserver/svnfs.pyt <module> sd
} ) 8 / L * / 6 G
Mini Shell