Current File : //usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyo
�
���^c @@ sm d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
Z
d d l Z d d l Z d d l
m
Z
d d l m Z d* Z d+ Z d d
l m Z y d d l Z e Z Wn e k
r� e Z n Xd d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l! Z d d l" m# Z# e
j$ e% � Z& d Z' d � Z( d � Z) d � Z* d � Z+ d � Z, d � Z- d � Z. d � Z/ d � Z0 d � Z1 d � Z2 d � Z3 d d � Z5 d d � Z6 d � Z7 d � Z8 e d � Z9 d d! � Z: d" � Z; d# � Z< d$ � Z= d% � Z> d& � Z? d' � Z@ d( � ZA d) � ZB d S(, u�
Mercurial Fileserver Backend
To enable, add ``hgfs`` to the :conf_master:`fileserver_backend` option in the
Master config file.
.. code-block:: yaml
fileserver_backend:
- hgfs
.. note::
``hg`` also works here. Prior to the 2018.3.0 release, *only* ``hg`` would
work.
After enabling this backend, branches, bookmarks, and tags in a remote
mercurial repository are exposed to salt as different environments. This
feature is managed by the :conf_master:`fileserver_backend` option in the salt
master config file.
This fileserver has an additional option :conf_master:`hgfs_branch_method` that
will set the desired branch method. Possible values are: ``branches``,
``bookmarks``, or ``mixed``. If using ``branches`` or ``mixed``, the
``default`` branch will be mapped to ``base``.
.. versionchanged:: 2014.1.0
The :conf_master:`hgfs_base` master config parameter was added, allowing
for a branch other than ``default`` to be used for the ``base``
environment, and allowing for a ``base`` environment to be specified when
using an :conf_master:`hgfs_branch_method` of ``bookmarks``.
:depends: - mercurial
- python bindings for mercurial (``python-hglib``)
i ( t absolute_importt print_functiont unicode_literalsN( t datetime( t FileserverConfigErroru branchesu bookmarksu mixedu baseu
branch_methodu
mountpointu root( t six( t tagifyu hgc C@ sZ t t d k r t St s+ t j d � t St d t k rV t j d t d t � t St S( u-
Only load if mercurial is available
u fileserver_backendue Mercurial fileserver backend is enabled in configuration but could not be loaded, is hglib installed?u hgfs_branch_methodu6 Invalid hgfs_branch_method '%s'. Valid methods are: %s( t __virtualname__t __opts__t Falset HAS_HGt logt errort VALID_BRANCH_METHODS( ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt __virtual__R s
c C@ s
| j � S( u5
Returns all branches for the specified repo
( t branches( t repo( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt
_all_branchese s c C@ sN y5 g t | � D] } | d | k r | ^ q d SWn t k
rI t SXd S( u9
Find the requested branch in the specified repo
i N( R t
IndexErrorR ( R t namet x( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _get_brancho s 5
c C@ s | j � d S( u6
Returns all bookmarks for the specified repo
i ( t bookmarks( R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _all_bookmarksy s c C@ sN y5 g t | � D] } | d | k r | ^ q d SWn t k
rI t SXd S( u;
Find the requested bookmark in the specified repo
i N( R R R ( R R R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt
_get_bookmark� s 5
c C@ s- g | j � D] } | d d k r
| ^ q
S( u1
Returns all tags for the specified repo
i u tip( t tags( R R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _all_tags� s c C@ sN y5 g t | � D] } | d | k r | ^ q d SWn t k
rI t SXd S( u6
Find the requested tag in the specified repo
i N( R R R ( R R R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _get_tag� s 5
c C@ s� | d k r | d } n | | d k s8 | t � k r� | d d k rl t | d | � pk t | d | � S| d d k r� t | d | � p� t | d | � S| d d k r� t | d | � p� t | d | � p� t | d | � Sn t S( u1
Return ref tuple if ref is in the repo.
u baseu
branch_methodu branchesu repou bookmarksu mixed( t envsR R R R ( R R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _get_ref� s
c C@ s t d � � d S( uB
Fatal fileserver configuration issue, raise an exception
u$ Failed to load hg fileserver backendN( R ( ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _failhard� s c C@ s t j j t d d � } t } g } i } x. t D]& } t j t d j | � � | | <q2 Wx�t d D]�} t j
| � } t | t � r�t
t | � � } t g t j t j j j | | � � D]! \ } } | t j | � f ^ q� � }
|
s
t j d | � t � n |
j d | d � } | t k rXt j d | | d j t � � t � n t } x= d � |
D� D]+ } t j d
| | d j t � � t } qoW| r�t � n | j |
� n | } t | t j � s�t j d | � t � n y! t j j j | d � | d <Wn t k
r$n Xt t t j d
d � � }
|
| � j! � } t j j | | � } t j j" | � s�t j# | � n t j$ | � s�t% j& | � t } n y t% j' | � } Wn] t% j j( k
r�t j d | | � t � n- t) k
r } t j d | | � t � n Xy | j* d d � } Wn t% j j+ k
rVd } n X| s�t j j | d d � } t j j- j. | d � �9 } | j/ d � | j/ t j j0 j1 d j | � � � Wd QXn | j i | d 6| d 6| d 6| d 6t j j t d d d j | � � d 6� | j2 | � | j3 � qg W| r t j j t d d � } y� t j j- j. | d � �s } t4 j5 � j6 d � } | j/ d j | � � x> | D]6 } | j/ t j j0 j1 d! j | d | d � � � q�WWd QXWn t7 k
r�q Xt j8 d" | � n | S(# uE
Return a list of hglib objects for the various hgfs remotes
u cachediru hgfsu hgfs_{0}u hgfs_remotesu� Invalid per-remote configuration for hgfs remote %s. If no per-remote parameters are being specified, there may be a trailing colon after the URL, which should be removed. Check the master configuration file.u
branch_methodud Invalid branch_method '%s' for remote %s. Valid branch methods are: %s. This remote will be ignored.u , c s@ s! | ] } | t k r | Vq d S( N( t PER_REMOTE_OVERRIDES( t .0R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pys <genexpr>� s u| Invalid configuration parameter '%s' for remote %s. Valid parameters are: %s. See the documentation for further information.uZ Invalid hgfs remote %s. Remotes must be strings, you may need to enclose the URL in quotesu
mountpointu hash_typeu md5u� Cache path %s (corresponding remote: %s) exists but is not a valid mercurial repository. You will need to manually delete this directory on the master to continue to use this hgfs remote.u< Exception '%s' encountered while initializing hgfs remote %st namesu pathsu .hgu hgrcu w+u [paths]
u default = {0}
Nu repou urlu hashu
{0}.update.lku lockfileu hgfs/remote_map.txtu %d %b %Y %H:%M:%S.%fu # hgfs_remote map as of {0}
u
{0} = {1}
u Wrote new hgfs_remote map to %s(9 t ost patht joinR R R R t text_typet formatt copyt deepcopyt
isinstancet dictt nextt itert iteritemst saltt utilst datat repack_dictlistR R R t getR
t Truet updatet string_typest urlt strip_protot TypeErrort getattrt hashlibt hexdigestt isdirt makedirst listdirt hglibt initt opent ServerErrort Exceptiont configt CommandErrort Nonet filest fopent writet stringutilst to_strt appendt closeR t nowt strftimet OSErrort info( t bp_t
new_remotet repost per_remote_defaultst paramt remotet repo_conft repo_urlt keyt valt per_remote_conft
branch_methodt per_remote_errorst hash_typet repo_hasht rp_R t exct refst
hgconfpatht hgconfigt
remote_mapt fp_t timestamp( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyR@ � s�
$J
,
c C@ s� t j j t d d � } y t j | � } Wn t k
rE g } n Xt � } x7 | D]/ } y | j | d � WqV t k
r� qV XqV Wg } xW | D]O } | d k r� q� n t j j | | � } t j j | � r� | j
| � q� q� Wg } | r`xh | D]] } y t j | � Wn3 t k
rH} t
j d | | � | j
| � q� Xt
j d | � q� Wn x | D] }
| j |
� qgWt | � | f S( uC
Remove cache directories for remotes no longer configured
u cachediru hgfsu hashu refsu0 Unable to remove old hgfs remote cachedir %s: %su hgfs removed old cachedir %s( u hashu refs( R" R# R$ R R> RP R@ t removet
ValueErrorR< RL t shutilt rmtreeR R t debugt bool( RR t cachedir_lsRT R t to_removet itemR# t failedt rdirRb t fdir( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _clear_old_remotesX s@
c C@ s� t j j t d d � } t j j t d d � } g } xi | | f D][ } t j j | � rE y t j | � Wq� t k
r� } | j d j | | � � q� XqE qE W| S( u%
Completely clear hgfs cache
u cachediru hgfsu file_lists/hgfsu Unable to delete {0}: {1}(
R" R# R$ R t existsRk Rl RP RL R& ( t fsb_cachedirt
list_cachedirt errorsRs Rb ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt clear_cache� s $c C@ s� d � } t | t � r" | | � Sg } g } x� t � D]� } | r� y t j | d | � sc w8 n Wq� t k
r� t j | d t j | � � s� q8 q� q� Xn | | � \ } } | j | � | j | � q8 W| | f S( u�
Clear update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c S@ s� d � } g } g } t j j | d � r� y t j | d � Wns t k
r� } | j t j k r� y t j | d � Wq� t k
r� } | | | | � q� Xq� | | | | � q� Xd j | d � } t
j | � | j | � n | | f S( Nc S@ s; d j | d | d | � } t j | � | j | � d S( Nu0 Unable to remove update lock for {0} ({1}): {2} u urlu lockfile( R& R Rm RL ( t errlistR Rb t msg( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt
_add_error� s
u lockfileu Removed lock for {0}u url(
R" R# Rv Ri RP t errnot EISDIRRk Rl R& R Rm RL ( R R} t successRr Rb R| ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _do_clear_lock� s"
u url( R) R* R@ t fnmatchR8 R R% t extend( RW R� t clearedRy R R� Rr ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt
clear_lock� s"
c C@ s� d � } t | t � r" | | � Sg } g } x� t � D]� } | r� y t j | d | � sc w8 n Wq� t k
r� t j | d t j | � � s� q8 q� q� Xn | | � \ } } | j | � | j | � q8 W| | f S( u�
Place an update.lk
``remote`` can either be a dictionary containing repo configuration
information, or a pattern. If the latter, then remotes for which the URL
matches the pattern will be locked.
c S@ s� g } g } t j j | d � s� y' t j j j | d d � � Wd QXWnP t t f k
r� } d j | d | d | � } t
j | � | j | � q� Xd j | d � } t
j | � | j | � n | | f S( Nu lockfileu wu- Unable to set update lock for {0} ({1}): {2} u urlu Set lock for {0}(
R" R# Rv R. R/ RG RH t IOErrorRP R& R Rm RL ( R R� Rr Rb R| ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _do_lock� s
u url( R) R* R@ R� R8 R R% R� ( RW R� t lockedRy R R� Rr ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt lock� s"
c
C@ s� i t d 6d d 6} t � \ | d <} x:| D]2} t j j | d � rk t j d | d | d � q. n t | � \ } } | r� t j d | d � q. n t j d | d � | d j
� | d j � } y | d j � Wn3 t
k
r} t j d
| | d d t j �n2 X| d j � } | d | d k rHt | d <n | d j � t | � q. Wt j j t d
d � } | j d t � t k s�t j j | � rGt j j | � } t j j | � s�t j | � n t d t � }
t j j t � } t j j j | d � �- } | j | j! |
� � t j" d | � Wd QXn t j d t � r�t j j# j$ d t d t d d t d t �}
|
j% | t& d d g d d �� n y* t j' j( t j j t d
d � t) � Wn t* t+ f k
r�n Xd S( u0
Execute an hg pull on all of the repos
u changedu hgfsu backendu lockfileu� Update lockfile is present for hgfs remote %s, skipping. If this warning persists, it is possible that the update process was interrupted. Removing %s or running 'salt-run fileserver.clear_lock hgfs' will allow updates to continue for this remote.u urlu7 Unable to set update lock for hgfs remote %s, skipping.u hgfs is fetching from %su repou1 Exception %s caught while updating hgfs remote %st exc_info_on_logleveli u cachediru hgfs/envs.pt ignore_cacheu wb+u Wrote env cache data to %sNu fileserver_eventsu masteru sock_diru transportt optst listenu updatet prefixu
fileserveru hgfs/hash(, R Ru R" R# Rv R t warningR� R Rm RA t tipt pullRC t loggingt DEBUGR3 RM R� R$ R R2 t isfilet dirnameR= R R. t payloadt SerialR/ RG RH RI t dumpst tracet eventt get_eventt
fire_eventR t
fileservert reap_fileserver_cache_dirt find_fileR� RP ( R0 RT R t _Ry t curtipRb t newtipt env_cachet env_cachedirt new_envst serialRg R� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyR4 � sj
+
% c C@ s� t d r- t j j j d d � t d } n
t d } t d rd t j j j d d � t d } n
t d } t j j j | d | d | �S(
uc
Check if an environment is exposed by comparing it against a whitelist and
blacklist.
u hgfs_env_whitelistu Neonur The hgfs_env_whitelist config option has been renamed to hgfs_saltenv_whitelist. Please update your configuration.u hgfs_saltenv_whitelistu hgfs_env_blacklistur The hgfs_env_blacklist config option has been renamed to hgfs_saltenv_blacklist. Please update your configuration.u hgfs_saltenv_blacklistt whitelistt blacklist( R R. R/ t versionst
warn_untilRJ t check_whitelist_blacklist( t envR� R� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _env_is_exposedB s
c
C@ s� | sG t j j t d d � } t j j t | � } | d
k rG | Sn t � } xt � D] } | d j
� | d d k r� xK t | d � D]6 } | d } | | d k r� d } n | j | � q� Wn | d d k r*xK t
| d � D]6 } | d } | | d k rd } n | j | � q� Wn | j g t | d � D] } | d ^ qA� | d j � qZ Wg t | � D] } t | � rw| ^ qwS(
u@
Return a list of refs that can be used as environments
u cachediru hgfs/envs.pu repou
branch_methodu branchesu mixedi u baseu bookmarksN( u branchesu mixed( u bookmarksu mixed( R" R# R$ R R. R� t check_env_cacheRF t setR@ RA R t addR R4 R RM t sortedR� (
R� R� t cache_matcht retR t brancht branch_namet bookmarkt
bookmark_nameR ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyR b s.
.c
K@ sj i d d 6d d 6} t j j | � s5 | t � k r9 | St j j t d d | | � } t j j t d d | d j | � � } t j j t d d | d j | � � } t j j t d d | d j | � � } t j j | � } t j j | � } t j j | � sHy t j | � WqHt
k
rDt j | � t j | � qHXn t j j | � s�y t j | � Wq�t
k
r�t j | � t j | � q�Xn x�t � D]�}
|
d
r�| j
|
d
t j j � r�q�n | t |
d
� j t j j � } |
d r#t j j |
d | � } n |
d j � t |
| � } | sZ|
d j � q�n t j j | | � t j j | � r�t j j | � r�t j j j | d
� �I }
|
j � } | | d k r�| | d <| | d <|
d j � | SWd QXn y1 |
d j d j | � g d | d d | �Wn( t j j k
rR|
d j � q�n Xt j j j | d � � Wd QXx< t j | � D]+ } y t j | � Wq�t! k
r�q�Xq�Wt j j j | d � � }
|
j" | d � Wd QXy t j | � Wn t
t# f k
r
n X| | d <| | d <y t$ t j% | � � | d <Wn t! k
rRn X|
d j � | SW| S( u
Find the first file to match the path and ref, read the file out of hg
and send the path to the newly cached file
u u pathu relu cachediru hgfs/refsu hgfs/hashu
{0}.hash.*u {0}.hash.blob_sha1u {0}.lku
mountpointu rootu repou ri Nu path:{0}t revt outputu wu w+u stat(& R" R# t isabsR R$ R R& R� R< R= RP Ri R@ t
startswitht sept lent lstripRA R RM R. R� t wait_lockR� R/ RG RH t readt catR? R RE t globRC RI R� t listt stat( R# t tgt_envt kwargst fndt destt hashes_globt blobshadestt lk_fnt destdirt hashdirR t repo_patht refRg t shat filename( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyR� s�
!
#
$
'
c
@ sO d � k r � j d � n i d d 6d d 6} t � f d � d D� � sP | S| d s^ | S| d | d <� j d
d
� } t j j | d � } t j j j
| d � �� } | j � d � | j t
d � } | r
t j r
t j j j | � r
| j t � } n | r;| r;t j j j | | � } | | d
<n | | d <Wd
QX| S( u?
Return a chunk from a file based on the data received
u envu u datau destc 3@ s | ] } | � k Vq d S( N( ( R R ( t load( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pys <genexpr>� s u pathu locu saltenvu relu gzipu rbu file_buffer_sizeN( u pathu locu saltenv( t popt allR2 RF R" R# t normpathR. R/ RG RH t seekR� R R t PY3t is_binaryt decodet __salt_system_encoding__t gzip_utilt compress( R� R� R� t gzipt fpathRg R0 ( ( R� s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt
serve_file� s*
%
c @ sD d � k r � j d � n t � f d � d D� � s< d Si t d d 6} | d } | d } t j j t d d � d d
j | t d � � } t j j | � s� t j j
j | t d � | d <t j j j
| d � � } | j | d � Wd
QX| St j j j
| d � �& } t j j j | j � � | d <Wd
QX| Sd
S( uL
Return a file hash, the hash type is set in the master config file
u envc 3@ s | ] } | � k Vq d S( N( ( R R ( R� ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pys <genexpr> s u pathu saltenvu u hash_typeu relu cachediru hgfs/hashu {0}.hash.{1}u hsumu w+Nu rb( u pathu saltenv( R� R� R R" R# R$ R& R� R. R/ t hashutilst get_hashRG RH RI RJ t
to_unicodeR� ( R� R� R� t relpathR# t hashdestRg ( ( R� s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt file_hash s(
%c C@ s] d | k r | j d � n t j j t d d � } t j j | � s� y t j | � Wq� t j k
r t j d | � g SXn t j j | d j
| d � � } t j j | d j
| d � � } t j j
t | | | � \ } } } | d
k r� | S| rYi } t | � | d <t | � | d <| rIt j j t | | | � n | j | g � Sg S( uD
Return a dict containing the file lists for files and dirs
u envu cachediru file_lists/hgfsu Unable to make cachedir %su {0}.pu saltenvu .{0}.wu filesu dirsN( R� R" R# R$ R R< R= R R t criticalR& R. R� t check_file_list_cacheRF t _get_file_listt
_get_dir_listt write_file_list_cacheR2 ( R� t formRx t
list_cachet w_lockR� t
refresh_cachet
save_cacheR� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt _file_lists# s0 "" c C@ s
t | d � S( uR
Return a list of all files on the file server in a specified environment
u files( R� ( R� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt file_listG s c C@ s d | k r | j d � n d | k s; | d t � k r? g St � } x� t � D]� } | d j � t | | d � } | r� | d j d | d � } x` | D]U } t j j | d | d � } | j
d � s� | j t j j | d | � � q� q� Wn | d j
� qR Wt | � S(
uO
Get a list of all files on the file server in a specified environment
u envu saltenvu repoR� i i u rootu ../u
mountpoint( R� R R� R@ RA R t manifestR" R# R� R� R� R$ RM R� ( R� R� R R� R� t tupR� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyR� N s
*c C@ s g S( u>
Return a list of all empty directories on the master
( ( R� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt file_list_emptydirsg s c C@ s
t | d � S( u8
Return a list of all directories on the master
u dirs( R� ( R� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt dir_listo s c C@ s� d | k r | j d � n d | k s; | d t � k r? g St � } x t � D]} | d j � t | | d � } | rV| d j d | d � } x� | D]� } | d } | j d d � } x� t | � d k rNt j
j | d | d � } | d
k r5| j d � s5| j
t j
j | d | � � q5n | d j d d � } q� Wq� Wn | d j � qR W| d r�| j
| d � n t | � S(
u5
Get a list of all directories on the master
u envu saltenvu repoR� i i u /i u rootu .u ../u
mountpoint( R� R R� R@ RA R R� t rsplitR� R" R# R� R� R� R$ RM R� ( R� R� R R� R� R� t filepatht splitR� ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyR� v s.
&!
( u branchesu bookmarksu mixed( u baseu
branch_methodu
mountpointu root(C t __doc__t
__future__R R R R' R~ R� R� R: R� R" Rk R t salt.exceptionsR R
R t salt.extR R? R3 R
t ImportErrorR t salt.utils.dataR. t salt.utils.filest salt.utils.gzip_utilt salt.utils.hashutilst salt.utils.stringutilst salt.utils.urlt salt.utils.versionst salt.fileservert salt.utils.eventR t getLoggert __name__R R R R R R R R R R R R@ Ru Rz RF R� R� R4 R� R R� R� R� R� R� R� R� R� R� ( ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/hgfs.pyt <module>% sp
� ) 8 / J j $
Mini Shell