Current File : //lib/python2.7/site-packages/salt/fileserver/azurefs.pyc
�
���^c @@ s� d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z
d d l Z
d d l Z
d d l
Z
d d l Z
d d l Z
d d l Z
d d l m Z yC d d l Z e e j j � e d � k r� e d � � n e Z Wn e e f k
re Z n Xd d l m Z d Z e j � Z d � Z! d
d � Z" d � Z# d
� Z$ d � Z% d � Z& d � Z' d � Z( d � Z) d � Z* d � Z+ d S( u�
The backend for serving files from the Azure blob storage service.
.. versionadded:: 2015.8.0
To enable, add ``azurefs`` to the :conf_master:`fileserver_backend` option in
the Master config file.
.. code-block:: yaml
fileserver_backend:
- azurefs
Starting in Salt 2018.3.0, this fileserver requires the standalone Azure
Storage SDK for Python. Theoretically any version >= v0.20.0 should work, but
it was developed against the v0.33.0 version.
Each storage container will be mapped to an environment. By default, containers
will be mapped to the ``base`` environment. You can override this behavior with
the ``saltenv`` configuration option. You can have an unlimited number of
storage containers, and can have a storage container serve multiple
environments, or have multiple storage containers mapped to the same
environment. Normal first-found rules apply, and storage containers are
searched in the order they are defined.
You must have either an account_key or a sas_token defined for each container,
if it is private. If you use a sas_token, it must have READ and LIST
permissions.
.. code-block:: yaml
azurefs:
- account_name: my_storage
account_key: 'fNH9cRp0+qVIVYZ+5rnZAhHc9ycOUcJnHtzpfOr0W0sxrtL2KVLuMe1xDfLwmfed+JJInZaEdWVCPHD4d/oqeA=='
container_name: my_container
- account_name: my_storage
sas_token: 'ss=b&sp=&sv=2015-07-08&sig=cohxXabx8FQdXsSEHyUXMjsSfNH2tZ2OB97Ou44pkRE%3D&srt=co&se=2017-04-18T21%3A38%3A01Z'
container_name: my_dev_container
saltenv: dev
- account_name: my_storage
container_name: my_public_container
.. note::
Do not include the leading ? for sas_token if generated from the web
i ( t absolute_importt print_functiont unicode_literalsN( t LooseVersionu 0.20.0u+ azure.storage.__version__ must be >= 0.20.0( t sixu azurefsc C@ s? t t d k r t St s t Sd t k r. t St � s; t St S( uQ
Only load if defined in fileserver_backend and azure.storage is present
u fileserver_backendu azurefs( t __virtualname__t __opts__t Falset HAS_AZUREt _validate_configt True( ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt __virtual__S s u basec K@ s� i d d 6d d 6} x� t j d g � D]� } | j d d � | k rK q' n t j j t | � | � } t j j | � r' t j j t | � r' | | d <| | d <y t
t j | � � | d <Wn t k
r� n X| Sq' W| S( u6
Search the environment for the relative path
u u pathu relu azurefsu saltenvu baseu stat(
R t gett ost patht joint _get_container_patht isfilet saltt
fileservert is_file_ignoredt listt statt Exception( R t saltenvt kwargst fndt containert full( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt find_filef s
c C@ sL g } x3 t j d g � D] } | j | j d d � � q Wt t | � � S( u_
Each container configuration can have an environment setting, or defaults
to base
u azurefsu saltenvu base( R R t appendR t set( t saltenvsR ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt envs� s c
@ s[ i d d 6d d 6} d } t � f d � | D� � s\ t j d d j | j � � � � | S| d sj | S| d
| d <� j d d � } t j j | d � } t
j j j
| d � �� } | j � d � | j t d
� } | rt j rt
j j j | � r| j t � } n | rG| rGt
j j j | | � } | | d <n | | d <Wd QX| S( u?
Return a chunk from a file based on the data received
u u datau destu pathu locu saltenvc 3@ s | ] } | � k Vq d S( N( ( t .0t x( t load( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pys <genexpr>� s u< Not all of the required keys present in payload. Missing: %su , u relu gzipu rbu file_buffer_sizeN( u pathu locu saltenv( t allt logt debugR t
differenceR t NoneR
R t normpathR t utilst filest fopent seekt readR R t PY3t is_binaryt decodet __salt_system_encoding__t gzip_utilt compress( R$ R t rett required_load_keyst gzipt fpatht fp_t data( ( R$ s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt
serve_file� s.
%
c C@ sP xIt d D]=} t | � } yU t j j | � sB t j | � n/ t j j | � sq t j | � t j | � n Wn# t k
r� } t
j d � q n Xt | � } | d } y | j
| � } Wn# t k
r� } t
j d � q n Xg | D] } | j ^ q� } t | � } x� t j j j | � D]� \ } }
} x� | D]{ } t j j | | � }
t j j |
| � } | | k r;t j j |
d |
� y t j |
� Wq�t k
r�q�Xq;q;W|
r%| r%t j | � q%q%Wxy| D]q} t j j | | j � }
t } t j j |
� rj| j j j } t j t j j j |
d � j d � � } | | k rpt! } qpn t! } | r�t j j t j j" |
� � s�t j t j j" |
� � n |
d } t j j | |
� t j j# j$ | d � � Wd QXy | j% | | j |
� Wn# t k
r+} t
j d
� q�n Xy t j | � WqTt k
rPqTXq�q�W| d } | d } t j j | | � t j j# j$ | d � � Wd QXt j j# j$ | d � � } t j j& j' | | � Wd QXy t j | � Wn t k
r�n Xy- t j j t d d d
� } t j | � Wq t k
rGt
j d � q Xq Wd S( u"
Update caches of the storage containers.
Compares the md5 of the files on disk to the md5 of the blobs in the
container, and only updates if necessary.
Also processes deletions by walking the container caches and comparing
with the list of blobs in the container
u azurefsu3 Error occurred creating cache directory for azurefsu container_nameu- Error occurred fetching blob list for azurefsu .lku md5u hexu wNu) Error occurred fetching blob from azurefsu .listu cachediru hashesu; Problem occurred trying to invalidate hash cach for azurefs(( R R R
R t existst makedirst isdirt shutilt rmtreeR R& t exceptiont _get_container_servicet
list_blobst nameR R R+ t os_walkR t relpathR t wait_lockt unlinkR t
propertiest content_settingst content_md5t base64t b64encodet hashutilst get_hashR2 R
t dirnameR, R- t get_blob_to_patht jsont dump( R R t exct blob_serviceRE t blob_listt blobt
blob_namest blob_sett roott dirsR, t ft fnameRG t updatet
source_md5t local_md5t lk_fnt container_listR: t
hash_cachedir( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyR_ � s�
%
*
c @ s� t � f d � d D� � s d Si t d d 6} | d } | d } t j j t d d d � } t j j j | � d d
j | t d � � } t j j | � sAt j j
t j j | � � s� t j t j j | � � n t j j
j | t d � | d <t j j j | d � �' } | j t j j j | d � � Wd
QX| St j j j | d � �& } t j j j | j � � | d <Wd
QX| Sd
S( uL
Return a file hash based on the hash type set in the master config
c 3@ s | ] } | � k Vq d S( N( ( R" R# ( R$ ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pys <genexpr> s u pathu saltenvu u hash_typeu relu cachediru azurefsu hashesu {0}.hash.{1}u hsumu w+Nu rb( u pathu saltenv( u N( R% R) R R
R R R R+ t formatR R= RQ R> RO RP R, R- t writet stringutilst to_strt
to_unicodeR/ ( R$ R R6 RG R Rd t hashdestR: ( ( R$ s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt file_hash s(
&%c C@ s� t � } y� x� t d D]� } | j d d � | d k r? q n t | � d } | d } t j j | | d � t j j | � s� q n t j
j j | d � �) } | j
t t j
j j | � � � Wd QXq WWn t k
r� } t j d � n Xt | � S(
u?
Return a list of all files in a specified environment
u azurefsu saltenvu baseu .listu .lki u rNu� azurefs: an error ocurred retrieving file lists. It should be resolved next time the fileserver updates. Please do not manually modify the azurefs cache directory.( R R R R R R RH R
R R= R+ R, R- R_ RS R$ R R& t errorR ( R$ R6 R Rc t lkR: RU ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt file_list. s
0c C@ sk t � } t | � } xI | D]A } | } x2 | r\ t j j | � } | r+ | j | � q+ q+ Wq Wt | � S( uE
Return a list of all directories in a specified environment
( R Rn R
R RQ t addR ( R$ R6 R, R] RQ ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt dir_listF s
c C@ se t j j t d d � } d j | j d d � | j d d � | j d d � � } t j j | | � S( u�
Get the cache path for the container in question
Cache paths are generate by combining the account name, container name,
and saltenv, separated by underscores
u cachediru azurefsu {0}_{1}_{2}u account_nameu u container_nameu saltenvu base( R
R R R Re R ( R R[ t
container_dir( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyR U s
c C@ s� d | k r/ t j j | d d | d �} nE d | k r^ t j j | d d | d �} n t j j | d � } | j � } | S( u�
Get the azure block blob service for the container in question
Try account_key, sas_token, and no auth in that order
u account_keyu account_namet account_keyu sas_tokent sas_token( t azuret storaget CloudStorageAccountt create_block_blob_service( R t accountRV ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyRC c s # # c C@ s� t t d t � s$ t j d � t Sxa t d D]U } t | t � sX t j d | � t Sd | k sp d | k r/ t j d | � t Sq/ Wt S( uF
Validate azurefs config, return False if it doesn't validate
u azurefsu? azurefs configuration is not formed as a list, skipping azurefsud One or more entries in the azurefs configuration list are not formed as a dict. Skipping azurefs: %su account_nameu container_nameu\ An azurefs container configuration is missing either an account_name or a container_name: %s( t
isinstanceR R R& Rl R t dictR
( R ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyR s s
(, t __doc__t
__future__R R R RM t loggingR
R@ t salt.fileserverR t salt.utils.filest salt.utils.gzip_utilt salt.utils.hashutilst salt.utils.jsont salt.utils.patht salt.utils.stringutilst salt.utils.versionsR t
azure.storageRt Ru t __version__t ImportErrorR
R t AttributeErrorR t salt.extR R t getLoggerR& R R R! R< R_ Rk Rn Rp R RC R ( ( ( s; /usr/lib/python2.7/site-packages/salt/fileserver/azurefs.pyt <module>/ sD
" a
Mini Shell