Current File : //lib/python2.7/site-packages/salt/fileserver/s3fs.pyc
�
���^c @@ s� d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
j Z d d l
Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m Z e j e � Z d Z e Z d � Z d � Z! d
d � Z" d � Z# d
� Z$ d � Z% d � Z& d � Z' d � Z( d � Z) d � Z* d � Z+ d � Z, d � Z- d � Z. d � Z/ d � Z0 d � Z1 d � Z2 d � Z3 e4 d � Z5 d � Z6 d S( uN
Amazon S3 Fileserver Backend
.. versionadded:: 0.16.0
This backend exposes directories in S3 buckets as Salt environments. To enable
this backend, add ``s3fs`` to the :conf_master:`fileserver_backend` option in the
Master config file.
.. code-block:: yaml
fileserver_backend:
- s3fs
S3 credentials must also be set in the master config file:
.. code-block:: yaml
s3.keyid: GKTADJGHEIQSXMKKRBJ08H
s3.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
Alternatively, if on EC2 these credentials can be automatically loaded from
instance metadata.
This fileserver supports two modes of operation for the buckets:
1. :strong:`A single bucket per environment`
.. code-block:: yaml
s3.buckets:
production:
- bucket1
- bucket2
staging:
- bucket3
- bucket4
2. :strong:`Multiple environments per bucket`
.. code-block:: yaml
s3.buckets:
- bucket1
- bucket2
- bucket3
- bucket4
Note that bucket names must be all lowercase both in the AWS console and in
Salt, otherwise you may encounter ``SignatureDoesNotMatch`` errors.
A multiple-environment bucket must adhere to the following root directory
structure::
s3://<bucket name>/<environment>/<files>
.. note:: This fileserver back-end requires the use of the MD5 hashing algorithm.
MD5 may not be compliant with all security policies.
.. note:: This fileserver back-end is only compatible with MD5 ETag hashes in
the S3 metadata. This means that you must use SSE-S3 or plaintext for
bucket encryption, and that you must not use multipart upload when
uploading to your bucket. More information here:
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
Objects without an MD5 ETag will be fetched on every fileserver update.
If you deal with objects greater than 8MB, then you should use the
following AWS CLI config to avoid mutipart upload:
.. code-block:: text
s3 =
multipart_threshold = 1024MB
More info here:
https://docs.aws.amazon.com/cli/latest/topic/s3-config.html
i ( t absolute_importt print_functiont unicode_literalsN( t six( t filter( t quotei c C@ s t � } t | j � � S( u^
Return a list of directories within the bucket that can be
used as environments.
( t _initt listt keys( t metadata( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt envsn s c
C@ s� t � } t r� t j d � x� t j | � D]� \ } } x� t | � D]u } xl t j | � D][ \ } } xL | D]D } t | | | � } t j d | | | � t | | | | | � qn Wq[ WqE Wq, Wt j d � n d S( u/
Update the cache file for the bucket.
u Syncing local cache from S3...u %s - %s : %su# Sync local cache from S3 completed.N( R t S3_SYNC_ON_UPDATEt logt infoR t iteritemst _find_filest _get_cached_file_namet _get_file_from_s3( R t saltenvt env_metat bucket_filest buckett filest file_patht cached_file_path( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt updatey s
&u basec
K@ s; d | k r | j d � n i d d 6d d 6} t � } | sL | | k rP | St | | � } t � s� t j j | | � } n xi | D]a } xW t j | � D]C \ } } | | k r� t
j t | � r� | | d <| | d <Pq� q� Wq� Pq� W| d s| d r| St
| d | | � } t | | | d | | � | S( u�
Look through the buckets cache file for a match.
If the field is found, it is retrieved from S3 only if its cached version
is missing, or if the MD5 does not match.
u envu bucketu pathN( t popt NoneR R t _is_env_per_buckett ost patht joinR R t fst is_file_ignoredt __opts__R R (
R R t kwargst fndR t env_filesR t bucket_nameR R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt find_file� s.
c C@ s� d | k r | j d � n i } d | k r2 | Sd | k sU d | k sU | d rY | St | d | d | d � } t j j | � r� t j j j | � | d <d | d <n | S( u!
Return an MD5 file hash
u envu saltenvu pathu bucketu hsumu md5u hash_type( R R R R t isfilet saltt utilst hashutilst get_hash( t loadR$ t retR ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt file_hash� s #
c
C@ s� d | k r | j d � n i d d 6d d 6} d | k sT d | k sT d | k rX | Sd | k sp d | k rt | S| j d d
� } t | d | d | d � } t | d g | d � d
| d <t j j j | d � �� } | j | d � | j
t d � } | r<t j
r<t j j j | � r<| j t � } n | rm| rmt j j j | | � } | | d <n | | d <Wd
QX| S( u?
Return a chunk from a file based on the data received
u envu u datau destu pathu locu saltenvu bucketu gzipi u rbu file_buffer_sizeN( R t getR R t _trim_env_off_pathR) R* R t fopent seekt readR" R t PY3t is_binaryt decodet __salt_system_encoding__t gzip_utilt compress( R- R$ R. t gzipR t fp_t data( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt
serve_file� s0
$
"%
c C@ s� d | k r | j d � n g } d | k r2 | S| d } t � } | sX | | k r\ | Sxp t | | � D]^ } xU t j | � D]D } g | D] } t j t | � s� | ^ q� } | t | | � 7} q� Wqm W| S( uR
Return a list of all files on the file server in a specified environment
u envu saltenv( R R R R t
itervaluesR R! R" R1 ( R- R. R R R t bucketst fR ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt file_list� s
+c C@ s t � g S( u>
Return a list of all empty directories on the master
( R ( R- ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt file_list_emptydirs s c C@ s� d | k r | j d � n g } d | k r2 | S| d } t � } | sX | | k r\ | Sxj t | | � D]X } xO t j | � D]> } t | | d t �} | g | D] } | r� | ^ q� 7} q� Wqm W| S( u8
Return a list of all directories on the master
u envu saltenvt
trim_slash( R R t
_find_dirsR R? R1 t True( R- R. R R R t dirst _f( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt dir_list# s
+c C@ s� d t k r t d n d
} d t k r2 t d n d
} d t k rN t d n d
} d t k rj t d n d
} d t k r� t d n d
} d t k r� t d n d
} d t k r� t d n d
} d t k r� t d n d
} | | | | | | | | f S( u,
Get AWS keys from pillar or config
u s3.keyu s3.keyidu s3.service_urlu
s3.verify_sslu
aws.kms.keyidu
aws.kmw.keyidu s3.locationu
s3.path_styleu s3.https_enableN( R" R ( t keyt keyidt service_urlt
verify_sslt kms_keyidt locationt
path_stylet https_enable( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt _get_s3_keyA s c C@ s} t � } t j � t } d } y+ t j j | � | k rI t | � } n Wn t k
r] n X| d k ry t | � } n | S( ux
Connect to S3 and download the metadata for each file in all buckets
specified and cache the data to disk.
N(
t _get_buckets_cache_filenamet timet S3_CACHE_EXPIRER R R t getmtimet _read_buckets_cache_filet OSErrort _refresh_buckets_cache_file( t
cache_filet expR ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyR \ s
c C@ s t j j t d d � S( u,
Return the path to the s3cache dir
u cachediru s3cache( R R R R" ( ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt _get_cache_dirs s c C@ s\ t j j t � | | | � } t j j t j j | � � sX t j t j j | � � n | S( u<
Return the cached file name for a bucket path file
( R R R R\ t existst dirnamet makedirs( R& R R R ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyR | s c C@ s> t � } t j j | � s+ t j | � n t j j | d � S( ui
Return the filename of the cache for bucket contents.
Create the path if it does not exist.
u buckets_files.cache( R\ R R R] R_ R ( t cache_dir( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyRS � s c @ s t j d � t � \ } } � � } � � � i } | | � � � � � f d � } t � r�xRt j t � � D]9\ } } g } x| D]} i }
| | � } | s� q� n g | D] } d | k r� | ^ q� |
| <| j |
� |
| s� i }
x6 | D]. } d | k sd | k r� |
j | � q� q� Wy t j d |
d | � w� Wq�t
k
r�d |
k r�t j d |
d | � q� q�t j d � i Sq�Xq� q� W| | | <qq Wnx�t � D]�} | | � } | s�q�n g | D] } d | k r�| ^ q�} | s�i }
x6 | D]. } d | k s/d | k r|
j | � qqWy t j d |
d | � w�Wq�t
k
r�d |
k r�t j d |
d | � q�q�t j d � i Sq�Xn g | D], } t j j
| d � j d d � d
^ q�} t | � } x� | D]� } g | D] } | d j | � r | ^ q } | | k rIg | | <n t } x8 | | D], }
| |
k rZ|
| c | 7<t } PqZqZW| s�| | j i | | 6� q�q�Wq�Wt j j | � r�t j | � n t j d � t j j j | d � � } t j | | � Wd
QX| S( ub
Retrieve the content of all buckets and cache the metadata to the buckets
cache file
u Refreshing buckets cache filec @ s� g d } } x� t r� t d d | d | d | d | d � d � d � d
t d � d � d
i | d 6� } g } x+ | D]# } d | k r� Pn | j | � qy W| j | � t g | D] } | j d d � d k ^ q� � r� Pn | d d } q W| S( Nu u s3.queryRJ RK RN R RL RM RO t
return_binRP RQ t paramsu markeru Keyu IsTruncatedu falsei����( RF t __utils__t Falset appendt extendt allR0 ( R RJ RK R. t markert tmpt headerst header( RQ RO RP RL RM ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt
__get_s3_meta� s.
1u Keyu Codeu Messageu '%s' response for bucket '%s'u2 S3 Error! Do you have any files in your S3 bucket?u /i i u Writing buckets cache fileu wN( R t debugRR R R R t _get_bucketsRe R t warningt KeyErrorR R R^ t splitt sett
startswithRd RF R( t removeR) R* R R2 t picklet dump( RZ RJ RK RN R Rl R R@ t bucket_files_listR& R t s3_metat kt
meta_responseR t environmentsR% t foundR< ( ( RQ RO RP RL RM s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyRY � s�
!!
)
%
9
,
#
c C@ st t j d � t j j j | d � �I } y t j | � } Wn, t j t t
t t t
f k
ri d } n XWd QX| S( u7
Return the contents of the buckets cache file
u Reading buckets cache fileu rbN( R Rm R) R* R R2 Ru R- t UnpicklingErrort AttributeErrort EOFErrort ImportErrort
IndexErrorRp R ( RZ R< R= ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyRW , s
c C@ s� g } i } x� | D]� } x� t j | � D]� \ } } g | D] } | d ^ q< } g | D] } | j d � sY | ^ qY } | | k r� t | | <| j i | | 6� q) x. | D]& } | | k r� | | c | 7<Pq� q� Wq) Wq W| S( uA
Looks for all the files in the S3 bucket cache metadata
u Keyu /( R R t endswithRF Re ( R R. R| t bucket_dictR& R= Ry t filepathsR ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyR = s
(
c
C@ s; g } i } x(| D] } xt j | � D]\ } } t � } xj g | D] } | d ^ qH D]K } d } x< | j d � d D]' }
| |
d } | j | � | } q| Wq\ W| | k r� t | | <| j i t | � | 6� q) xN | D]F } | | k r� | | c t | � 7<t t | | � � | | <Pq� q� Wq) Wq W| S( u�
Looks for all the directories in the S3 bucket cache metadata.
Supports trailing '/' keys (as created by S3 console) as well as
directories discovered in the path of file keys.
u Keyu u /i����( R R Rr Rq t addRF Re R (
R R. R| R� R& R= t dirpathsRy R t prefixt partt directoryR ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyRE T s(
$
c C@ s� | | k r | | n i } i } x' | D] } | | k r) | | } q) q) Wt t t d � | � � � } x] | D]U } d | k rq | d | k rq y | d j d � | d <Wn t k
r� n X| Sqq Wd S( uA
Looks for a file's metadata in the S3 bucket cache file
c S@ s
d | k S( Nu Key( ( Ry ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt <lambda>} s u Keyu ETagu "N( R R t stripRp ( R R& R R R t bucket_metaR t
files_metat item_meta( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt _find_file_metat s
c C@ s d t k r t d Si S( u*
Return the configuration buckets
u
s3.buckets( R" ( ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyRn � s c C@ s� t � \ } } } } } }
} } t j j | � rTt | | | | � }
|
rT|
d } | j d � d k r� | } t j j j | d � } | | k rNd SqQt j
| � } | j } t j j
| j � } t j j |
d d � } | t |
d � k rQ| | k rQt j d � t d
d | d | d
| d d d | d | d | d |
d t | � d | d t d | d | �
} | d k rNx� | d j � D]� \ } } | j � } | j � } t j | � j � d k r�t j j | d � } q�t j | � j � d k r�t | � } q�q�W| | k rK| | k rKt j d | | | � d SqNqQqTn t d
d | d | d
| d | d | d | d |
d t | � d | d | d | � d S( uw
Checks the local cache for the file, if it's old or missing go grab the
file from S3 and update the cache
u ETagu -i����u md5Nu LastModifiedu %Y-%m-%dT%H:%M:%S.%fZu Sizeui cached file size equal to metadata size and cached file mtime later than metadata last modification time.u s3.queryRJ RK RN t methodu HEADR RL RM RO R t
local_filet full_headersRP RQ u headersu
last-modifiedu %a, %d %b %Y %H:%M:%S %Zu content-lengthuW %s - %s : %s skipped download since cached file size equal to and mtime after s3 values( RR R R R( R� t findR) R* R+ R, t statt st_sizet datetimet
fromtimestampt st_mtimet strptimet intR Rm Rc t _quoteRF R t itemsR� R t text_typet lowerR
( R R R& R R RJ RK RL RM RN RO RP RQ t file_metat file_etagt file_md5t
cached_md5t cached_file_statt cached_file_sizet cached_file_mtimet cached_file_lastmodR. t header_namet header_valuet namet valuet
s3_file_mtimet s3_file_size( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyR � sx !
c C@ sO t � r d n
t | � d } | r+ d n d } g | D] } | | | !^ q8 S( uH
Return a list of file paths with the saltenv directory removed
i i����N( R R t len( t pathsR RD t env_lent slash_lent d( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyR1 � s c C@ s? t � } t | t � r t St | t � r/ t St d � � d S( u�
Return the configuration mode, either buckets per environment or a list of
buckets that have environment dirs in their root
u) Incorrect s3.buckets type given in configN( Rn t
isinstancet dictRF R Rd t
ValueError( R@ ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyR � s (7 t __doc__t
__future__R R R R� R RT Ru t loggingt salt.fileservert
fileserverR t salt.modulesR) t salt.utils.filest salt.utils.gzip_utilt salt.utils.hashutilst salt.utils.versionst salt.extR t salt.ext.six.movesR t salt.ext.six.moves.urllib.parseR R� t getLoggert __name__R RU RF R R
R R' R/ R> RB RC RI RR R R\ R RS RY RW R RE R� Rn R Rd R1 R ( ( ( s8 /usr/lib/python2.7/site-packages/salt/fileserver/s3fs.pyt <module>O sP , '
� R
Mini Shell