Current File : //lib/python2.7/site-packages/salt/pillar/s3.pyo
�
���^c @@ sd d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l m
Z
d d l m Z d d l
m Z d d l m Z d d l m Z d d l Z d d l Z e j e � Z d e f d
� � YZ d d e d e d d d d d
e e e d �
Z d � Z d � Z! d � Z" d � Z# d � Z$ d � Z% d � Z& d � Z' d � Z( d S( u�
Copy pillar data from a bucket in Amazon S3
The S3 pillar can be configured in the master config file with the following
options
.. code-block:: yaml
ext_pillar:
- s3:
bucket: my.fancy.pillar.bucket
keyid: KASKFJWAKJASJKDAJKSD
key: ksladfDLKDALSFKSD93q032sdDasdfasdflsadkf
multiple_env: False
environment: base
prefix: somewhere/overthere
verify_ssl: True
service_url: s3.amazonaws.com
kms_keyid: 01234567-89ab-cdef-0123-4567890abcde
s3_cache_expire: 30
s3_sync_on_update: True
path_style: False
https_enable: True
The ``bucket`` parameter specifies the target S3 bucket. It is required.
The ``keyid`` parameter specifies the key id to use when access the S3 bucket.
If it is not provided, an attempt to fetch it from EC2 instance meta-data will
be made.
The ``key`` parameter specifies the key to use when access the S3 bucket. If it
is not provided, an attempt to fetch it from EC2 instance meta-data will be made.
The ``multiple_env`` defaults to False. It specifies whether the pillar should
interpret top level folders as pillar environments (see mode section below).
The ``environment`` defaults to 'base'. It specifies which environment the
bucket represents when in single environments mode (see mode section below). It
is ignored if multiple_env is True.
The ``prefix`` defaults to ''. It specifies a key prefix to use when searching
for data in the bucket for the pillar. It works when multiple_env is True or False.
Essentially it tells ext_pillar to look for your pillar data in a 'subdirectory'
of your S3 bucket
The ``verify_ssl`` parameter defaults to True. It specifies whether to check for
valid S3 SSL certificates. *NOTE* If you use bucket names with periods, this
must be set to False else an invalid certificate error will be thrown (issue
#12200).
The ``service_url`` parameter defaults to 's3.amazonaws.com'. It specifies the
base url to use for accessing S3.
The ``kms_keyid`` parameter is optional. It specifies the ID of the Key
Management Service (KMS) master key that was used to encrypt the object.
The ``s3_cache_expire`` parameter defaults to 30s. It specifies expiration
time of S3 metadata cache file.
The ``s3_sync_on_update`` parameter defaults to True. It specifies if cache
is synced on update rather than jit.
The ``path_style`` parameter defaults to False. It specifies whether to use
path style requests or dns style requests
The ``https_enable`` parameter defaults to True. It specifies whether to use
https protocol or http protocol
This pillar can operate in two modes, single environment per bucket or multiple
environments per bucket.
Single environment mode must have this bucket structure:
.. code-block:: text
s3://<bucket name>/<prefix>/<files>
Multiple environment mode must have this bucket structure:
.. code-block:: text
s3://<bucket name>/<prefix>/<environment>/<files>
If you wish to define your pillar data entirely within S3 it's recommended
that you use the `prefix=` parameter and specify one entry in ext_pillar
for each environment rather than specifying multiple_env. This is due
to issue #22471 (https://github.com/saltstack/salt/issues/22471)
i ( t absolute_importt print_functiont unicode_literalsN( t deepcopy( t six( t filter( t quote( t Pillart
S3Credentialsc B@ s e Z e d d e e d � Z RS( c
C@ sU | | _ | | _ | | _ | | _ | | _ | | _ | | _ | | _ | | _ d S( N( t keyt keyidt kms_keyidt buckett service_urlt
verify_sslt locationt
path_stylet https_enable(
t selfR R
R R
R R R R R ( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt __init__t s N( t __name__t
__module__t Truet Nonet FalseR ( ( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR s s u baseu i c
C@ s t | | | |
| | | | | � } t j j t j j t � | | � � } | ru t j j t j j | | � � } n t d j | g � | g k r� i St | | | | | | � } |
rnt j
d � x� t j | � D]� \ } } xu t j t
| � � D]^ \ } } xO | D]G } t | | | � } t j
d | | | � t | | | | | | � q Wq� Wq� Wt j
d � n t t � } | r�t j j | | � g n | g | d | <g | d D] } d | k r�| ^ q�| d <t | t | | � } | j d t � } | S( u7
Execute a command and read the output as YAML
u pillar_rootsu% Syncing local pillar cache from S3...u %s - %s : %su* Sync local pillar cache from S3 completed.u
ext_pillaru s3t ext( R t ost patht normpatht joint _get_cache_dirt __opts__t gett _initt logt infoR t iteritemst _find_filest _get_cached_file_namet _get_file_from_s3R R t
__grains__t compile_pillarR ( t minion_idt pillarR R R
R R t multiple_envt environmentt prefixR
R t s3_cache_expiret s3_sync_on_updateR R t s3_credst
pillar_dirt metadatat saltenvt env_metat filest file_patht cached_file_patht optst xt pilt compiled_pillar( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt
ext_pillar� s4 $
"
/-c C@ s� t | | � } t j � | } t j j | � rF t j j | � } n d } | | k } t j d | | rp d n d | | | � | r� t | | | | | � }
n t | � }
t j d |
� |
S( ux
Connect to S3 and download the metadata for each file in all buckets
specified and cache the data to disk.
i uD S3 bucket cache file %s is %sexpired, mtime_diff=%ss, expiration=%ssu u not u S3 bucket retrieved pillars %s(
t _get_buckets_cache_filenamet timeR R t isfilet getmtimeR" t debugt _refresh_buckets_cache_filet _read_buckets_cache_file( t credsR R, R- R. R/ t
cache_filet expt cache_file_mtimet expiredt pillars( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR! � s$ c C@ sL t j j t d d � } t j j | � sH t j d � t j | � n | S( uI
Get pillar cache directory. Initialize it if it does not exist.
u cachediru pillar_s3fsu Initializing S3 Pillar Cache( R R R R t isdirR" RB t makedirs( t cache_dir( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR � s
c C@ s\ t j j t � | | | � } t j j t j j | � � sX t j t j j | � � n | S( u<
Return the cached file name for a bucket path file
( R R R R t existst dirnameRL ( R R4 R R7 ( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR& � s c C@ sJ t � } t j j | � s+ t j | � n t j j | d j | | � � S( ui
Return the filename of the cache for bucket contents.
Create the path if it does not exist.
u {0}-{1}-files.cache( R R R RN RL R t format( R R. RM ( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR> s c @ s� � � f d � } d � } d � } t j d � i } � j } | s� t j d � i }
| � } | rJ| | � |
| <|
| | <qJn� t j d � | � } | rJ| | � } | | � }
x� |
D]� } g | D] } | d j | � r� | ^ q� } | | k ri | | <n | | | k r/g | | | <n | | | c | 7<q� Wn t j j | � rlt j | � n t j d � t j j
j | d � � } t j
| | � Wd
QX| S( ub
Retrieve the content of all buckets and cache the metadata to the buckets
cache file
c @ so t d d � j d � j d � j d � j d � j d � j d � j d t d
i � d 6d � j d
� j
� S( Nu s3.queryR R
R R R
R R t
return_bint paramsu prefixR R ( t __utils__R R
R R R
R R R R R ( ( RE R. ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt
__get_s3_meta s
c S@ s# g | D] } d | k r | ^ q S( Nu Key( ( t s3_metat k( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt __get_pillar_files_from_s3_meta# s c S@ sC g | D], } t j j | d � j d d � d ^ q } t | � S( Nu Keyu /i i ( R R RO t splitt set( R6 RV t environments( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt __get_pillar_environments' s 9u' Refreshing S3 buckets pillar cache fileu" Single environment per bucket modeu$ Multiple environment per bucket modeu Keyu$ Writing S3 buckets pillar cache fileu wN( R" RB R t
startswithR R R@ t removet saltt utilsR6 t fopent picklet dump( RE RF R, R- R. RT RW R[ R3 R t bucket_filesRU R6 RZ R4 RV t env_filest fp_( ( RE R. s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyRC
s>
,
c C@ sA t j d � t j j j | d � � } t j | � } Wd QX| S( u7
Return the contents of the buckets cache file
u Reading buckets cache fileu rbN( R" RB R^ R_ R6 R` Ra t load( RF Re t data( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyRD ` s
c C@ s� i } x� t j | � D]t \ } } | | k r; g | | <n g | D] } | d ^ qB } | | c g | D] } | j d � si | ^ qi 7<q W| S( uA
Looks for all the files in the S3 bucket cache metadata
u Keyu /( R R$ t endswith( R3 t retR Rg RV t filePaths( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR% m s
6c C@ s� | | k r | | n i } | | k r2 | | n i } t t t d � | � � � } x. | D]& } d | k r] | d | k r] | Sq] Wd S( uA
Looks for a file's metadata in the S3 bucket cache file
c S@ s
d | k S( Nu Key( ( RV ( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt <lambda>� s u KeyN( t listR ( R3 R R4 R R5 t bucket_metat
files_metat item_meta( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt _find_file_meta� s
c C@ s t j j | � r� t | | | | � } | rR d j t t t j | d � � � n d } t
j j j
| d � } t j d | | | � | | k r� d Sn t d d | j d | j d | j d
| d | j d t | � d
| d | j d | j d | j d | j � d S( uw
Checks the local cache for the file, if it's old or missing go grab the
file from S3 and update the cache
u u ETagu md5u% Cached file: path=%s, md5=%s, etag=%sNu s3.queryR R
R R R
R t
local_fileR R R R ( R R R@ Rp R Rl R t strt isalnumR R^ R_ t hashutilst get_hashR" RB RS R R
R R
t _quoteR R R R ( RE R3 R4 R R R8 t file_metat file_md5t
cached_md5( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyR' � s( 1
() t __doc__t
__future__R R R t loggingR R? Ra t copyR t salt.extR t salt.ext.six.movesR t salt.ext.six.moves.urllib.parseR Rv t salt.pillarR t salt.utils.filesR^ t salt.utils.hashutilst getLoggerR R" t objectR R R R R= R! R R& R> RC RD R% Rp R' ( ( ( s2 /usr/lib/python2.7/site-packages/salt/pillar/s3.pyt <module>Y sH 0 $
S
Mini Shell