Current File : //lib/python2.7/site-packages/salt/states/boto_datapipeline.pyo
�
���^c @@ s6 d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l
Z d d l m Z d d l
m Z d � Z d d d d d d d d d d d
�
Z d � Z d � Z d
� Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d d � Z d d d d d � Z d S( u�
Manage Data Pipelines
.. versionadded:: 2016.3.0
Be aware that this interacts with Amazon's services, and so may incur charges.
This module uses ``boto3``, which can be installed via package, or pip.
This module accepts explicit AWS credentials but can also utilize
IAM roles assigned to the instance through Instance Profiles. Dynamic
credentials are then automatically obtained from AWS API and no further
configuration is necessary. More information available `here
<http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html>`_.
If IAM roles are not used you need to specify them either in a pillar file or
in the minion's config file:
.. code-block:: yaml
datapipeline.keyid: GKTADJGHEIQSXMKKRBJ08H
datapipeline.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
It's also possible to specify ``key``, ``keyid`` and ``region`` via a profile,
either passed in as a dict, or as a string to pull from pillars or minion
config:
.. code-block:: yaml
myprofile:
keyid: GKTADJGHEIQSXMKKRBJ08H
key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
region: us-east-1
.. code-block:: yaml
Ensure daily data pipeline exists:
boto_datapipeline.present:
- name: my-datapipeline
- pipeline_objects:
DefaultSchedule:
name: Every 1 day
fields:
period: 1 Day
type: Schedule
startAt: FIRST_ACTIVATION_DATE_TIME
- parameter_values:
myDDBTableName: my-dynamo-table
i ( t absolute_importt print_functiont unicode_literalsN( t six( t zipc C@ s d t k r d St S( u)
Only load if boto is available.
u! boto_datapipeline.create_pipelineu boto_datapipeline( t __salt__t False( ( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyt __virtual__A s u" boto_datapipeline_pipeline_objectsu# boto_datapipeline_parameter_objectsu" boto_datapipeline_parameter_valuesc C@ s i | d 6t d 6d d 6i d 6} | p+ i } | p7 i } | pC i } t | t | | � t | | � t | | � d | d | d | d |
�\ } }
| r� d
j | � | d <| St d r� d j | � | d <d | d <| St d
| | d | d | d | d |
�} d | k r;t | d <d j | | d � | d <| S| d } t d | t | | � d t | | � d t | | � d | d | d | d |
�} d | k r t
| � r�t d | d | d | d | d |
�} d | k r t | d <d j | | d � | d <| St d
| | d | d | d | d |
�} d | k rjt | d <d j | | d � | d <| S| d } t d | t | | � d t | | � d t | | � d | d | d | d |
�} n d | k r t | d <d j | | d � | d <| Sn t d | d | d | d | d |
�} d | k r\t | d <d j | | d � | d <| St d | d d d | d | d | d |
�} d | k r�i } n t | d � } |
s�d | d d <d j | � | d <n* t |
| � | d d <d j | � | d <| S( u�
Ensure the data pipeline exists with matching definition.
name
Name of the service to ensure a data pipeline exists for.
pipeline_objects
Pipeline objects to use. Will override objects read from pillars.
pipeline_objects_from_pillars
The pillar key to use for lookup.
parameter_objects
Parameter objects to use. Will override objects read from pillars.
parameter_objects_from_pillars
The pillar key to use for lookup.
parameter_values
Parameter values to use. Will override values read from pillars.
parameter_values_from_pillars
The pillar key to use for lookup.
region
Region to connect to.
key
Secret key to be used.
keyid
Access key to be used.
profile
A dict with region, key and keyid, or a pillar key (string)
that contains a dict with region, key and keyid.
u nameu resultu u commentu changest regiont keyt keyidt profileu AWS data pipeline {0} presentu testu1 Data pipeline {0} is set to be created or updatedu! boto_datapipeline.create_pipelineu erroru' Failed to create data pipeline {0}: {1}u) boto_datapipeline.put_pipeline_definitiont parameter_objectst parameter_valuesu! boto_datapipeline.delete_pipelineu' Failed to delete data pipeline {0}: {1}u# boto_datapipeline.activate_pipelineu) boto_datapipeline.get_pipeline_definitiont versionu activeu Pipeline created.u newu Data pipeline {0} createdu diffu Data pipeline {0} updatedN(
t Truet! _pipeline_present_with_definitiont _pipeline_objectst _parameter_objectst _parameter_valuest formatt __opts__t NoneR R t _immutable_fields_errort _standardizet _diff( t namet pipeline_objectst pipeline_objects_from_pillarsR t parameter_objects_from_pillarsR
t parameter_values_from_pillarsR R R
R t rett presentt old_pipeline_definitiont result_create_pipelinet pipeline_idt result_pipeline_definitiont result_delete_pipelinet result_activate_pipelinet pipeline_definition_resultt new_pipeline_definition( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR H s� -"
c C@ s> x7 | d D]+ } x" | d D] } d | k r t Sq Wq Wt S( u Return true if update pipeline failed due to immutable fields
Some fields cannot be changed after a pipeline has been activated.
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-manage-pipeline-modify-console.html#dp-edit-pipeline-limits
u erroru errorsu can not be changed( R R ( R$ t et e2( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR � s
c C@ s t d | d | d | d | d | �} d | k r> t i f S| d } t d | d d
d | d | d | d | �}
d |
k r� t i f St |
d � } | j d � } | j d � }
| j d
� } t t | � t | � � o� t |
| � o� t | | � } | | f S( u�
Return true if the pipeline exists and the definition matches.
name
The name of the pipeline.
expected_pipeline_objects
Pipeline objects that must match the definition.
expected_parameter_objects
Parameter objects that must match the definition.
expected_parameter_values
Parameter values that must match the definition.
region
Region to connect to.
key
Secret key to be used.
keyid
Access key to be used.
profile
A dict with region, key and keyid, or a pillar key (string)
that contains a dict with region, key and keyid.
u' boto_datapipeline.pipeline_id_from_nameR R R
R u erroru resultu) boto_datapipeline.get_pipeline_definitionR u activeu pipelineObjectsu parameterObjectsu parameterValues( R R R t gett _recursive_comparet _cleaned( R t expected_pipeline_objectst expected_parameter_objectst expected_parameter_valuesR R R
R t result_pipeline_idR# R' t pipeline_definitionR R R
R ( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR s4
c C@ s� t j | � } x{ | D]s } | d d k r xZ | d D]K } | d d k r7 | d } t j j | d � } | j d � | d <q7 q7 Wq q W| S( u� Return standardized pipeline objects to be used for comparing
Remove year, month, and day components of the startDateTime so that data
pipelines with the same time of day but different days are considered
equal.
u idu DefaultScheduleu fieldsu keyu
startDateTimeu stringValueu %Y-%m-%dT%H:%M:%Su %H:%M:%S( t copyt deepcopyt datetimet strptimet strftime( R R t pipeline_objectt field_objectt start_date_time_stringt start_date_time( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR- N s
!c C@ s- t | t � r� | d k r$ g } n t | � t | � k r@ t S| j d t � | j d t � x0 t | | � D] \ } } t | | � sp t Sqp Wt St | t
� r | d k r� i } n t
| � } t
| � } t | � t | � k r� t Sx) | D]! } t | | | | � s� t Sq� Wt S| | k Sd S( u<
Return v1 == v2. Compares list, dict, recursively.
R N( t
isinstancet listR t lenR t sortt
_id_or_keyR R, R t dictt sorted( t v1t v2t xt yt k( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR, a s.
c C@ s> t | t � r: d | k r# | d Sd | k r: | d Sn | S( u0
Return the value at key 'id' or 'key'.
u idu key( R< RA ( t list_item( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR@ s c C@ s� | j d d � | j d d � t j j j t j t j j j | d d �j
t � t j j j | d d �j
t � � � } d j | � S( u5
Return string diff of pipeline definitions.
u ResponseMetadatat indenti u N(
t popR t saltt utilst datat decodet difflibt unified_difft jsont dumpst
splitlinesR t join( R! R( t diff( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR � s !*c @ s, � f d � � t j | � } � | � | S( u�
Return standardized format for lists/dictionaries.
Lists of dictionaries are sorted by the value of the dictionary at
its primary key ('id' or 'key'). OrderedDict's are converted to
basic dictionaries.
c @ s� t | t � r= | j d t � xm | D] } � | � q& WnR t | t � r� t | � } x4 t j | � D] \ } } � | � � | � qh Wn d S( NR ( R< R= R? R@ RA R t iteritems( t structuret eachRG t v( t mutating_helper( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyRZ � s
( R3 R4 ( RW t
new_structure( ( RZ sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR � s
c C@ sa t j t d | � � } | j | � t t | � � } x" | D] } t | d � | d <q? W| S( u�
Return a list of pipeline objects that compose the pipeline
pipeline_objects_from_pillars
The pillar key to use for lookup
pipeline_object_overrides
Pipeline objects to use. Will override objects read from pillars.
u
pillar.getu fields( R3 R4 R t updateR t _dict_to_list_idst _properties_from_dict( R t pipeline_object_overridest from_pillarsR R8 ( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR � s
c C@ sa t j t d | � � } | j | � t t | � � } x" | D] } t | d � | d <q? W| S( u
Return a list of parameter objects that configure the pipeline
parameter_objects_from_pillars
The pillar key to use for lookup
parameter_object_overrides
Parameter objects to use. Will override objects read from pillars.
u
pillar.getu
attributes( R3 R4 R R\ R R] R^ ( R t parameter_object_overridesR` R t parameter_object( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR � s
c C@ sB t j t d | � � } | j | � t | � } t | d d �S( u
Return a dictionary of parameter values that configure the pipeline
parameter_values_from_pillars
The pillar key to use for lookup
parameter_value_overrides
Parameter values to use. Will override values read from pillars.
u
pillar.gett key_nameu id( R3 R4 R R\ R R^ ( R t parameter_value_overridesR` R
( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR � s
c C@ sQ g } xD t j | � D]3 \ } } i | d 6} | j | � | j | � q W| S( u�
Convert a dictionary to a list of dictionaries, where each element has
a key value pair {'id': key}. This makes it easy to override pillar values
while still satisfying the boto api.
u id( R RV R\ t append( t objectst
list_with_idsR t valuet element( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR] � s
u keyc C@ sv g } xi t j | � D]X \ } } t | t � rS | j i | | 6| d d 6� q | j i | | 6| d 6� q W| S( u�
Transforms dictionary into pipeline object properties.
The output format conforms to boto's specification.
Example input:
{
'a': '1',
'b': {
'ref': '2'
},
}
Example output:
[
{
'key': 'a',
'stringValue': '1',
},
{
'key': 'b',
'refValue': '2',
},
]
u refu refValueu stringValue( R RV R< RA Re ( t dRc t fieldsR Rh ( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyR^ � s c
C@ s� i | d 6t d 6d d 6i d 6} t d | d | d | d | d
| �} d | k r� | d } t d r� d
j | � | d <d | d <| St d | d | d | d | d
| �i | d 6| d d <d | d d <n d j | � | d <| S( u�
Ensure a pipeline with the service_name does not exist
name
Name of the service to ensure a data pipeline does not exist for.
region
Region to connect to.
key
Secret key to be used.
keyid
Access key to be used.
profile
A dict with region, key and keyid, or a pillar key (string)
that contains a dict with region, key and keyid.
u nameu resultu u commentu changesu' boto_datapipeline.pipeline_id_from_nameR R R
R u erroru testu$ Data pipeline {0} set to be deleted.u! boto_datapipeline.delete_pipelineu pipeline_idu oldu newu AWS data pipeline {0} absent.N( R R R R R ( R R R R
R R R1 R# ( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyt absent s. "
( t __doc__t
__future__R R R R3 R5 RO t salt.utils.dataRK t salt.utils.jsont salt.extR t salt.ext.six.movesR R R R R R R- R, R@ R R R R R R] R^ Rl ( ( ( sA /usr/lib/python2.7/site-packages/salt/states/boto_datapipeline.pyt <module>2 s< � B )
Mini Shell