Current File : //lib/python2.7/site-packages/salt/states/boto_dynamodb.pyc
�
���^c @@ s| d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
m Z d d l j
j Z e j d e j d d d e j � e j � Z d e f d
� � YZ d � Z d d d d d d d d d d d d d d d d d
d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z! d � Z" d d d d d � Z# d S( u�
Manage DynamoDB Tables
======================
.. versionadded:: 2015.5.0
Create and destroy DynamoDB tables. Be aware that this interacts with Amazon's
services, and so may incur charges.
This module uses ``boto``, which can be installed via package, or pip.
This module accepts explicit DynamoDB credentials but can also utilize
IAM roles assigned to the instance through Instance Profiles. Dynamic
credentials are then automatically obtained from AWS API and no further
configuration is necessary. More information available `here
<http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html>`_.
If IAM roles are not used you need to specify them either in a pillar file or
in the minion's config file:
.. code-block:: yaml
keyid: GKTADJGHEIQSXMKKRBJ08H
key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
region: us-east-1
It's also possible to specify ``key``, ``keyid`` and ``region`` via a
profile, either passed in as a dict, or as a string to pull from
pillars or minion config:
.. code-block:: yaml
myprofile:
keyid: GKTADJGHEIQSXMKKRBJ08H
key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
region: us-east-1
.. code-block:: yaml
Ensure DynamoDB table does not exist:
boto_dynamodb.absent:
- table_name: new_table
- keyid: GKTADJGHEIQSXMKKRBJ08H
- key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
- region: us-east-1
Ensure DynamoDB table exists:
boto_dynamodb.present:
- table_name: new_table
- read_capacity_units: 1
- write_capacity_units: 2
- hash_key: primary_id
- hash_key_data_type: N
- range_key: start_timestamp
- range_key_data_type: N
- keyid: GKTADJGHEIQSXMKKRBJ08H
- key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
- region: us-east-1
- local_indexes:
- index:
- name: "primary_id_end_timestamp_index"
- hash_key: primary_id
- hash_key_data_type: N
- range_key: end_timestamp
- range_key_data_type: N
- global_indexes:
- index:
- name: "name_end_timestamp_index"
- hash_key: name
- hash_key_data_type: S
- range_key: end_timestamp
- range_key_data_type: N
- read_capacity_units: 3
- write_capacity_units: 4
It's possible to specify cloudwatch alarms that will be setup along with the
DynamoDB table. Note the alarm name will be defined by the name attribute
provided, plus the DynamoDB resource name.
.. code-block:: yaml
Ensure DynamoDB table exists:
boto_dynamodb.present:
- name: new_table
- read_capacity_units: 1
- write_capacity_units: 2
- hash_key: primary_id
- hash_key_data_type: N
- range_key: start_timestamp
- range_key_data_type: N
- alarms:
ConsumedWriteCapacityUnits:
name: 'DynamoDB ConsumedWriteCapacityUnits **MANAGED BY SALT**'
attributes:
metric: ConsumedWriteCapacityUnits
namespace: AWS/DynamoDB
statistic: Sum
comparison: '>='
# threshold_percent is used to calculate the actual threshold
# based on the provisioned capacity for the table.
threshold_percent: 0.75
period: 300
evaluation_periods: 2
unit: Count
description: 'DynamoDB ConsumedWriteCapacityUnits'
alarm_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ]
insufficient_data_actions: []
ok_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ]
- keyid: GKTADJGHEIQSXMKKRBJ08H
- key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
- region: us-east-1
You can also use alarms from pillars, and override values from the pillar
alarms by setting overrides on the resource. Note that 'boto_dynamodb_alarms'
will be used as a default value for all resources, if defined and can be
used to ensure alarms are always set for a resource.
Setting the alarms in a pillar:
.. code-block:: yaml
boto_dynamodb_alarms:
ConsumedWriteCapacityUnits:
name: 'DynamoDB ConsumedWriteCapacityUnits **MANAGED BY SALT**'
attributes:
metric: ConsumedWriteCapacityUnits
namespace: AWS/DynamoDB
statistic: Sum
comparison: '>='
# threshold_percent is used to calculate the actual threshold
# based on the provisioned capacity for the table.
threshold_percent: 0.75
period: 300
evaluation_periods: 2
unit: Count
description: 'DynamoDB ConsumedWriteCapacityUnits'
alarm_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ]
insufficient_data_actions: []
ok_actions: [ 'arn:aws:sns:us-east-1:1234:my-alarm' ]
Ensure DynamoDB table exists:
boto_dynamodb.present:
- name: new_table
- read_capacity_units: 1
- write_capacity_units: 2
- hash_key: primary_id
- hash_key_data_type: N
- range_key: start_timestamp
- range_key_data_type: N
- alarms:
ConsumedWriteCapacityUnits:
attributes:
threshold_percent: 0.90
period: 900
i ( t absolute_importt print_functiont unicode_literalsN( t sixt levelt formatu. %(asctime)s %(name)s %(levelname)s %(message)st streamt GsiNotUpdatableErrorc B@ s e Z d Z RS( u7 Raised when a global secondary index cannot be updated.( t __name__t
__module__t __doc__( ( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyR � s c C@ s d t k r d n t } | S( u2
Only load if boto_dynamodb is available.
u boto_dynamodb.existsu
boto_dynamodb( t __salt__t False( t ret( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyt __virtual__� s u boto_dynamodb_alarmsu boto_dynamodb_backup_configsc " C@ s� i | d 6t d 6d d 6i d 6} | rH d g | d <| | d <| } n g } i } i } t d | | | | | � } | s�t d r� d5 | d <d
j | � | d <| St d | | | | | | | |
| | |
| | �
} | st | d <d j | � | d <t | | | � | S| j d
j | � � | | d <| | d <| | d <|
| d <| | d <| | d <|
| d <| | d <| | d <n | j d j | � � t d | | | | | � } | j d i � j d i � } | j d � } | j d � } | | k o| | k } | s�t d r>d5 | d <| j d j | � � qt d | d i | d 6| d! 6d" | d# | d$ | d% | �} | s�t | d <d& j | � | d <t | | | � | S| j d' j | � � | f | d <| f | d <| f | d <| f | d <n | j d( j | � � | j d i � j d) g � } t | | | | | | | | | | �
} | d s�| j | d � | d | d <| d t k r�d* j
| � | d <t | | | � | Sn t | | | | | | | | | � } t j
| d | d � | d <| j | d � | d sa| d | d <| d t k rad* j
| � | d <t | | | � | Sn t j t d+ | g � � } x� | D]� } t d, | d- | d d. | d/ d0 | d1 d2 | d3 � }! |! d t d5 g k r2|! d | d <| j |! d � |! j d � rg|! j d � f | d d4 j | d � <qgq�d* j
| d |! d g � | d <t | | | � | Sq�Wd* j
| � | d <t | | | � | S(6 u�
Ensure the DynamoDB table exists. Table throughput can be updated after
table creation.
Global secondary indexes (GSIs) are managed with some exceptions:
- If a GSI deletion is detected, a failure will occur (deletes should be
done manually in the AWS console).
- If multiple GSIs are added in a single Salt call, a failure will occur
(boto supports one creation at a time). Note that this only applies after
table creation; multiple GSIs can be created during table creation.
- Updates to existing GSIs are limited to read/write capacity only
(DynamoDB limitation).
name
Name of the DynamoDB table
table_name
Name of the DynamoDB table (deprecated)
region
Region to connect to.
key
Secret key to be used.
keyid
Access key to be used.
profile
A dict with region, key and keyid, or a pillar key (string)
that contains a dict with region, key and keyid.
read_capacity_units
The read throughput for this table
write_capacity_units
The write throughput for this table
hash_key
The name of the attribute that will be used as the hash key
for this table
hash_key_data_type
The DynamoDB datatype of the hash key
range_key
The name of the attribute that will be used as the range key
for this table
range_key_data_type
The DynamoDB datatype of the range key
local_indexes
The local indexes you would like to create
global_indexes
The global indexes you would like to create
backup_configs_from_pillars
Pillars to use to configure DataPipeline backups
u nameu resultu u commentu changesuM boto_dynamodb.present: `table_name` is deprecated. Please use `name` instead.u warningsu boto_dynamodb.existsu testu$ DynamoDB table {0} would be created.u boto_dynamodb.create_tableu Failed to create table {0}u+ DynamoDB table {0} was successfully createdu tableu read_capacity_unitsu write_capacity_unitsu hash_keyu hash_key_data_typeu range_keyu range_key_data_typeu
local_indexesu global_indexesu DynamoDB table {0} existsu boto_dynamodb.describeu Tableu ProvisionedThroughputu WriteCapacityUnitsu ReadCapacityUnitsu( DynamoDB table {0} is set to be updated.u boto_dynamodb.updatet
throughputu readu writet regiont keyt keyidt profileu Failed to update table {0}u+ DynamoDB table {0} was successfully updatedu% DynamoDB table {0} throughput matchesu GlobalSecondaryIndexesu ,
u
pillar.gett namet
schedule_namet periodu periodt utc_houru utc_hourt s3_base_locationu s3_base_locationu backup_datapipeline_{0}N( t TrueR t __opts__t NoneR R t _add_changest appendt gett _global_indexes_presentt joint _alarms_presentt
dictupdatet updatet copyt deepcopyt# _ensure_backup_datapipeline_present(" R t
table_nameR R R R t read_capacity_unitst write_capacity_unitst alarmst alarms_from_pillart hash_keyt hash_key_data_typet range_keyt range_key_data_typet
local_indexest global_indexest backup_configs_from_pillarsR
t commentst changes_oldt changes_newt table_existst
is_createdt descriptiont provisioned_throughputt current_write_capacity_unitst current_read_capacity_unitst throughput_matchest
is_updatedt provisioned_indexest _rett datapipeline_configst configt datapipeline_ret( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyt present� s� Q"
-!c C@ s2 | r | | d d <n | r. | | d d <n d S( Nu changesu oldu new( ( R
R4 R5 ( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyR � s c
C@ s* i t d 6}
| r, t d � | D� � } n i } t | j � � } i }
| r� x� | D]� } | j � d } d
} x6 | D]. } | j � d g k rz | j � d } qz qz W| s� t |
d <d j | � |
d <|
S| |
| <qW Wn t | t |
j � � � \ } } } | r8t |
d <d j d j | � � |
d <|
St
| � d k rkt |
d <d
j | � |
d <|
S| r�t t | � � } t
|
| | | | | |
| | | | � |
d s�|
Sn | r�t |
| | | | | |
| | | | | � |
d s�|
Sn d | k r&d | k r&| j d � n |
S( u; Handles global secondary index for the table present state.u resultc s@ s | ] } | d | f Vq d S( u IndexNameN( ( t .0t index( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pys <genexpr>� s i u nameu" Index name not found for table {0}u commentuT Deletion of GSIs ({0}) is not supported! Please do this manually in the AWS console.u , i uj Creation of multiple GSIs ({0}) is not supported due to API limitations. Please create them one at a time.u global_indexesu" All global secondary indexes matchN( R t dictt sett keyst valuesR R R t _partition_index_namesR t lent nextt itert _add_global_secondary_indext _update_global_secondary_indexesR ( R> R1 R4 R5 R3 R R R R R R
t provisioned_gsi_configt provisioned_index_namest
gsi_configRE t index_configt
index_namet entryt existing_index_namest new_index_namest index_names_to_be_deleted( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyR � sZ
!
c C@ sc t � } t � } x7 | D]/ } | | k r; | j | � q | j | � q W| | } | | | f S( uO Returns 3 disjoint sets of indexes: existing, to be created, and to be deleted.( RG t add( RQ t index_namesRV RW R RX ( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyRJ � s
c C@ s� t d r. d | d <d j | | � | d <d S| j d i � t d | t d | | d t �d
| d | d | d
|
�} | r� | j d j | � � | | | d | <n t | d <d j | � | d <d S( u4 Updates ret iff there was a failure or in test mode.u testu resultu+ Dynamo table {0} will have a GSI added: {1}u commentNu global_indexesu+ boto_dynamodb.create_global_secondary_indexu boto_dynamodb.extract_indext global_indexR R R R u Created GSI {0}u Failed to create GSI {0}( R R R t
setdefaultR R R R ( R
R RT R4 R5 R3 RR R R R R t success( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyRN � s&
c C@ sV y t | | | � \ } }
Wn1 t k
rO } t | d <t j | � | d <d SX|
rRt d r� d | d <d j | d j |
j � � � | d <d S| j
d i � | j
d i � t d | |
d | d
| d |
d | �} | r2| j d
j |
� � xR |
D]* } | | | d | <|
| | d | <qWqRt | d <d j |
� | d <n d S( u4 Updates ret iff there was a failure or in test mode.u resultu commentNu testu, Dynamo table {0} will have GSIs updated: {1}u , u global_indexesu+ boto_dynamodb.update_global_secondary_indexR R R R u% Updated GSIs with new throughputs {0}u$ Failed to update GSI throughputs {0}(
t _determine_gsi_updatesR R R t text_typeR R R R RH R\ R R ( R
R4 R5 R3 RV RP RR R R R R R t provisioned_throughputst
index_updatest eR] RT ( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyRO s<
c C@ s� i } i } x�| D]�} | | } t d | | d t �j � } x� | D]� } | | k rJ | d k rJ | | } | | }
| d k r� | d |
d k r� t d � � qt | j d g � � t |
j d g � � k rt d � � qq| |
k rt d j | | � � � qqJ qJ W| j d � } | j d
� } | j d � }
i | d 6|
d
6| | <| j d � } | j d
� } | j d � } | | k s�|
| k r i | d 6| d
6| | <q q W| | f S( Nu boto_dynamodb.extract_indexR[ u ProvisionedThroughputu
Projectionu ProjectionTypeu! GSI projection types do not matchu NonKeyAttributesu0 NonKeyAttributes do not match for GSI projectionu0 GSI property {0} cannot be updated for index {1}u ReadCapacityUnitsu WriteCapacityUnitsu readu write( R R t schemaR RG R R ( RV RP RR R` Ra RT t current_configt
new_configR t new_valuet
current_valuet current_throughputt current_readt
current_writet new_throughputt new_readt new_write( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyR^ B s8
0 c C@ sb t j t d | i � � } | r7 t j | | � } n i | d 6t d 6d d 6i d 6}
xt j | � D]�\ } } | d | d | d <| d | d d | d d <i | g d
6| d d <| d d d
k r2d | d k r2t j | | d d � | d d <| d d =| d d c | d d 9<n | d d d k r�d | d k r�t j | | d d � | d d <| d d =| d d c | d d 9<n i | d d 6| d d 6| d 6| d 6| d 6| d 6}
t
d |
� } | d s| d |
d <n | j d i � i k r7| d |
d | d <n d | k ri |
d c | d 7<qi qi W|
S( uA helper method for present. ensure that cloudwatch_alarms are setu
config.optionu nameu resultu u commentu changesu u
attributesu descriptionu TableNameu
dimensionsu metricu ConsumedWriteCapacityUnitsu thresholdu threshold_percentu periodu ConsumedReadCapacityUnitsu regionu keyu keyidu profileu boto_cloudwatch_alarm.present( R$ R% R R" R# R R t iteritemst matht ceilt
__states__R ( R R* R+ R) R( R R R R t tmpt merged_return_valuet _t infot kwargst results( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyR! i sD " # #
c C@ s� i d j | | � d 6i i | d 6i | d 6d d 6t | | � j � d 6d 6d 6d 6i | d
6d j | | � d 6d
6} t d | � S( Nu {0}-{1}-backupu nameu periodu Scheduleu typeu
startDateTimeu fieldsu DefaultScheduleu pipeline_objectsu myDDBTableNameu {0}/{1}/u
myOutputS3Locu parameter_valuesu boto_datapipeline.present( R t _next_datetime_with_utc_hourt isoformatRq ( R R R R R Rv ( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyR& � s "c C@ s t | � | S( uV
For a given table_name, returns hash of the table_name limited by max_value.
( t hash( R' t max_value( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyt' _get_deterministic_value_for_table_name� s c C@ s� t j j � } t j d | j d | j d | j d | d t | d � d t | d � � } | t j j � k r� t j d d � } | | 7} n | S(
ux
Datapipeline API is throttling us, as all the pipelines are started at the same time.
We would like to uniformly distribute the startTime over a 60 minute window.
Return the next future utc datetime where
hour == utc_hour
minute = A value between 0-59 (depending on table name)
second = A value between 0-59 (depending on table name)
t yeart montht dayt hourt minutei< t secondt daysi ( t datetimet datet todayR} R~ R R| t utcnowt timedelta( R' R R� t start_date_timet one_day( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyRx � s
c C@ s i | d 6t d 6d d 6i d 6} t d | | | | | � } | s[ d j | � | d <| St d r� d j | � | d <d | d <| St d
| | | | | � } | r� d j | � | d <| d j d d
j | � � | d j d d j | � � n d j | � | d <t | d <| S( ug
Ensure the DynamoDB table does not exist.
name
Name of the DynamoDB table.
region
Region to connect to.
key
Secret key to be used.
keyid
Access key to be used.
profile
A dict with region, key and keyid, or a pillar key (string)
that contains a dict with region, key and keyid.
u nameu resultu u commentu changesu boto_dynamodb.existsu! DynamoDB table {0} does not existu testu' DynamoDB table {0} is set to be deletedu boto_dynamodb.deleteu Deleted DynamoDB table {0}u oldu Table {0} existsu newu Table {0} deletedu# Failed to delete DynamoDB table {0}N( R R R R R R\ R ( R R R R R R
t existst
is_deleted( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyt absent� s, "
($ R
t
__future__R R R R� Ro t syst loggingR$ t salt.extR t salt.utils.dictupdatet utilsR" t basicConfigt INFOt stdoutt getLoggert logt ExceptionR R R RC R R RJ RN RO R^ R! R& R| Rx R� ( ( ( s= /usr/lib/python2.7/site-packages/salt/states/boto_dynamodb.pyt <module>� s\
� ?
( ' 6
Mini Shell