Current File : //usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyc
�
���^c @@ s d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l m
Z
d d l Z y; d d l
Z
d d l Z e j d � j e j � e Z Wn e k
r� e Z n Xe j e � Z d Z d � Z d � Z d d d d d � Z d d d d d
� Z d d d d d � Z d d d d d � Z d d d d d
� Z d d d d d � Z! d d d d d � Z" d d d d d � Z# d d d d d � Z$ d � Z% d � Z& e d d d d d � Z' d d d d d � Z( d � Z) d � Z* d � Z+ d S( u�
Connection module for Amazon Kinesis
.. versionadded:: 2017.7.0
:configuration: This module accepts explicit Kinesis credentials but can also
utilize IAM roles assigned to the instance trough Instance Profiles.
Dynamic credentials are then automatically obtained from AWS API and no
further configuration is necessary. More Information available at:
.. code-block:: text
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
If IAM roles are not used you need to specify them either in a pillar or
in the minion's config file:
.. code-block:: yaml
kinesis.keyid: GKTADJGHEIQSXMKKRBJ08H
kinesis.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
A region may also be specified in the configuration:
.. code-block:: yaml
kinesis.region: us-east-1
If a region is not specified, the default is us-east-1.
It's also possible to specify key, keyid and region via a profile, either
as a passed in dict, or as a string to pull from pillars or minion config:
.. code-block:: yaml
myprofile:
keyid: GKTADJGHEIQSXMKKRBJ08H
key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
region: us-east-1
:depends: boto3
i ( t absolute_importt print_functiont unicode_literalsN( t rangeu boto3u boto_kinesisc C@ s7 t j j j � } | t k r3 t d t d � t S| S( u-
Only load if boto3 libraries exist.
u boto3.assign_funcsu kinesis( t saltt utilst versionst check_boto_reqst Truet __utils__t __name__t __virtualname__( t
has_boto_reqs( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt __virtual__K s
c C@ s t | d d | �S( u�
Stream info from AWS, via describe_stream
Only returns the first "page" of shards (up to 100); use _get_full_stream() for all shards.
CLI example::
salt myminion boto_kinesis._get_basic_stream my_stream existing_conn
u describe_streamt
StreamName( t _execute_with_retries( t stream_namet conn( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt _get_basic_streamV s c C@ s� t d | d | d | d | � } i } t | | � d } | } xc | d d r� t | d d | d
| d d d d
�} | d } | d d c | d d 7<qC W| | d <| S( u�
Get complete stream info from AWS, via describe_stream, including all shards.
CLI example::
salt myminion boto_kinesis._get_full_stream my_stream region=us-east-1
t regiont keyt keyidt profileu resultu StreamDescriptionu
HasMoreShardsu describe_streamR t ExclusiveStartShardIdu Shardsi����u ShardId( t _get_connR R ( R R R R R R t rt streamt full_stream( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt _get_full_streamb s !
c
C@ s� t d | d | d | d | � } d } d } d } xa | d k r� t j t | | � � | d 7} t | | � } d | k r� | S| d
d d } q6 W| d
d d
r� t | | | | | � } n | S( u^
Get complete stream info from AWS, returning only when the stream is in the ACTIVE state.
Continues to retry when stream is updating or creating.
If the stream is deleted during retries, the loop will catch the error and break.
CLI example::
salt myminion boto_kinesis.get_stream_when_active my_stream region=us-east-1
R R R R i i
u ACTIVEi u erroru resultu StreamDescriptionu StreamStatusu
HasMoreShardsN( R t Nonet timet sleept _jittered_backoffR R (
R R R R R R t
stream_statust attemptt max_retry_delayt stream_response( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt get_stream_when_active| s
!
c C@ sk t d | d | d | d | � } i } t | | � } d | k r] t | d <| d | d <n
t | d <| S( u�
Check if the stream exists. Returns False and the error if it does not.
CLI example::
salt myminion boto_kinesis.exists my_stream region=us-east-1
R R R R u erroru result( R R t FalseR ( R R R R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt exists� s !
c C@ sY t d | d | d | d | � } t | d d | d | �} d | k rU t | d <n | S(
u�
Create a stream with name stream_name and initial number of shards num_shards.
CLI example::
salt myminion boto_kinesis.create_stream my_stream N region=us-east-1
R R R R u
create_streamt
ShardCountR u erroru result( R R R ( R t
num_shardsR R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt
create_stream� s !
c C@ sS t d | d | d | d | � } t | d d | �} d | k rO t | d <n | S( u�
Delete the stream with name stream_name. This cannot be undone! All data will be lost!!
CLI example::
salt myminion boto_kinesis.delete_stream my_stream region=us-east-1
R R R R u
delete_streamR u erroru result( R R R ( R R R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt
delete_stream� s !
c C@ sY t d | d | d | d | � } t | d d | d | �} d | k rU t | d <n | S(
u�
Increase stream retention period to retention_hours
CLI example::
salt myminion boto_kinesis.increase_stream_retention_period my_stream N region=us-east-1
R R R R u increase_stream_retention_periodR t RetentionPeriodHoursu erroru result( R R R ( R t retention_hoursR R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt increase_stream_retention_period� s !
c C@ sY t d | d | d | d | � } t | d d | d | �} d | k rU t | d <n | S(
u�
Decrease stream retention period to retention_hours
CLI example::
salt myminion boto_kinesis.decrease_stream_retention_period my_stream N region=us-east-1
R R R R u decrease_stream_retention_periodR R, u erroru result( R R R ( R R- R R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt decrease_stream_retention_period� s !
c C@ sY t d | d | d | d | � } t | d d | d | �} d | k rU t | d <n | S(
u�
Enable enhanced monitoring for the specified shard-level metrics on stream stream_name
CLI example::
salt myminion boto_kinesis.enable_enhanced_monitoring my_stream ["metrics", "to", "enable"] region=us-east-1
R R R R u enable_enhanced_monitoringR t ShardLevelMetricsu erroru result( R R R ( R t metricsR R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt enable_enhanced_monitoring� s !
c C@ sY t d | d | d | d | � } t | d d | d | �} d | k rU t | d <n | S(
u�
Disable enhanced monitoring for the specified shard-level metrics on stream stream_name
CLI example::
salt myminion boto_kinesis.disable_enhanced_monitoring my_stream ["metrics", "to", "disable"] region=us-east-1
R R R R u disable_enhanced_monitoringR R0 u erroru result( R R R ( R R1 R R R R R R ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt disable_enhanced_monitoring s !
c C@ s d } d } g | d <x� | d D]� } | d } d | d k rW t j d | � q! n | d j | � t | d d � | d d <t | d d
� | d d
<| d d | k r� | d d } n | d d
| k r! | d d
} q! q! W| d j d d � � | | | f S(
u
Collect some data: number of open shards, key range, etc.
Modifies stream_details to add a sorted list of OpenShards.
Returns (min_hash_key, max_hash_key, stream_details)
CLI example::
salt myminion boto_kinesis.get_info_for_reshard existing_stream_details
i u
OpenShardsu Shardsu ShardIdu EndingSequenceNumberu SequenceNumberRangeu skipping closed shard %su HashKeyRangeu StartingHashKeyu
EndingHashKeyR c S@ s t | d d � S( Nu HashKeyRangeu StartingHashKey( t long_int( t shard( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt <lambda>> s ( t logt debugt appendR4 t sort( t stream_detailst min_hash_keyt max_hash_keyR5 t shard_id( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt get_info_for_reshard" s&
c C@ s' t j d k r t | � St | � Sd S( uS
The hash key is a 128-bit int, sent as a string.
It's necessary to convert to int/long for comparison operations.
This helper method handles python 2/3 incompatibility
CLI example::
salt myminion boto_kinesis.long_int some_MD5_hash_as_string
:return: long object if python 2.X, int object if python 3.X
i N( i ( t syst version_infot longt int( t hash_key( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyR4 C s
c C@ s� t d | d | d | d | � } i } t | | | | | � } d | k rO | S| d d }
t |
� \ } } }
t j d t |
d � | | � xEt |
d � D]3\ }
} | d
} d | d k r� t j d
| � q� n | d d } | d d } | | | |
|
} | | | |
d |
} | | k r@| } n t j d |
| | | | k � t j d |
| | | | k � | | k r�d | d <| S| | k r�q� n | | d k r7| rt j d | | | � t | d d | d | d t | d � �} n t j d | | | � d | k r3t | d <n | St
|
| � } | scd j | � | d <| S| r�t j d | | | � t | d d | d | d | �} n t j d | | | � d | k r�t | d <n | Sq� Wt j d! � t | d <| S(" u
Reshard a kinesis stream. Each call to this function will wait until the stream is ACTIVE,
then make a single split or merge operation. This function decides where to split or merge
with the assumption that the ultimate goal is a balanced partition space.
For safety, user must past in force=True; otherwise, the function will dry run.
CLI example::
salt myminion boto_kinesis.reshard my_stream N True region=us-east-1
:return: True if a split or merge was found/performed, False if nothing is needed
R R R R u erroru resultu StreamDescriptionu5 found %s open shards, min_hash_key %s max_hash_key %su
OpenShardsu ShardIdu EndingSequenceNumberu SequenceNumberRangeu) this should never happen! closed shard %su HashKeyRangeu StartingHashKeyu
EndingHashKeyi u$ Shard %s (%s) should start at %s: %su" Shard %s (%s) should end at %s: %su3 starting hash keys mismatch, don't know what to do!u) %s should end at %s, actual %s, splittingu split_shardR t ShardToSplitt NewStartingHashKeyu* %s should end at %s, actual %s would splitu# failed to find next shard after {0}u+ %s should continue past %s, merging with %su merge_shardst ShardToMerget AdjacentShardToMergeu/ %s should continue past %s, would merge with %su" No split or merge action necessary(
R R% R? R7 R8 t lent enumerateR t strR t _get_next_open_shardt formatR& ( R t desired_sizet forceR R R R R R R$ R; R< R= t shard_numR5 R> t starting_hash_keyt ending_hash_keyt expected_starting_hash_keyt expected_ending_hash_keyt
next_shard_id( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt reshardU s� !
c C@ s� t d | d | d | d | � } g } d } x� | d k r� | rO i | d 6n i } t | d | � } d | k rw | S| r� | j d � r� | d n i } | | j d
g � 7} | j d t � t d f k r� | d
n d } q0 Wi | d 6S( u�
Return a list of all streams visible to the current account
CLI example:
.. code-block:: bash
salt myminion boto_kinesis.list_streams
R R R R u u ExclusiveStartStreamNameu list_streamsu erroru resultu StreamNamesu HasMoreStreamsu truei����N( R R R t getR& R ( R R R R R t streamst exclusive_start_stream_namet argst ret( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt list_streams� s
! %2c C@ sK t } x> | d D]2 } | d } | | k r9 t } q n | r | Sq Wd S( u�
Return the next open shard after shard_id
CLI example::
salt myminion boto_kinesis._get_next_open_shard existing_stream_details shard_id
u
OpenShardsu ShardIdN( R& R ( R; R> t foundR5 t current_shard_id( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyRL � s
c
K@ s/ i } d } d } x� t | � D]� } t j d | | � y' t | | � } | | � | d <| SWq t j j k
r� } | j d d } d | k s� d | k r� t j d d
t �t
j t | | � � q | j d | d <t j
| d � d
| d <| Sq Xq Wd j | | � | d <t j
| d � | S( u�
Retry if we're rate limited by AWS or blocked by another call.
Give up and return error message if resource not found or argument is invalid.
conn
The connection established by the calling method via _get_conn()
function
The function to call on conn. i.e. create_stream
**kwargs
Any kwargs required by the above function, with their keywords
i.e. StreamName=stream_name
Returns:
The result dict with the HTTP response and JSON data if applicable
as 'result', or an error as 'error'
CLI example::
salt myminion boto_kinesis._execute_with_retries existing_conn function_name function_kwargs
i i
u attempt: %s function: %su resultu Erroru Codeu LimitExceededExceptionu ResourceInUseExceptionu Retrying due to AWS exceptiont exc_infou erroru7 Tried to execute function {0} {1} times, but was unableN( R R7 t infot getattrt botocoret
exceptionst ClientErrort responseR8 R R R R t errorR RM (
R t functiont kwargsR t max_attemptsR# R" t fnt et
error_code( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyR � s*
c C@ s t t j � d | | � S( u�
Basic exponential backoff
CLI example::
salt myminion boto_kinesis._jittered_backoff current_attempt_number max_delay_in_seconds
i ( t mint random( R" R# ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyR " s (, t __doc__t
__future__R R R t loggingR Rn R@ t salt.ext.six.movesR t salt.utils.versionsR t boto3Rb t getLoggert setLevelt CRITICALR t HAS_BOTOt ImportErrorR& R
R7 R R
R R R R% R' R* R+ R. R/ R2 R3 R? R4 RV R\ RL R R ( ( ( s= /usr/lib/python2.7/site-packages/salt/modules/boto_kinesis.pyt <module>, sF
! n 4
Mini Shell