Current File : //usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyo
�
���^c @@ s� d Z d d l m Z m Z m Z d d l Z y6 d d l Z d d l m Z d d l m
Z e Z
Wn e k
r} e Z
n Xy d d l m Z e Z Wn e k
r� e Z n Xd d l m Z d d l m Z d Z i d
d 6Z e j e � Z d Z d
Z d Z d Z d a! d � Z" d � Z# d � Z$ d d � Z% d � Z& d � Z' d � Z( d � Z) d � Z* d d � Z+ d � Z, d � Z- d d � Z. d � Z/ d � Z0 d S( uZ
Redis
=====
Redis plugin for the Salt caching subsystem.
.. versionadded:: 2017.7.0
As Redis provides a simple mechanism for very fast key-value store, in order to
privde the necessary features for the Salt caching subsystem, the following
conventions are used:
- A Redis key consists of the bank name and the cache key separated by ``/``, e.g.:
``$KEY_minions/alpha/stuff`` where ``minions/alpha`` is the bank name
and ``stuff`` is the key name.
- As the caching subsystem is organised as a tree, we need to store the caching
path and identify the bank and its offspring. At the same time, Redis is
linear and we need to avoid doing ``keys <pattern>`` which is very
inefficient as it goes through all the keys on the remote Redis server.
Instead, each bank hierarchy has a Redis SET associated which stores the list
of sub-banks. By default, these keys begin with ``$BANK_``.
- In addition, each key name is stored in a separate SET of all the keys within
a bank. By default, these SETs begin with ``$BANKEYS_``.
For example, to store the key ``my-key`` under the bank ``root-bank/sub-bank/leaf-bank``,
the following hierarchy will be built:
.. code-block:: text
127.0.0.1:6379> SMEMBERS $BANK_root-bank
1) "sub-bank"
127.0.0.1:6379> SMEMBERS $BANK_root-bank/sub-bank
1) "leaf-bank"
127.0.0.1:6379> SMEMBERS $BANKEYS_root-bank/sub-bank/leaf-bank
1) "my-key"
127.0.0.1:6379> GET $KEY_root-bank/sub-bank/leaf-bank/my-key
"my-value"
There are three types of keys stored:
- ``$BANK_*`` is a Redis SET containing the list of banks under the current bank
- ``$BANKEYS_*`` is a Redis SET containing the list of keys under the current bank
- ``$KEY_*`` keeps the value of the key
These prefixes and the separator can be adjusted using the configuration options:
bank_prefix: ``$BANK``
The prefix used for the name of the Redis key storing the list of sub-banks.
bank_keys_prefix: ``$BANKEYS``
The prefix used for the name of the Redis keyt storing the list of keys under a certain bank.
key_prefix: ``$KEY``
The prefix of the Redis keys having the value of the keys to be cached under
a certain bank.
separator: ``_``
The separator between the prefix and the key body.
The connection details can be specified using:
host: ``localhost``
The hostname of the Redis server.
port: ``6379``
The Redis server port.
cluster_mode: ``False``
Whether cluster_mode is enabled or not
cluster.startup_nodes:
A list of host, port dictionaries pointing to cluster members. At least one is required
but multiple nodes are better
.. code-block:: yaml
cache.redis.cluster.startup_nodes
- host: redis-member-1
port: 6379
- host: redis-member-2
port: 6379
cluster.skip_full_coverage_check: ``False``
Some cluster providers restrict certain redis commands such as CONFIG for enhanced security.
Set this option to true to skip checks that required advanced privileges.
.. note::
Most cloud hosted redis clusters will require this to be set to ``True``
db: ``'0'``
The database index.
.. note::
The database index must be specified as string not as integer value!
password:
Redis connection password.
unix_socket_path:
.. versionadded:: 2018.3.1
Path to a UNIX socket for access. Overrides `host` / `port`.
Configuration Example:
.. code-block:: yaml
cache.redis.host: localhost
cache.redis.port: 6379
cache.redis.db: '0'
cache.redis.password: my pass
cache.redis.bank_prefix: #BANK
cache.redis.bank_keys_prefix: #BANKEYS
cache.redis.key_prefix: #KEY
cache.redis.separator: '@'
Cluster Configuration Example:
.. code-block:: yaml
cache.redis.cluster_mode: true
cache.redis.cluster.skip_full_coverage_check: true
cache.redis.cluster.startup_nodes:
- host: redis-member-1
port: 6379
- host: redis-member-2
port: 6379
cache.redis.db: '0'
cache.redis.password: my pass
cache.redis.bank_prefix: #BANK
cache.redis.bank_keys_prefix: #BANKEYS
cache.redis.key_prefix: #KEY
cache.redis.separator: '@'
i ( t absolute_importt print_functiont unicode_literalsN( t ConnectionError( t
ResponseError( t StrictRedisCluster( t range( t SaltCacheErroru redisu listu list_u $BANKu $KEYu $BANKEYSu _c C@ s2 t s t d f St r. t � d r. t d f St S( u�
The redis library must be installed for this module to work.
The redis redis cluster library must be installed if cluster_mode is True
u( Please install the python-redis package.u cluster_modeu, Please install the redis-py-cluster package.( t HAS_REDISt Falset HAS_REDIS_CLUSTERt _get_redis_cache_optst __virtualname__( ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt __virtual__� s
c C@ s i S( N( ( t kwargs( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt init_kwargs� s c C@ s� i t j d d � d 6t j d d � d 6t j d d � d 6t j d d
� d 6t j d d
� d 6t j d t � d 6t j d i � d 6t j d t � d 6S( uG
Return the Redis server connection details from the __opts__.
u cache.redis.hostu localhostu hostu cache.redis.porti� u portu cache.redis.unix_socket_pathu unix_socket_pathu cache.redis.dbu 0u dbu cache.redis.passwordu u passwordu cache.redis.cluster_modeu cluster_modeu! cache.redis.cluster.startup_nodesu
startup_nodesu, cache.redis.cluster.skip_full_coverage_checku skip_full_coverage_checkN( t __opts__t gett NoneR ( ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyR � s c
C@ s� t r
t S| s t � } n | d rF t d | d d | d � a n8 t j | d | d d | d d
| d d | d
�a t S( uL
Return the Redis server instance.
Caching the object instance.
u cluster_modet
startup_nodesu
startup_nodest skip_full_coverage_checku skip_full_coverage_checku hostu portt unix_socket_pathu unix_socket_patht dbu dbt passwordu password( t REDIS_SERVERR R t redist StrictRedis( t opts( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt _get_redis_server� s
c C@ sP i t j d t � d 6t j d t � d 6t j d t � d 6t j d t � d 6S( u7
Build the key opts based on the user options.
u cache.redis.bank_prefixu bank_prefixu cache.redis.bank_keys_prefixu bank_keys_prefixu cache.redis.key_prefixu
key_prefixu cache.redis.separatoru separator( R R t _BANK_PREFIXt _BANK_KEYS_PREFIXt _KEY_PREFIXt
_SEPARATOR( ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt _get_redis_keys_opts� s
c C@ s- t � } d j d | d d | d d | � S( u;
Return the Redis key for the bank given the name.
u {prefix}{separator}{bank}t prefixu bank_prefixt separatoru separatort bank( R! t format( R$ R ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt _get_bank_redis_key s
c C@ s3 t � } d j d | d d | d d | d | � S( uD
Return the Redis key given the bank name and the key name.
u {prefix}{separator}{bank}/{key}R" u
key_prefixR# u separatorR$ t key( R! R% ( R$ R' R ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt _get_key_redis_key s
c C@ s- t � } d j d | d d | d d | � S( u]
Return the Redis key for the SET of keys under a certain bank, given the bank name.
u {prefix}{separator}{bank}R" u bank_keys_prefixR# u separatorR$ ( R! R% ( R$ R ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt _get_bank_keys_redis_key s
c C@ sy | j d � } | d } xY | d D]M } t | � } | j | | � t j d | | � d j d | d | � } q$ Wt S( u�
Build the bank hierarchy from the root of the tree.
If already exists, it won't rewrite.
It's using the Redis pipeline,
so there will be only one interaction with the remote server.
u /i i u Adding %s to %su {curr_path}/{bank_name}t curr_patht bank_name( t splitR& t saddt logt debugR% t True( R$ t
redis_pipet bank_listt parent_bank_pathR+ t prev_bank_redis_key( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt _build_bank_hier% s
u c C@ s� | s | n d j d | d | � } | g } t | � } | j | � } | sR | Sx* | D]" } | j t | | d | �� qY W| S( u�
A simple tree tarversal algorithm that builds the list of banks to remove,
starting from an arbitrary node in the tree.
u
{path}/{bank}t pathR$ ( R% R& t smemberst extendt _get_banks_to_remove( t redis_serverR$ R6 t current_patht bank_paths_to_removet bank_keyt child_bankst
child_bank( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyR9 9 s $
c
C@ s� t � } | j � } t | | � } t | � } yw t | | � t d j | � } | j | | � t j d | | | � | j
| | � t j d | | � | j � WnJ t t
f k
r� } d j d | d | � } t j | � t | � � n Xd S( u(
Store the data in a Redis key.
u serialu& Setting the value for %s under %s (%s)u Adding %s to %su- Cannot set the Redis cache key {rkey}: {rerr}t rkeyt rerrN( R t pipelineR( R) R5 t __context__t dumpst setR. R/ R- t executet RedisConnectionErrort RedisResponseErrorR% t errorR (
R$ R' t dataR: R1 t redis_keyt redis_bank_keyst valueRA t mesg( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt storeQ s"
c C@ s� t � } t | | � } d } y | j | � } WnJ t t f k
r} } d j d | d | � } t j | � t | � � n X| d k r� i St
d j | � S( u*
Fetch data from the Redis cache.
u/ Cannot fetch the Redis cache key {rkey}: {rerr}R@ RA u serialN( R R( R R RG RH R% R. RI R RC t loads( R$ R' R: RK t redis_valueRA RN ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt fetchh s
c C@ sx t � } | j � } | d
k r�t | | � } x: | D]2 } t | � } | j | � t j d | | � q7 Wy t j d � | j � } WnS t t
f k
r� } d j d d j | � d | � } t j
| � t | � � n Xt | � }
x*t |
� D]� } | | } | | }
x@ | D]8 } t |
| � } | j | � t j d | |
| � qWt |
� } | j | � t j d |
| � t |
� } | j | � t j d |
| � q� Wnd t | | � } | j | � t j d | | | � t | � } | j | | � t j d
| | | � y | j � WnJ t t
f k
rs} d j d | d | � } t j
| � t | � � n Xt S( ud
Remove the key from the cache bank with all the key content. If no key is specified, remove
the entire bank with all keys and sub-banks inside.
This function is using the Redis pipelining for best performance.
However, when removing a whole bank,
in order to re-create the tree, there are a couple of requests made. In total:
- one for node in the hierarchy sub-tree, starting from the bank node
- one pipelined request to get the keys under all banks in the sub-tree
- one pipeline request to remove the corresponding keys
This is not quite optimal, as if we need to flush a bank having
a very long list of sub-banks, the number of requests to build the sub-tree may grow quite big.
An improvement for this would be loading a custom Lua script in the Redis instance of the user
(using the ``register_script`` feature) and call it whenever we flush.
This script would only need to build this sub-tree causing problems. It can be added later and the behaviour
should not change as the user needs to explicitly allow Salt inject scripts in their Redis instance.
u% Fetching the keys of the %s bank (%s)u Executing the pipe...uB Cannot retrieve the keys under these cache banks: {rbanks}: {rerr}t rbanksu , RA u* Removing the key %s under the %s bank (%s)u/ Removing the bank-keys key for the %s bank (%s)u Removing the %s bank (%s)u@ De-referencing the key %s from the bank-keys of the %s bank (%s)u1 Cannot flush the Redis cache bank {rbank}: {rerr}t rbankN( R RB R R9 R) R7 R. R/ RF RG RH R% t joinRI R t lenR R( t deleteR& t sremR0 ( R$ R' R: R1 R< t bank_to_removet bank_keys_redis_keyt subtree_keysRA RN t total_bankst indext bank_keyst bank_pathRK R= ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt flush{ sn
c C@ s� t � } t | � } y | j | � } WnJ t t f k
rt } d j d | d | � } t j | � t | � � n X| s g St | � S( u5
Lists entries stored in the specified bank.
u. Cannot list the Redis cache key {rkey}: {rerr}R@ RA (
R R& R7 RG RH R% R. RI R t list( R$ R: t bank_redis_keyt banksRA RN ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt list_� s
c C@ sz t � } t | � } y | j | | � SWnJ t t f k
ru } d j d | d | � } t j | � t | � � n Xd S( uB
Checks if the specified bank contains the specified key.
u2 Cannot retrieve the Redis cache key {rkey}: {rerr}R@ RA N( R R& t sismemberRG RH R% R. RI R ( R$ R' R: Rb RA RN ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt contains� s
(1 t __doc__t
__future__R R R t loggingR t redis.exceptionsR RG R RH R0 R t ImportErrorR t redisclusterR R
t salt.ext.six.movesR t salt.exceptionsR R t __func_alias__t getLoggert __file__R. R R R R R R R
R R R R! R& R( R) R5 R9 RO RR R` Rd Rf ( ( ( s: /usr/lib/python2.7/site-packages/salt/cache/redis_cache.pyt <module>� sN
_
Mini Shell