Current File : //usr/lib/python2.7/site-packages/salt/pillar/stack.pyc
�
���^c @@ s d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
m Z m Z d d l
m Z d d l Z d d l Z d d l Z e j e � Z d Z d
� Z d � Z d � Z d
� Z d � Z d � Z d � Z d S( u�A
Simple and flexible YAML ext_pillar which can read pillar from within pillar.
.. versionadded:: 2016.3.0
`PillarStack <https://github.com/bbinet/pillarstack>`_ is a custom saltstack
``ext_pillar`` which was inspired by `varstack
<https://github.com/conversis/varstack>`_ but is heavily based on Jinja2 for
maximum flexibility.
Any issue should be reported to the upstream project at:
https://github.com/bbinet/pillarstack/issues
It supports the following features:
- multiple config files that are jinja2 templates with support for ``pillar``,
``__grains__``, ``__salt__``, ``__opts__`` objects
- a config file renders as an ordered list of files (paths of these files are
relative to the current config file)
- this list of files are read in ordered as jinja2 templates with support for
``stack``, ``pillar``, ``__grains__``, ``__salt__``, ``__opts__`` objects
- all these rendered files are then parsed as ``yaml``
- then all yaml dicts are merged in order with support for the following
merging strategies: ``merge-first``, ``merge-last``, ``remove``, and
``overwrite``
- stack config files can be matched based on ``pillar``, ``grains``, or
``opts`` values, which make it possible to support kind of self-contained
environments
Installation
------------
PillarStack is already bundled with Salt since 2016.3.0 version so there is
nothing to install from version 2016.3.0.
If you use an older Salt version or you want to override PillarStack with a
more recent one, follow the installation procedure below.
Installing the PillarStack ``ext_pillar`` is as simple as dropping the
``stack.py`` file in the ``<extension_modules>/pillar`` directory (no external
python module required), given that ``extension_modules`` is set in your
salt-master configuration, see:
http://docs.saltstack.com/en/latest/ref/configuration/master.html#extension-modules
Configuration in Salt
---------------------
Like any other external pillar, its configuration takes place through the
``ext_pillar`` key in the master config file.
However, you can configure PillarStack in 3 different ways:
Single config file
~~~~~~~~~~~~~~~~~~
This is the simplest option, you just need to set the path to your single
PillarStack config file like below:
.. code:: yaml
ext_pillar:
- stack: /path/to/stack.cfg
List of config files
~~~~~~~~~~~~~~~~~~~~
You can also provide a list of config files:
.. code:: yaml
ext_pillar:
- stack:
- /path/to/stack1.cfg
- /path/to/stack2.cfg
Select config files through grains|pillar|opts matching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also opt for a much more flexible configuration: PillarStack allows one
to select the config files for the current minion based on matching values from
either grains, or pillar, or opts objects.
Here is an example of such a configuration, which should speak by itself:
.. code:: yaml
ext_pillar:
- stack:
pillar:environment:
dev: /path/to/dev/stack.cfg
prod: /path/to/prod/stack.cfg
grains:custom:grain:
value:
- /path/to/stack1.cfg
- /path/to/stack2.cfg
opts:custom:opt:
value: /path/to/stack0.cfg
PillarStack configuration files
-------------------------------
The config files that are referenced in the above ``ext_pillar`` configuration
are jinja2 templates which must render as a simple ordered list of ``yaml``
files that will then be merged to build pillar data.
The path of these ``yaml`` files must be relative to the directory of the
PillarStack config file. These paths support unix style pathname pattern
expansion through the
`Python glob module <https://docs.python.org/2/library/glob.html>`.
The following variables are available in jinja2 templating of PillarStack
configuration files:
- ``pillar``: the pillar data (as passed by Salt to our ``ext_pillar``
function)
- ``minion_id``: the minion id ;-)
- ``__opts__``: a dictionary of mostly Salt configuration options
- ``__grains__``: a dictionary of the grains of the minion making this pillar
call
- ``__salt__``: a dictionary of Salt module functions, useful so you don't have
to duplicate functions that already exist (note: runs on the master)
So you can use all the power of jinja2 to build your list of ``yaml`` files
that will be merged in pillar data.
For example, you could have a PillarStack config file which looks like:
.. code:: jinja
$ cat /path/to/stack/config.cfg
core.yml
common/*.yml
osarchs/{{ __grains__['osarch'] }}.yml
oscodenames/{{ __grains__['oscodename'] }}.yml
{%- for role in pillar.get('roles', []) %}
roles/{{ role }}.yml
{%- endfor %}
minions/{{ minion_id }}.yml
And the whole directory structure could look like:
.. code::
$ tree /path/to/stack/
/path/to/stack/
├── config.cfg
├── core.yml
├── common/
│ ├── xxx.yml
│ └── yyy.yml
├── osarchs/
│ ├── amd64.yml
│ └── armhf.yml
├── oscodenames/
│ ├── wheezy.yml
│ └── jessie.yml
├── roles/
│ ├── web.yml
│ └── db.yml
└── minions/
├── test-1-dev.yml
└── test-2-dev.yml
Overall process
---------------
In the above PillarStack configuration, given that test-1-dev minion is an
amd64 platform running Debian Jessie, and which pillar ``roles`` is ``["db"]``,
the following ``yaml`` files would be merged in order:
- ``core.yml``
- ``common/xxx.yml``
- ``common/yyy.yml``
- ``osarchs/amd64.yml``
- ``oscodenames/jessie.yml``
- ``roles/db.yml``
- ``minions/test-1-dev.yml``
Before merging, every files above will be preprocessed as Jinja2 templates.
The following variables are available in Jinja2 templating of ``yaml`` files:
- ``stack``: the PillarStack pillar data object that has currently been merged
(data from previous ``yaml`` files in PillarStack configuration)
- ``pillar``: the pillar data (as passed by Salt to our ``ext_pillar``
function)
- ``minion_id``: the minion id ;-)
- ``__opts__``: a dictionary of mostly Salt configuration options
- ``__grains__``: a dictionary of the grains of the minion making this pillar
call
- ``__salt__``: a dictionary of Salt module functions, useful so you don't have
to duplicate functions that already exist (note: runs on the master)
So you can use all the power of jinja2 to build your pillar data, and even use
other pillar values that has already been merged by PillarStack (from previous
``yaml`` files in PillarStack configuration) through the ``stack`` variable.
Once a ``yaml`` file has been preprocessed by Jinja2, we obtain a Python dict -
let's call it ``yml_data`` - then, PillarStack will merge this ``yml_data``
dict in the main ``stack`` dict (which contains already merged PillarStack
pillar data).
By default, PillarStack will deeply merge ``yml_data`` in ``stack`` (similarly
to the ``recurse`` salt ``pillar_source_merging_strategy``), but 3 merging
strategies are currently available for you to choose (see next section).
Once every ``yaml`` files have been processed, the ``stack`` dict will contain
your whole own pillar data, merged in order by PillarStack.
So PillarStack ``ext_pillar`` returns the ``stack`` dict, the contents of which
Salt takes care to merge in with all of the other pillars and finally return
the whole pillar to the minion.
Merging strategies
------------------
The way the data from a new ``yaml_data`` dict is merged with the existing
``stack`` data can be controlled by specifying a merging strategy. Right now
this strategy can either be ``merge-last`` (the default), ``merge-first``,
``remove``, or ``overwrite``.
Note that scalar values like strings, integers, booleans, etc. are always
evaluated using the ``overwrite`` strategy (other strategies don't make sense
in that case).
The merging strategy can be set by including a dict in the form of:
.. code:: yaml
__: <merging strategy>
as the first item of the dict or list.
This allows fine grained control over the merging process.
``merge-last`` (default) strategy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the ``merge-last`` strategy is selected (the default), then content of dict
or list variables is merged recursively with previous definitions of this
variable (similarly to the ``recurse`` salt
``pillar_source_merging_strategy``).
This allows for extending previously defined data.
``merge-first`` strategy
~~~~~~~~~~~~~~~~~~~~~~~~
If the ``merge-first`` strategy is selected, then the content of dict or list
variables are swapped between the ``yaml_data`` and ``stack`` objects before
being merged recursively with the ``merge-last`` previous strategy.
``remove`` strategy
~~~~~~~~~~~~~~~~~~~
If the ``remove`` strategy is selected, then content of dict or list variables
in ``stack`` are removed only if the corresponding item is present in the
``yaml_data`` dict.
This allows for removing items from previously defined data.
``overwrite`` strategy
~~~~~~~~~~~~~~~~~~~~~~
If the ``overwrite`` strategy is selected, then the content of dict or list
variables in ``stack`` is overwritten by the content of ``yaml_data`` dict.
So this allows one to overwrite variables from previous definitions.
Merging examples
----------------
Let's go through small examples that should clarify what's going on when a
``yaml_data`` dict is merged in the ``stack`` dict.
When you don't specify any strategy, the default ``merge-last`` strategy is
selected:
+----------------------+-----------------------+-------------------------+
| ``stack`` | ``yaml_data`` | ``stack`` (after merge) |
+======================+=======================+=========================+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | tom: | tom: |
| uid: 500 | uid: 1000 | uid: 1000 |
| roles: | roles: | roles: |
| - sysadmin | - developer | - sysadmin |
| root: | mat: | - developer |
| uid: 0 | uid: 1001 | mat: |
| | | uid: 1001 |
| | | root: |
| | | uid: 0 |
+----------------------+-----------------------+-------------------------+
Then you can select a custom merging strategy using the ``__`` key in a dict:
+----------------------+-----------------------+-------------------------+
| ``stack`` | ``yaml_data`` | ``stack`` (after merge) |
+======================+=======================+=========================+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: merge-last | tom: |
| uid: 500 | tom: | uid: 1000 |
| roles: | uid: 1000 | roles: |
| - sysadmin | roles: | - sysadmin |
| root: | - developer | - developer |
| uid: 0 | mat: | mat: |
| | uid: 1001 | uid: 1001 |
| | | root: |
| | | uid: 0 |
+----------------------+-----------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: merge-first | tom: |
| uid: 500 | tom: | uid: 500 |
| roles: | uid: 1000 | roles: |
| - sysadmin | roles: | - developer |
| root: | - developer | - sysadmin |
| uid: 0 | mat: | mat: |
| | uid: 1001 | uid: 1001 |
| | | root: |
| | | uid: 0 |
+----------------------+-----------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: remove | root: |
| uid: 500 | tom: | uid: 0 |
| roles: | mat: | |
| - sysadmin | | |
| root: | | |
| uid: 0 | | |
+----------------------+-----------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| tom: | __: overwrite | tom: |
| uid: 500 | tom: | uid: 1000 |
| roles: | uid: 1000 | roles: |
| - sysadmin | roles: | - developer |
| root: | - developer | mat: |
| uid: 0 | mat: | uid: 1001 |
| | uid: 1001 | |
+----------------------+-----------------------+-------------------------+
You can also select a custom merging strategy using a ``__`` object in a list:
+----------------+-------------------------+-------------------------+
| ``stack`` | ``yaml_data`` | ``stack`` (after merge) |
+================+=========================+=========================+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: merge-last | - tom |
| - root | - mat | - root |
| | | - mat |
+----------------+-------------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: merge-first | - mat |
| - root | - mat | - tom |
| | | - root |
+----------------+-------------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: remove | - root |
| - root | - mat | |
| | - tom | |
+----------------+-------------------------+-------------------------+
| .. code:: yaml | .. code:: yaml | .. code:: yaml |
| | | |
| users: | users: | users: |
| - tom | - __: overwrite | - mat |
| - root | - mat | |
+----------------+-------------------------+-------------------------+
i ( t absolute_importt print_functiont unicode_literalsN( t FileSystemLoadert Environment( t sixu overwriteu merge-firstu
merge-lastu removec O@ sg i } t | � } i t j t j j j | � d 6t j t j j j t � d 6t j t j j j t � d 6} x� t j
| � D]� \ } } | j d d � \ } } | | k r� t d j
| | j � � � � n | j | | | d � g � }
t |
t � s|
g }
n | |
7} q| WxK | D]C } t j j | � sJt j d | � q n t | | | | � } q W| S( Nu pillaru grainsu optsu :i u3 Unknown traverse option "{0}", should be one of {1}u3 Ignoring pillar stack cfg "%s": file does not exist( t listt functoolst partialt saltt utilst datat traverse_dict_and_listt
__grains__t __opts__R t iteritemst splitt Exceptiont formatt keyst gett Nonet
isinstancet ost patht isfilet logt infot _process_stack_cfg( t minion_idt pillart argst kwargst stackt stack_config_filest traverset matchert matchst tt cfgst cfg( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyt
ext_pillar� s,
c C@ s t j | j t j � � S( N( t posixpatht joinR R t sep( R ( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyt _to_unix_slashes� s c C@ s� t j d | � t j j | � \ } } t d t | � d d t j j j
g � } | j j i t
d 6t d 6t d 6i t j j j d 6| d 6d
6| d 6| d 6� x!t | j | � j d
| � � D]� } | j � s� q� n t j t j j | | � � } | st j d | | � q� n x� t | � D]� } t j d | | � t t j j | | � � }
t j j j | j |
� j d
| d | � � } t | t � s�t j d | � q'n t | | � } q'Wq� W| S( Nu
Config: %st loadert
extensionsu
jinja2.ext.dou __opts__u __salt__u
__grains__u traverseu cfg_pathu __stack__u minion_idu pillarR! uB Ignoring pillar stack template "%s": can't find from root dir "%s"u YAML: basedir=%s, path=%st ymlpathuK Ignoring pillar stack template "%s": Can't parse as a valid yaml dictionary(! R t debugR R R R R R R
t jinjat SerializerExtensiont globalst updateR t __salt__R
R R t _parse_stack_cfgt get_templatet rendert stript globR+ R t sortedR- t relpatht yamlt safe_loadR t dictt _merge_dict( R( R! R R t basedirt filenamet jenvt itemt pathsR t unix_patht obj( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyR � s@ * "
0 c C@ s� | r� t | t � rX | j d d � xl t j | � D] \ } } t | � | | <q5 Wq� t | t � r� t | d t � r� d | d k r� | d =q� n | S( Nu __i ( R R@ t popR R R t _cleanupR ( RH t kt v( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyRJ � s "
c C@ s� | j d d � } | t k r9 t d j | t � � � n | d k rO t | � Sx/t j | � D] \ } } | d k r� | j | d � q_ n | | k rm| d k r� | | } t | � | | <| } n t | | � t | � k rt j
d | | | � t | � | | <q}t | t � r7t
| | | � | | <q}t | t � r`t | | | � | | <q}| | | <q_ t | � | | <q_ W| Sd S( Nu __u
merge-lastu, Unknown strategy "{0}", should be one of {1}u overwriteu removeu merge-firstu+ Force overwrite, types differ: '%s' != '%s'( RI t
strategiesR R RJ R R R t typeR R1 R R@ RA R t _merge_list( R! RH t strategyRK RL t stack_k( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyRA � s2
c C@ s� d } | rG t | d t � rG d | d k rG | d d } | d =n | t k rn t d j | t � � � n | d k r~ | S| d k r� g | D] } | | k r� | ^ q� S| d k r� | | S| | Sd S( Nu
merge-lasti u __u, Unknown strategy "{0}", should be one of {1}u overwriteu removeu merge-first( R R@ RM R R ( R! RH RP RE ( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyRO � s )
# c C@ sL y, t j j j | � } t | t � r+ | SWn t k
rA } n X| j � S( u(
Allow top level cfg to be YAML
( R R
R> R? R R R t
splitlines( t contentRH t e( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyR7 s ( u overwriteu merge-firstu
merge-lastu remove( t __doc__t
__future__R R R R R; R R* t loggingt jinja2R R t salt.extR t salt.utils.dataR t salt.utils.jinjat salt.utils.yamlt getLoggert __name__R RM R) R- R RJ RA RO R7 ( ( ( s5 /usr/lib/python2.7/site-packages/salt/pillar/stack.pyt <module>w s( ' !
Mini Shell