Current File : //lib/python2.7/site-packages/salt/modules/highstate_doc.pyc
�
���^c @@ s0 d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l
j j Z
d d l Z d Z e j e � Z d Z d e Z e d Z d � Z d � Z d
� Z d � Z d � Z d d
d � Z d � Z d � Z d d d � Z d � Z d � Z! d � Z" e# d � Z$ d � Z% d S( u�
This module renders highstate configuration into a more human readable format.
How it works:
`highstate or lowstate` data is parsed with a `proccesser` this defaults to `highstate_doc.proccesser_markdown`.
The proccessed data is passed to a `jinja` template that builds up the document content.
configuration: Pillar
.. code-block:: yaml
# the following defaults can be overrided
highstate_doc.config:
# list of regex of state names to ignore in `highstate_doc.proccess_lowstates`
filter_id_regex:
- '.*!doc_skip$'
# list of regex of state functions to ignore in `highstate_doc.proccess_lowstates`
filter_state_function_regex:
- 'file.accumulated'
# dict of regex to replace text after `highstate_doc.render`. (remove passwords)
text_replace_regex:
'password:.*^': '[PASSWORD]'
# limit size of files that can be included in doc (10000 bytes)
max_render_file_size: 10000
# advanced option to set a custom lowstate proccesser
proccesser: highstate_doc.proccesser_markdown
State example
.. code-block:: yaml
{{sls}} note:
highstate_doc.note:
- name: example
- order: 0
- contents: |
example `highstate_doc.note`
------------------
This state does not do anything to the system! It is only used by a `proccesser`
you can use `requisites` and `order` to move your docs around the rendered file.
{{sls}} a file we dont want in the doc !doc_skip:
file.managed:
- name: /root/passwords
- contents: 'password: sadefgq34y45h56q'
# also could use `highstate_doc.config: text_replace_regex` to replace
# password string. `password:.*^': '[PASSWORD]`
To create the help document build a State that uses `highstate_doc.render`.
For preformance it's advised to not included this state in your `top.sls` file.
.. code-block:: yaml
# example `salt://makereadme.sls`
make helpfile:
file.managed:
- name: /root/README.md
- contents: {{salt.highstate_doc.render()|json}}
- show_diff: {{opts['test']}}
- mode: '0640'
- order: last
Run our `makereadme.sls` state to create `/root/README.md`.
.. code-block:: bash
# first ensure `highstate` return without errors or changes
salt-call state.highstate
salt-call state.apply makereadme
# or if you dont want the extra `make helpfile` state
salt-call --out=newline_values_only salt.highstate_doc.render > /root/README.md ; chmod 0600 /root/README.md
Creating a document collection
------------------------------
From the master we can run the following script to
creates a collection of all your minion documents.
.. code-block:: bash
salt '*' state.apply makereadme
.. code-block:: python
#!/bin/python
import os
import salt.client
s = salt.client.LocalClient()
# NOTE: because of issues with `cp.push` use `highstate_doc.read_file`
o = s.cmd('*', 'highstate_doc.read_file', ['/root/README.md'])
for m in o:
d = o.get(m)
if d and not d.endswith('is not available.'):
# mkdir m
#directory = os.path.dirname(file_path)
if not os.path.exists(m):
os.makedirs(m)
with open(m + '/README.md','wb') as f:
f.write(d)
print('ADDED: ' + m + '/README.md')
Once the master has a collection of all the README files.
You can use pandoc to create HTML versions of the markdown.
.. code-block:: bash
# proccess all the readme.md files to readme.html
if which pandoc; then echo "Found pandoc"; else echo "** Missing pandoc"; exit 1; fi
if which gs; then echo "Found gs"; else echo "** Missing gs(ghostscript)"; exit 1; fi
readme_files=$(find $dest -type f -path "*/README.md" -print)
for f in $readme_files ; do
ff=${f#$dest/}
minion=${ff%%/*}
echo "proccess: $dest/${minion}/$(basename $f)"
cat $dest/${minion}/$(basename $f) | pandoc --standalone --from markdown_github --to html --include-in-header $dest/style.html > $dest/${minion}/$(basename $f).html
done
It is also nice to put the help files in source control.
# git init
git add -A
git commit -am 'updated docs'
git push -f
Other hints
-----------
If you wish to customize the document format:
.. code-block:: yaml
# you could also create a new `proccesser` for perhaps reStructuredText
# highstate_doc.config:
# proccesser: doc_custom.proccesser_rst
# example `salt://makereadme.jinja`
"""
{{opts['id']}}
==========================================
{# lowstates is set from highstate_doc.render() #}
{# if lowstates is missing use salt.highstate_doc.proccess_lowstates() #}
{% for s in lowstates %}
{{s.id}}
-----------------------------------------------------------------
{{s.function}}
{{s.markdown.requisite}}
{{s.markdown.details}}
{%- endfor %}
"""
# example `salt://makereadme.sls`
{% import_text "makereadme.jinja" as makereadme %}
{{sls}} or:
file.managed:
- name: /root/README_other.md
- contents: {{salt.highstate_doc.render(jinja_template_text=makereadme)|json}}
- mode: '0640'
Some `replace_text_regex` values that might be helpful.
## CERTS
'-----BEGIN RSA PRIVATE KEY-----[
\S]{0,2200}': 'XXXXXXX'
'-----BEGIN CERTIFICATE-----[
\S]{0,2200}': 'XXXXXXX'
'-----BEGIN DH PARAMETERS-----[
\S]{0,2200}': 'XXXXXXX'
'-----BEGIN PRIVATE KEY-----[
\S]{0,2200}': 'XXXXXXX'
'-----BEGIN OPENSSH PRIVATE KEY-----[
\S]{0,2200}': 'XXXXXXX'
'ssh-rsa .* ': 'ssh-rsa XXXXXXX '
'ssh-dss .* ': 'ssh-dss XXXXXXX '
## DB
'DB_PASS.*': 'DB_PASS = XXXXXXX'
'5432:*:*:.*': '5432:*:XXXXXXX'
"'PASSWORD': .*": "'PASSWORD': 'XXXXXXX',"
" PASSWORD '.*'": " PASSWORD 'XXXXXXX'"
'PGPASSWORD=.* ': 'PGPASSWORD=XXXXXXX'
"_replication password '.*'": "_replication password 'XXXXXXX'"
## OTHER
'EMAIL_HOST_PASSWORD =.*': 'EMAIL_HOST_PASSWORD =XXXXXXX'
"net ads join -U '.*@MFCFADS.MATH.EXAMPLE.CA.* ": "net ads join -U '.*@MFCFADS.MATH.EXAMPLE.CA%XXXXXXX "
"net ads join -U '.*@NEXUS.EXAMPLE.CA.* ": "net ads join -U '.*@NEXUS.EXAMPLE.CA%XXXXXXX "
'install-uptrack .* --autoinstall': 'install-uptrack XXXXXXX --autoinstall'
'accesskey = .*': 'accesskey = XXXXXXX'
'auth_pass .*': 'auth_pass XXXXXXX'
'PSK "0x.*': 'PSK "0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
'SECRET_KEY.*': 'SECRET_KEY = XXXXXXX'
"password=.*": "password=XXXXXXX"
'<password>.*</password>': '<password>XXXXXXX</password>'
'<salt>.*</salt>': '<salt>XXXXXXX</salt>'
'application.secret = ".*"': 'application.secret = "XXXXXXX"'
'url = "postgres://.*"': 'url = "postgres://XXXXXXX"'
'PASS_.*_PASS': 'PASS_XXXXXXX_PASS'
## HTACCESS
':{PLAIN}.*': ':{PLAIN}XXXXXXX'
i ( t absolute_importt print_functiont unicode_literalsNu
highstate_docu�
{% for s in lowstates %}
`{{s.id_full}}`
-----------------------------------------------------------------
* state: {{s.state_function}}
* name: `{{s.name}}`
{{s.markdown.requisites}}
{{s.markdown.details}}
{%- endfor %}
u�
Configuration Managment
===============================================================================
```
####################################################
fqdn: {{grains.get('fqdn')}}
os: {{grains.get('os')}}
osfinger: {{grains.get('osfinger')}}
mem_total: {{grains.get('mem_total')}}MB
num_cpus: {{grains.get('num_cpus')}}
ipv4: {{grains.get('ipv4')}}
master: {{opts.get('master')}}
####################################################
```
This system is fully or partly managed using Salt.
The following sections are a rendered view of what the configuration management system
controlled on this system. Each item is handled in order from top to bottom unless some
requisites like `require` force other ordering.
u�
{% if vars.get('doc_other', True) -%}
Other information
=====================================================================================
```
salt grain: ip_interfaces
-----------------------------------------------------------------
{{grains['ip_interfaces']|dictsort}}
salt grain: hwaddr_interfaces
-----------------------------------------------------------------
{{grains['hwaddr_interfaces']|dictsort}}
{% if not grains['os'] == 'Windows' %}
{% if salt['cmd.has_exec']('ip') -%}
# ip address show
-----------------------------------------------------------------
{{salt['cmd.run']('ip address show | sed "/valid_lft/d"')}}
# ip route list table all
-----------------------------------------------------------------
{{salt['cmd.run']('ip route list table all')}}
{% endif %}
{% if salt['cmd.has_exec']('iptables') %}
{%- if salt['cmd.has_exec']('iptables-save') -%}
# iptables-save
-----------------------------------------------------------------
{{salt['cmd.run']("iptables --list > /dev/null; iptables-save | \grep -v -F '#' | sed '/^:/s@\[[0-9]\{1,\}:[0-9]\{1,\}\]@[0:0]@g'")}}
# ip6tables-save
-----------------------------------------------------------------
{{salt['cmd.run']("ip6tables --list > /dev/null; ip6tables-save | \grep -v -F '#' | sed '/^:/s@\[[0-9]\{1,\}:[0-9]\{1,\}\]@[0:0]@g'")}}
{%- else -%}
# iptables --list-rules
-----------------------------------------------------------------
{{salt['cmd.run']('iptables --list-rules')}}
# ip6tables --list-rules
-----------------------------------------------------------------
{{salt['cmd.run']('ip6tables --list-rules')}}
{% endif %}
{% endif %}
{% if salt['cmd.has_exec']('firewall-cmd') -%}
# firewall-cmd --list-all
-----------------------------------------------------------------
{{salt['cmd.run']('firewall-cmd --list-all')}}
{% endif %}
# mount
-----------------------------------------------------------------
{{salt['cmd.run']('mount')}}
{% endif %}
c K@ s t S( u�
Return text for a simple markdown jinja template
This function can be used from the `highstate_doc.render` modules `jinja_template_function` option.
( t! markdown_basic_jinja_template_txt( t kwargs( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt markdown_basic_jinja_templateR s c K@ s t S( u�
Return text for a markdown jinja template that included a header
This function can be used from the `highstate_doc.render` modules `jinja_template_function` option.
( t# markdown_default_jinja_template_txt( R ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt markdown_default_jinja_template[ s c K@ s t S( u�
Return text for an advanced markdown jinja template
This function can be used from the `highstate_doc.render` modules `jinja_template_function` option.
( t$ markdown_advanced_jinja_template_txt( R ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt markdown_full_jinja_templated s c K@ s� i d g d 6g d 6i d 6d d 6d d 6d
d 6} d
t � k rn d j t � } | j t d | i � � n x8 t | j � � t | j � � @D] } | | | | <q� W| S( u
Return configuration
u .*!doc_skipu filter_id_regexu filter_function_regexu replace_text_regexu! highstate_doc.proccesser_markdownu
proccesseri' u max_render_file_sizeu noteu __salt__u
{0}.configu
config.getN( t Nonet globalst formatt __virtualname__t updatet __salt__t sett keys( R t configt
config_keyt k( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _get_configm s
)c C@ sn d } y@ t j j j | d � �" } t j j j | j � � } Wd QXWn! t k
ri } t j | � d SX| S( ue
output the contents of a file:
this is a workaround if the cp.push module does not work.
https://github.com/saltstack/salt/issues/37133
help the master output the contents of a document
that might be saved on the minions filesystem.
.. code-block:: python
#!/bin/python
import os
import salt.client
s = salt.client.LocalClient()
o = s.cmd('*', 'highstate_doc.read_file', ['/root/README.md'])
for m in o:
d = o.get(m)
if d and not d.endswith('is not available.'):
# mkdir m
#directory = os.path.dirname(file_path)
if not os.path.exists(m):
os.makedirs(m)
with open(m + '/README.md','wb') as fin:
fin.write(d)
print('ADDED: ' + m + '/README.md')
u u rN( t saltt utilst filest fopent stringutilst
to_unicodet readt Exceptiont logt errorR
( t namet outt ft ex( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt read_file� s %
u- highstate_doc.markdown_default_jinja_templatec
K@ s� t | � } t | � } i d d 6| d 6| d 6t d 6t d 6t d 6t d 6| d 6} | } | d k r� | r� t | | � } n | d k r� t d � � n t j | | d
d �} | j
d � } x' | D] } t j | | | | � } q� W| S(
u�
Render highstate to a text format (default Markdown)
if `jinja_template_text` is not set, `jinja_template_function` is used.
jinja_template_text: jinja text that the render uses to create the document.
jinja_template_function: a salt module call that returns template text.
options:
highstate_doc.markdown_basic_jinja_template
highstate_doc.markdown_default_jinja_template
highstate_doc.markdown_full_jinja_template
u saltenvu configu lowstatesu saltu pillaru grainsu optsu kwargsu No jinja template textt tmplpathu replace_text_regexN(
R t proccess_lowstatesR
R t
__pillar__t
__grains__t __opts__R t tplt render_jinja_tmplt gett ret sub(
t jinja_template_textt jinja_template_functionR R t lowstatest contextt
template_textt txtt rtt r( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt render� s*
c C@ s� | d } | d } d j | d | d � } x( | d D] } t j | | � r9 t Sq9 Wx, | d D] } t j | | d � rd t Sqd Wt S( Nu stateu funu {0}.{1}u filter_function_regexu filter_id_regexu __id__( R R- t matcht Truet False( t sR t sst sft state_functiont b( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _blacklist_filter� s
c K@ s� g } t | � } | j d � } t d � } t | t � sL t d � � n7 t | � d k r� t | d t � s� t d � � q� n xF | D]> } t | | � r� q� n t | | | | � } | j | � q� W| S( u�
return proccessed lowstate data that was not blacklisted
render_module_function is used to provide your own.
defaults to from_lowstate
u
proccesseru state.show_lowstateuL ERROR: to see details run: [salt-call state.show_lowstate] <-----***-SEE-***i (
R R, R t
isinstancet listR t lent dictR@ t append( R t statesR t
proccessert lsR; t doc( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyR& � s
c C@ s� i } | d k r? d d d d d d d d d d
d d g } n t | j � � } | rj | t | � 8} n | r� | t | � M} n x | D] } | | | | <q� Wt | � d
k r� d St j j j | d t �S( u3
return a data dict in yaml string format.
u __env__u __id__u __sls__u funu nameu contextu orderu stateu requireu
require_inu watchu watch_ini t default_flow_styleN( R
R R RC R R t yamlt safe_dumpR: ( t datat whitelistt blacklistt yt ksetR ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _state_data_to_yaml_string� s -
c C@ s | j d d � S( uM
sanitize text data that is to be displayed in a markdown code block
u ```u ``[`][markdown parse fix]( t replace( t text( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _md_fix s c C@ sm d } t d | � } t | d d d d d d d g �} | rY | d
j | | � 7} n | j d � } | | j d � k rAt } y2 t d d
j | � � } d | k r� t } n Wn t k
r� } t } n X| r� d } n< t j j j
| d � �" }
t j j j |
j
� � } Wd QXt | � } | d j | | � 7} n( | d j d j d j t � | � � 7} | S( Nu u
file.statsRN u useru groupu modeu uidu gidu sizeu file stat {1}
```
{0}```
u max_render_file_sizeu cmd.shellu \file -i '{0}'u charset=binaryu [[skipped binary data]]u ru file data {1}
```
{0}
```
u ```
{0}
```
uA SKIPPED LARGE FILE!
Set {0}:max_render_file_size > {1} to render.u
{0}.config( R RR R R, R9 R: R R R R R R R R RU R
( t filenameR t rett
file_statsRP t file_sizet is_binaryt file_typeR# t file_dataR" ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _format_markdown_system_file s, $
! (c C@ sE | } d } x | D] } | j | d � } q W| j d d � } | S( Nu ~`!@#$%^&*()+={}[]:;"<>,.?/|'\u u u -( RS ( R t linkt symbalsR; ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _format_markdown_link0 s
c C@ s? d j | | � } | r. d j | t | � � Sd j | � Sd S( u4
format requisite as a link users can click
u {0}: {1}u * [{0}](#{1})
u * `{0}`
N( R R` ( t statet stateidt makelinkt fmt_id( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt _format_markdown_requisite9 s c K@ s4 | } d j | d | d � } d j | d | d � } d } | j d � r� | d 7} xI | j d g � D]5 } | t | j � d d | j � d d
� 7} ql W| d 7} n | j d � r$| d
7} xI | j d g � D]5 } | t | j � d d | j � d d
� 7} q� W| d 7} n | j d � r�t | j d � � d k r�| d 7} xI | j d g � D]5 } | t | j � d d | j � d d
� 7} qkW| d 7} n | j d � r#| d 7} xI | j d g � D]5 } | t | j � d d | j � d d
� 7} q�W| d 7} n d } | d k r�d | k r[| d j | d � 7} n d | k r�t d | d � } | r�| d j | � 7} q�| d j d j | d � � 7} q�n | d k r�| j d | j d � � }
| d j |
� 7} n | d k r�| d 7} t | � } | r8| d j | � 7} n d | k r�t d d! | j d � d" d# � } t | � d$ k s�d% | k r�x8 | D] }
| t |
| � 7} q�Wq�| d& 7} | d' 7} q�| d( 7} | d) 7} | d* 7} n | d+ k r_| j d, � r | d- j t | d, � � 7} n | j d � r_d. | j d � } | d- j t | � � 7} q_n | d/ k r�| t | d | � 7} n t | � d k r�t | � } | r�| d0 j | � 7} q�n i | d1 6| d d 6| d d 6| d d2 6| d d3 6| d4 6| d5 6i | j d6 � d7 6| j d6 � d8 6d9 6} | S(: uw
Takes low state data and returns a dict of proccessed data
that is by default used in a jinja template when rendering a markdown highstate_doc.
This `lowstate_item_markdown` given a lowstate item, returns a dict like:
.. code-block:: yaml
vars: # the raw lowstate_item that was proccessed
id: # the 'id' of the state.
id_full: # combo of the state type and id "state: id"
state: # name of the salt state module
function: # name of the state function
name: # value of 'name:' passed to the salt state module
state_function: # the state name and function name
markdown: # text data to describe a state
requisites: # requisite like [watch_in, require_in]
details: # state name, parameters and other details like file contents
u {0}.{1}u stateu funu {0}: {1}u __id__u u watchu run or update after changes in:
i i u
u watch_inu after changes, run or update:
u requireu require:
u
require_inu
required in:
u highstate_doc.noteu contentsu
{0}
u sourceu cp.get_file_stru ERROR: opening {0}u
pkg.installedu pkgsu nameu
```
install: {0}
```
u file.recurseu recurse copy of files
u ```
{0}
```
u !doc_recurseu file.findt patht typeu fi
u !doc_recurse_forceu3 > Skipping because more than 10 files to display.
u: > HINT: to force include !doc_recurse_force in state id.
u7 > For more details review logs and Salt state files.
u^ > HINT: for improved docs use multiple file.managed states or file.archive, git.latest. etc.
uC > HINT: to force doc to show all files in path add !doc_recurse .
u file.blockreplaceu contentu/ ensure block of content is in file
```
{0}
```
u ** source: u file.managedu ```
{0}```
u varsu functionu idu id_fullu state_functionu utf-8u
requisitesu detailsu markdown(
R R, Re t itemsRC R RR R] RU t decode( t
lowstate_itemR R R; R> t id_fullt
requisitest wt detailsRT t pkgsRP t findfilesR" R6 ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt proccesser_markdownD s�
3
3
*
3
3
&
"
(& t __doc__t
__future__R R R R- t loggingt salt.utils.filesR t salt.utils.stringutilst salt.utils.templatesR t templatesR* t salt.utils.yamlR
t getLoggert __name__R R R R R R R R R$ R
R7 R@ R& RR RU R] R` R9 Re Rq ( ( ( s> /usr/lib/python2.7/site-packages/salt/modules/highstate_doc.pyt <module>� s6
? &(
Mini Shell