Current File : //usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyc
�
���^c @@ s� d Z d d l m Z m Z m Z d d l m Z d d l Z d d l Z d d l Z d d l
Z d d l Z d d l
Z d d l m Z y" d d l Z d d l Z e Z Wn e k
r� e Z n Xe j e � Z d Z d a d � Z d d � Z e d e d
� � Z d � Z d � Z d d
� Z! d d � Z" d � Z# d � Z$ d � Z% d � Z& d � Z' e d d � Z( d � Z) d � Z* d � Z+ d S( u�
Return data to a PostgreSQL server with json data stored in Pg's jsonb data type
:maintainer: Dave Boucha <dave@saltstack.com>, Seth House <shouse@saltstack.com>, C. R. Oldham <cr@saltstack.com>
:maturity: Stable
:depends: python-psycopg2
:platform: all
.. note::
There are three PostgreSQL returners. Any can function as an external
:ref:`master job cache <external-job-cache>`. but each has different
features. SaltStack recommends
:mod:`returners.pgjsonb <salt.returners.pgjsonb>` if you are working with
a version of PostgreSQL that has the appropriate native binary JSON types.
Otherwise, review
:mod:`returners.postgres <salt.returners.postgres>` and
:mod:`returners.postgres_local_cache <salt.returners.postgres_local_cache>`
to see which module best suits your particular needs.
To enable this returner, the minion will need the python client for PostgreSQL
installed and the following values configured in the minion or master
config. These are the defaults:
.. code-block:: yaml
returner.pgjsonb.host: 'salt'
returner.pgjsonb.user: 'salt'
returner.pgjsonb.pass: 'salt'
returner.pgjsonb.db: 'salt'
returner.pgjsonb.port: 5432
SSL is optional. The defaults are set to None. If you do not want to use SSL,
either exclude these options or set them to None.
.. code-block:: yaml
returner.pgjsonb.sslmode: None
returner.pgjsonb.sslcert: None
returner.pgjsonb.sslkey: None
returner.pgjsonb.sslrootcert: None
returner.pgjsonb.sslcrl: None
.. versionadded:: 2017.5.0
Alternative configuration values can be used by prefacing the configuration
with `alternative.`. Any values not found in the alternative configuration will
be pulled from the default location. As stated above, SSL configuration is
optional. The following ssl options are simply for illustration purposes:
.. code-block:: yaml
alternative.pgjsonb.host: 'salt'
alternative.pgjsonb.user: 'salt'
alternative.pgjsonb.pass: 'salt'
alternative.pgjsonb.db: 'salt'
alternative.pgjsonb.port: 5432
alternative.pgjsonb.ssl_ca: '/etc/pki/mysql/certs/localhost.pem'
alternative.pgjsonb.ssl_cert: '/etc/pki/mysql/certs/localhost.crt'
alternative.pgjsonb.ssl_key: '/etc/pki/mysql/certs/localhost.key'
Should you wish the returner data to be cleaned out every so often, set
``keep_jobs`` to the number of hours for the jobs to live in the tables.
Setting it to ``0`` or leaving it unset will cause the data to stay in the tables.
Should you wish to archive jobs in a different table for later processing,
set ``archive_jobs`` to True. Salt will create 3 archive tables;
- ``jids_archive``
- ``salt_returns_archive``
- ``salt_events_archive``
and move the contents of ``jids``, ``salt_returns``, and ``salt_events`` that are
more than ``keep_jobs`` hours old to these tables.
.. versionadded:: 2019.2.0
Use the following Pg database schema:
.. code-block:: sql
CREATE DATABASE salt
WITH ENCODING 'utf-8';
--
-- Table structure for table `jids`
--
DROP TABLE IF EXISTS jids;
CREATE TABLE jids (
jid varchar(255) NOT NULL primary key,
load jsonb NOT NULL
);
CREATE INDEX idx_jids_jsonb on jids
USING gin (load)
WITH (fastupdate=on);
--
-- Table structure for table `salt_returns`
--
DROP TABLE IF EXISTS salt_returns;
CREATE TABLE salt_returns (
fun varchar(50) NOT NULL,
jid varchar(255) NOT NULL,
return jsonb NOT NULL,
id varchar(255) NOT NULL,
success varchar(10) NOT NULL,
full_ret jsonb NOT NULL,
alter_time TIMESTAMP WITH TIME ZONE DEFAULT NOW());
CREATE INDEX idx_salt_returns_id ON salt_returns (id);
CREATE INDEX idx_salt_returns_jid ON salt_returns (jid);
CREATE INDEX idx_salt_returns_fun ON salt_returns (fun);
CREATE INDEX idx_salt_returns_return ON salt_returns
USING gin (return) with (fastupdate=on);
CREATE INDEX idx_salt_returns_full_ret ON salt_returns
USING gin (full_ret) with (fastupdate=on);
--
-- Table structure for table `salt_events`
--
DROP TABLE IF EXISTS salt_events;
DROP SEQUENCE IF EXISTS seq_salt_events_id;
CREATE SEQUENCE seq_salt_events_id;
CREATE TABLE salt_events (
id BIGINT NOT NULL UNIQUE DEFAULT nextval('seq_salt_events_id'),
tag varchar(255) NOT NULL,
data jsonb NOT NULL,
alter_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
master_id varchar(255) NOT NULL);
CREATE INDEX idx_salt_events_tag on
salt_events (tag);
CREATE INDEX idx_salt_events_data ON salt_events
USING gin (data) with (fastupdate=on);
Required python modules: Psycopg2
To use this returner, append '--return pgjsonb' to the salt command.
.. code-block:: bash
salt '*' test.ping --return pgjsonb
To use the alternative configuration, append '--return_config alternative' to the salt command.
.. versionadded:: 2015.5.0
.. code-block:: bash
salt '*' test.ping --return pgjsonb --return_config alternative
To override individual configuration items, append --return_kwargs '{"key:": "value"}' to the salt command.
.. versionadded:: 2016.3.0
.. code-block:: bash
salt '*' test.ping --return pgjsonb --return_kwargs '{"db": "another-salt"}'
i ( t absolute_importt print_functiont unicode_literals( t contextmanagerN( t sixu pgjsonbu7 INSERT INTO jids (jid, load) VALUES (%(jid)s, %(load)s)c C@ s t s t d f St S( NuD Could not import pgjsonb returner; python-psycopg2 is not installed.( t HAS_PGt Falset True( ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt __virtual__� s
c
C@ s� i d d 6d d 6d d 6d d 6d d 6} i
d d 6d d 6d d 6d d 6d d 6d d 6d
d
6d d 6d d 6d
d
6} t j j d j t � | | d t d t d | �} d | k r� t | d � | d <n | S( u8
Returns options used for the MySQL connection.
u localhostu hostu saltu useru passu dbi8 u portu sslmodeu sslcertu sslkeyu sslrootcertu sslcrlu returner.{0}t __salt__t __opts__t defaults( t saltt returnerst get_returner_optionst formatt __virtualname__R R
t int( t retR t attrst _options( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt _get_options� s4
c c@ s} t | � } yw d � t j | � D� } t j d | j d � d | j d � d | j d � d | j d � d
| j d � | � } Wn4 t j k
r� } t j j d j
d
| � � � n X| j d k r� | j d k r� d a
n | j � } z~ y | VWnK t j k
rF} | j } t j j t j | � � | j d � | � n$ X| r]| j d � n
| j d � Wd | j � Xd S( u
Return a Pg cursor
c S@ s+ i | ]! \ } } | d k r | | � q S( u sslmodeu sslcertu sslkeyu sslrootcertu sslcrl( u sslmodeu sslcertu sslkeyu sslrootcertu sslcrl( ( t .0t kt v( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pys
<dictcomp>� s t hostu hostt portu portt dbnameu dbt useru usert passwordu passu5 pgjsonb returner could not connect to database: {exc}t exci�a u� INSERT INTO jids
(jid, load)
VALUES (%(jid)s, %(load)s)
ON CONFLICT (jid) DO UPDATE
SET load=%(load)su ROLLBACKu COMMITN( R R t iteritemst psycopg2t connectt gett OperationalErrorR t
exceptionst SaltMasterErrorR t server_versiont Nonet PG_SAVE_LOAD_SQLt cursort
DatabaseErrort argst syst stderrt writet text_typet executet close( R t commitR t ssl_optionst connR R) t errt error( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt _get_serv� s6
"
c
C@ s� y� t | d t ��l } d } | j | | d | d t j j | d � | d | j d t � t j j | � t j � f � Wd QXWn$ t j
j k
r� t j
d � n Xd S(
u$
Return data to a Pg server
R2 u� INSERT INTO salt_returns
(fun, jid, return, id, success, full_ret, alter_time)
VALUES (%s, %s, %s, %s, %s, %s, to_timestamp(%s))u funu jidu returnu idu successNuL Could not store return with pgjsonb returner. PostgreSQL server unavailable.( R7 R R0 R t extrast JsonR" R t timeR R$ R% t logt critical( R t curt sql( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt returner s c C@ s� t | d t ��t } xj | D]b } | j d d � } | j d d � } d } | j | | t j j | � t d t j � f � q WWd QXd S( u
Return event to Pg server
Requires that configuration be enabled via 'event_return'
option in master config.
R2 u tagu u datauu INSERT INTO salt_events (tag, data, master_id, alter_time)
VALUES (%s, %s, %s, to_timestamp(%s))u idN( R7 R R" R0 R R8 R9 R
R: ( t eventsR= t eventt tagt dataR> ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt event_return4 s
c C@ sa t d t � �L } y. | j t i | d 6t j j | � d 6� Wn t j k
rV n XWd QXd S( u/
Save the load to the specified jid id
R2 u jidu loadN( R7 R R0 R( R R8 R9 t IntegrityError( t jidt loadt minionsR= ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt save_loadE s %c C@ s d S( u&
Included for API consistency
N( ( RF RH t syndic_id( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt save_minionsT s c C@ sY t d d d t � �> } d } | j | | f � | j � } | rK | d Si SWd QXd S( u9
Return the load data that marks a specified jid
R R2 u% SELECT load FROM jids WHERE jid = %s;i N( R7 R' R R0 t fetchone( RF R= R> RC ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt get_load[ s c C@ s{ t d d d t � �` } d } | j | | f � | j � } i } | rm x! | D] \ } } | | | <qP Wn | SWd QXd S( uP
Return the information returned when the specified job id was executed
R R2 uD SELECT id, full_ret FROM salt_returns
WHERE jid = %sN( R7 R' R R0 t fetchall( RF R= R> RC R t miniont full_ret( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt get_jidi s c C@ s~ t d d d t � �c } d } | j | | f � | j � } i } | rp x$ | D] \ } } } | | | <qP Wn | SWd QXd S( uC
Return a dict of the last function called for all minions
R R2 u SELECT s.id,s.jid, s.full_ret
FROM salt_returns s
JOIN ( SELECT MAX(`jid`) as jid
from salt_returns GROUP BY fun, id) max
ON s.jid = max.jid
WHERE s.fun = %s
N( R7 R' R R0 RN ( t funR= R> RC R RO t _RP ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt get_fun{ s c C@ s~ t d d d t � �c } d } | j | � | j � } i } x0 | D]( \ } } t j j j | | � | | <qD W| SWd QXd S( u&
Return a list of all job ids
R R2 u* SELECT jid, load
FROM jidsN( R7 R' R R0 RN R t utilsRF t format_jid_instance( R= R> RC R RF RG ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt get_jids� s
c C@ sm t d d d t � �R } d } | j | � | j � } g } x | D] } | j | d � qD W| SWd QXd S( u"
Return a list of minions
R R2 u4 SELECT DISTINCT id
FROM salt_returnsi N( R7 R' R R0 RN t append( R= R> RC R RO ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt get_minions� s
c C@ s# | d k r | St j j j t � S( uO
Do any work necessary to prepare a JID, including sending a custom id
N( R' R RU RF t gen_jidR
( t nocachet
passed_jid( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt prep_jid� s c C@ s~ t � �o} y* d } | j | | f � | j d � WnK t j k
r� } | j } t j j t j | � � | j d � | � n Xy* d } | j | | f � | j d � WnK t j k
r� } | j } t j j t j | � � | j d � | � n Xy* d } | j | | f � | j d � WnK t j k
rs} | j } t j j t j | � � | j d � | � n XWd QXt
S( uy
Purge records from the returner tables.
:param job_age_in_seconds: Purge jobs older than this
:return:
u[ delete from jids where jid in (select distinct jid from salt_returns where alter_time < %s)u COMMITu ROLLBACKu. delete from salt_returns where alter_time < %su- delete from salt_events where alter_time < %sN( R7 R0 R R* R+ R, R- R. R R/ R ( t timestampR) R> R5 R6 ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt _purge_jobs� s:
c
C@ s� d d d g } t � �j} i } x� | D]� } yD | d } d j | | � } | j | � | j d � | | | <Wq( t j k
r� } | j } t j j t j
| � � | j d � | � q( Xq( Wy: d j | d d � } | j | | f � | j d � Wnm t j k
rK} | j } t j j t j
| � � | j d � | � n# t k
rm} t j
| � � n Xy: d j | d d � } | j | | f � | j d � WnK t j k
r�} | j } t j j t j
| � � | j d � | � n Xy: d j | d d � } | j | | f � | j d � WnK t j k
r}} | j } t j j t j
| � � | j d � | � n XWd
QXt | � S( u�
Copy rows to a set of backup tables, then purge rows.
:param timestamp: Archive rows older than this timestamp
:return:
u jidsu salt_returnsu salt_eventsu _archiveu) create table IF NOT exists {0} (LIKE {1})u COMMITu ROLLBACKul insert into {0} select * from {1} where jid in (select distinct jid from salt_returns where alter_time < %s)u7 insert into {0} select * from {1} where alter_time < %sN( R7 R R0 R R* R+ R, R- R. R R/ t ExceptionR; R6 R_ (
R^ t
source_tablesR) t
target_tablest
table_namet tmp_table_nameR> R5 R6 t e( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt
_archive_jobs� s`
c C@ s� t j d t � r� t t j d d � � d k r� yy t � �A } d j t d � } | j | � | j � } | d d } Wd QXt j d t � r� t | � n
t | � Wq� t
k
r� } t j | � q� Xn d S( u�
Called in the master's event loop every loop_interval. Archives and/or
deletes the events and job details from the database.
:return:
u keep_jobsi u/ select (NOW() - interval '{0}' hour) as stamp;Nu archive_jobs(
R
R" R R R7 R R0 RN Rf R_ R` R; R6 ( R= R> t rowst stampRe ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt clean_old_jobs! s 0
(, t __doc__t
__future__R R R t
contextlibR R, R: t loggingt salt.returnersR t salt.utils.jidt salt.exceptionst salt.extR R t psycopg2.extrasR R t ImportErrorR t getLoggert __name__R; R R( R R' R R7 R? RD RI RK RM RQ RT RW RY R] R_ Rf Ri ( ( ( s: /usr/lib/python2.7/site-packages/salt/returners/pgjsonb.pyt <module>� sF
%1 ( =
Mini Shell