Current File : //usr/lib/python2.7/site-packages/salt/returners/mysql.pyo
�
���^c @@ s d Z d d l m Z m Z m Z d d l m Z d d l Z d d l Z d d l Z
d d l Z
d d l Z
d d l
Z
d d l m Z y8 d d l Z d d l Z d d l Z d d l m Z Wny e k
r=yN d d l Z e j � d d l Z d d l Z d d l Z d d l m Z Wq>e k
r9d Z q>Xn Xe j e � Z d Z d � Z d d � Z e d e d
� � Z! d � Z" d � Z# d d
� Z$ d d � Z% d � Z& d � Z' d � Z( d � Z) e* d � Z+ d � Z, e d d � Z- d � Z. d � Z/ d � Z0 d S( u�
Return data to a mysql server
:maintainer: Dave Boucha <dave@saltstack.com>, Seth House <shouse@saltstack.com>
:maturity: mature
:depends: python-mysqldb
:platform: all
To enable this returner, the minion will need the python client for mysql
installed and the following values configured in the minion or master
config. These are the defaults:
.. code-block:: yaml
mysql.host: 'salt'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
SSL is optional. The defaults are set to None. If you do not want to use SSL,
either exclude these options or set them to None.
.. code-block:: yaml
mysql.ssl_ca: None
mysql.ssl_cert: None
mysql.ssl_key: None
Alternative configuration values can be used by prefacing the configuration
with `alternative.`. Any values not found in the alternative configuration will
be pulled from the default location. As stated above, SSL configuration is
optional. The following ssl options are simply for illustration purposes:
.. code-block:: yaml
alternative.mysql.host: 'salt'
alternative.mysql.user: 'salt'
alternative.mysql.pass: 'salt'
alternative.mysql.db: 'salt'
alternative.mysql.port: 3306
alternative.mysql.ssl_ca: '/etc/pki/mysql/certs/localhost.pem'
alternative.mysql.ssl_cert: '/etc/pki/mysql/certs/localhost.crt'
alternative.mysql.ssl_key: '/etc/pki/mysql/certs/localhost.key'
Should you wish the returner data to be cleaned out every so often, set
`keep_jobs` to the number of hours for the jobs to live in the tables.
Setting it to `0` or leaving it unset will cause the data to stay in the tables.
Should you wish to archive jobs in a different table for later processing,
set `archive_jobs` to True. Salt will create 3 archive tables
- `jids_archive`
- `salt_returns_archive`
- `salt_events_archive`
and move the contents of `jids`, `salt_returns`, and `salt_events` that are
more than `keep_jobs` hours old to these tables.
Use the following mysql database schema:
.. code-block:: sql
CREATE DATABASE `salt`
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_general_ci;
USE `salt`;
--
-- Table structure for table `jids`
--
DROP TABLE IF EXISTS `jids`;
CREATE TABLE `jids` (
`jid` varchar(255) NOT NULL,
`load` mediumtext NOT NULL,
UNIQUE KEY `jid` (`jid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE INDEX jid ON jids(jid) USING BTREE;
--
-- Table structure for table `salt_returns`
--
DROP TABLE IF EXISTS `salt_returns`;
CREATE TABLE `salt_returns` (
`fun` varchar(50) NOT NULL,
`jid` varchar(255) NOT NULL,
`return` mediumtext NOT NULL,
`id` varchar(255) NOT NULL,
`success` varchar(10) NOT NULL,
`full_ret` mediumtext NOT NULL,
`alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
KEY `id` (`id`),
KEY `jid` (`jid`),
KEY `fun` (`fun`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- Table structure for table `salt_events`
--
DROP TABLE IF EXISTS `salt_events`;
CREATE TABLE `salt_events` (
`id` BIGINT NOT NULL AUTO_INCREMENT,
`tag` varchar(255) NOT NULL,
`data` mediumtext NOT NULL,
`alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
`master_id` varchar(255) NOT NULL,
PRIMARY KEY (`id`),
KEY `tag` (`tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Required python modules: MySQLdb
To use the mysql returner, append '--return mysql' to the salt command.
.. code-block:: bash
salt '*' test.ping --return mysql
To use the alternative configuration, append '--return_config alternative' to the salt command.
.. versionadded:: 2015.5.0
.. code-block:: bash
salt '*' test.ping --return mysql --return_config alternative
To override individual configuration items, append --return_kwargs '{"key:": "value"}' to the salt command.
.. versionadded:: 2016.3.0
.. code-block:: bash
salt '*' test.ping --return mysql --return_kwargs '{"db": "another-salt"}'
i ( t absolute_importt print_functiont unicode_literals( t contextmanagerN( t six( t OperationalErroru mysqlc C@ s" t t � t d k r d n d f S( u:
Confirm that a python mysql client is installed.
u! No python mysql client installed.u N( t boolt MySQLdbt None( ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt __virtual__� s c
C@ s i d d 6d d 6d d 6d d 6d d 6d d 6d d 6d d
6} i d d 6d d 6d d 6d d 6d d 6d d 6d d 6d
d
6} t j j t | | d t d t d
| �} xm t j | � D]\ \ } } t | t j
� r� | j � d k r� d | | <n | d k r� t | � | | <q� q� W| S( u8
Returns options used for the MySQL connection.
u saltu hostu useru passu dbi� u portu ssl_cau ssl_certu ssl_keyt __salt__t __opts__t defaultsu noneN(
R t saltt returnerst get_returner_optionst __virtualname__R
R R t iteritemst
isinstancet string_typest lowert int( t retR t attrst _optionst kt v( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt _get_options� s8
$
c
c@ sD t | � } t } t rx d t k rx y+ t j d � t d } | j � t } Wqx t k
rt } t j d | � qx Xn | r�t j d � y� i } | j d � r� | j d � | d <n | j d � r� | j d � | d <n | j d � r| j d � | d
<n t j
d | j d � d
| j d � d | j d � d | j d � d | j d � d | � } y | t d <Wn t k
r�n XWq�t k
r�} t j
j d j d | � � � q�Xn | j � } y | VWnK t j k
r } | j } t j j t j | � � | j d � | � n$ X| r3| j d � n
| j d � d S( u
Return a mysql cursor
u mysql_returner_connu% Trying to reuse MySQL connection poolu OperationalError on ping: %su$ Generating new MySQL connection poolu ssl_cau cau ssl_certu certu ssl_keyu keyt hostu hostt useru usert passwdu passt dbu dbt portu portt sslu3 MySQL returner could not connect to database: {exc}t excu ROLLBACKu COMMITN( R t Truet __context__t logt debugt pingt FalseR t getR t connectt TypeErrorR
t
exceptionst SaltMasterErrort formatt cursort
DatabaseErrort argst syst stderrt writeR t text_typet execute(
R t commitR R* t connR" t ssl_optionsR/ t errt error( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt _get_serv� sV
%
c
C@ s� | d d k rC t d | j d t � � | d <t | d | � n y� t | d t ��i } d } | j | | d | d t j j j
| d � | d | j d
t � t j j j
| � f � Wd QXWn3 t j j k
r� } t
j | � t
j d � n Xd S(
u'
Return data to a mysql server
u jidu reqt nocacheu nocacheR7 u� INSERT INTO `salt_returns`
(`fun`, `jid`, `return`, `id`, `success`, `full_ret`)
VALUES (%s, %s, %s, %s, %s, %s)u funu returnu idu successNuE Could not store return with MySQL returner. MySQL server unavailable.( t prep_jidR) R( t save_loadR< R# R6 R
t utilst jsont dumpsR, R- R% t critical( R t curt sqlR" ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt returner! s #
c C@ s� t | d t ��n } xd | D]\ } | j d d � } | j d d � } d } | j | | t j j j | � t d f � q WWd QXd S( u�
Return event to mysql server
Requires that configuration be enabled via 'event_return'
option in master config.
R7 u tagu u datau_ INSERT INTO `salt_events` (`tag`, `data`, `master_id`)
VALUES (%s, %s, %s)u idN( R< R# R) R6 R
R@ RA RB R ( t eventsRD t eventt tagt dataRE ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt event_return: s
c C@ sb t d t � �M } d } y) | j | | t j j j | � f � Wn t j k
rW n XWd QXd S( u/
Save the load to the specified jid id
R7 u2 INSERT INTO `jids` (`jid`, `load`) VALUES (%s, %s)N( R< R# R6 R
R@ RA RB R t IntegrityError( t jidt loadt minionsRD RE ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyR? J s )c C@ s d S( u&
Included for API consistency
N( ( RM RO t syndic_id( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt save_minions[ s c C@ sh t d d d t � �M } d } | j | | f � | j � } | rZ t j j j | d � Si SWd QXd S( u9
Return the load data that marks a specified jid
R R7 u+ SELECT `load` FROM `jids` WHERE `jid` = %s;i N( R< R R# R6 t fetchoneR
R@ RA t loads( RM RD RE RJ ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt get_loadb s c C@ s� t d d d t � �o } d } | j | | f � | j � } i } | r| x0 | D]% \ } } t j j j | � | | <qP Wn | SWd QXd S( uP
Return the information returned when the specified job id was executed
R R7 uH SELECT id, full_ret FROM `salt_returns`
WHERE `jid` = %sN( R< R R# R6 t fetchallR
R@ RA RS ( RM RD RE RJ R t miniont full_ret( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt get_jidp s c C@ s� t d d d t � �r } d } | j | | f � | j � } i } | r x3 | D]( \ } } } t j j j | � | | <qP Wn | SWd QXd S( uC
Return a dict of the last function called for all minions
R R7 u SELECT s.id,s.jid, s.full_ret
FROM `salt_returns` s
JOIN ( SELECT MAX(`jid`) as jid
from `salt_returns` GROUP BY fun, id) max
ON s.jid = max.jid
WHERE s.fun = %s
N( R< R R# R6 RU R
R@ RA RS ( t funRD RE RJ R RV t _RW ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt get_fun� s c
C@ s� t d d d t � �x } d } | j | � | j � } i } xE | D]= } t j j j | d t j j j
| d � � | | d <qD W| SWd QXd S( u&
Return a list of all job ids
R R7 u9 SELECT DISTINCT `jid`, `load`
FROM `jids`i i N( R< R R# R6 RU R
R@ RM t format_jid_instanceRA RS ( RD RE RJ R RM ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt get_jids� s
(c C@ s� t d d d t � �� } d } d } | j | j | r< | n d | � � | j � } g } xD | D]< } | j t j j j
| d t j j j | d � � � qb W| SWd QXd S( u�
Return a list of all job ids
:param int count: show not more than the count of most recent jobs
:param bool filter_find_jobs: filter out 'saltutil.find_job' jobs
R R7 u� SELECT * FROM (
SELECT DISTINCT `jid` ,`load` FROM `jids`
{0}
ORDER BY `jid` DESC limit {1}
) `tmp`
ORDER BY `jid`;u5 WHERE `load` NOT LIKE '%"fun": "saltutil.find_job"%' u i i N(
R< R R# R6 R. RU t appendR
R@ RM t format_jid_instance_extRA RS ( t countt filter_find_jobRD RE t whereRJ R RM ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt get_jids_filter� s %
!c C@ sm t d d d t � �R } d } | j | � | j � } g } x | D] } | j | d � qD W| SWd QXd S( u"
Return a list of minions
R R7 u6 SELECT DISTINCT id
FROM `salt_returns`i N( R< R R# R6 RU R^ ( RD RE RJ R RV ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt get_minions� s
c C@ s# | d k r | St j j j t � S( uO
Do any work necessary to prepare a JID, including sending a custom id
N( R R
R@ RM t gen_jidR ( R= t
passed_jid( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyR> � s c C@ s� t � ��} y* d } | j | | f � | j d � WnT t j k
r� } t j d � t j t j | � � t j j
t j | � � � n Xy* d } | j | | f � | j d � WnT t j k
r
} t j d � t j t j | � � t j j
t j | � � � n Xy* d } | j | | f � | j d � WnT t j k
r�} t j d � t j t j | � � t j j
t j | � � � n XWd QXt S( uy
Purge records from the returner tables.
:param job_age_in_seconds: Purge jobs older than this
:return:
u] delete from `jids` where jid in (select distinct jid from salt_returns where alter_time < %s)u COMMITuE mysql returner archiver was unable to delete contents of table 'jids'u0 delete from `salt_returns` where alter_time < %suM mysql returner archiver was unable to delete contents of table 'salt_returns'u/ delete from `salt_events` where alter_time < %suL mysql returner archiver was unable to delete contents of table 'salt_events'N( R< R6 R t ErrorR% R; R R5 R
R, t SaltRunnerErrorR# ( t timestampRD RE t e( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt _purge_jobs� s4
%c
C@ s� d d d g } t � ��} i } x� | D]� } yD | d } d j | | � } | j | � | j d � | | | <Wq( t j k
r� } t j d � t j t j | � � t j
j t j | � � � q( Xq( Wy: d j | d d � } | j | | f � | j d � Wnv t j k
r]} t j d � t j t j | � � t j
j t j | � � � n# t k
r} t j | � � n Xy: d
j | d d � } | j | | f � | j d � WnT t j k
r} t j d � t j t j | � � t j
j t j | � � � n Xy: d
j | d d � } | j | | f � | j d � WnT t j k
r�} t j d � t j t j | � � t j
j t j | � � � n XWd
QXt
| � S( u�
Copy rows to a set of backup tables, then purge rows.
:param timestamp: Archive rows older than this timestamp
:return:
u jidsu salt_returnsu salt_eventsu _archiveu' create table if not exists {0} like {1}u COMMITu@ mysql returner archiver was unable to create the archive tables.up insert into `{0}` select * from `{1}` where jid in (select distinct jid from salt_returns where alter_time < %s)uC mysql returner archiver was unable to copy contents of table 'jids'u; insert into `{0}` select * from `{1}` where alter_time < %suK mysql returner archiver was unable to copy contents of table 'salt_returns'uJ mysql returner archiver was unable to copy contents of table 'salt_events'N( R< R. R6 R Rg R% R; R R5 R
R, Rh t ExceptionRk ( Ri t
source_tablesRD t
target_tablest
table_namet tmp_table_nameRE Rj ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt
_archive_jobs sX
#
%c C@ s t j d t � rt t j d d � � d k ryy t � �A } d j t d � } | j | � | j � } | d d } Wd QXt j d t � r� t | � n
t | � Wqt
j k
r� } t j
d � t j
t j | � � t j j t j | � � � qXn d S( u�
Called in the master's event loop every loop_interval. Archives and/or
deletes the events and job details from the database.
:return:
u keep_jobsi u3 select date_sub(now(), interval {0} hour) as stamp;Nu archive_jobsuD Mysql returner was unable to get timestamp for purge/archive of jobs( R R) R( R R< R. R6 RU Rq Rk R Rg R% R; R R5 R
R, Rh ( RD RE t rowst stampRj ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt clean_old_jobs= s 0
(1 t __doc__t
__future__R R R t
contextlibR R2 t loggingt salt.returnersR
t salt.utils.jidt salt.utils.jsont salt.exceptionst salt.extR R t MySQLdb.cursorst MySQLdb.converterst MySQLdb.connectionsR t ImportErrort pymysqlt install_as_MySQLdbt MySQLdb.errR t getLoggert __name__R% R R R R( R< RF RK R? RQ RT RX R[ R] R# Rc Rd R> Rk Rq Rt ( ( ( s8 /usr/lib/python2.7/site-packages/salt/returners/mysql.pyt <module>� sX
(: % 9
Mini Shell