Current File : //usr/lib/python2.7/site-packages/salt/utils/master.pyo
�
���^c @@ s d Z d d l m Z m Z d d l Z d d l Z d d l Z d d l m Z m Z d d l
Z d d l Z d d l
Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l Z d d l m Z d d l m Z d d l m Z d d l m! Z! e j" e# � Z$ d
� Z% d � Z& d � Z' d
e( f d � � YZ) d e f d � � YZ* d e f d � � YZ+ d e f d � � YZ, d � Z- e. d � Z/ d � Z0 e# d k r
e j1 j2 d � Z3 e, e3 � Z4 e4 j5 � n d S( ue
salt.utils.master
-----------------
Utilities that can only be used on a salt master.
i ( t absolute_importt unicode_literalsN( t Threadt Event( t
SaltException( t CacheCli( t MultiprocessingProcess( t six( t zmqc C@ s� g } t j j | d d � } t j j | � s5 | Sx� t j | � D]t } t j j | | � } y/ t | | � } | d k r� | j | � n WqE t t f k
r� t
j d | � qE XqE W| S( u0
Return the running jobs on this minion
u cachediru procu. %s removed during processing by master processN( t ost patht joint isdirt listdirt _read_proc_filet Nonet appendt IOErrort OSErrort logt trace( t optst rett proc_dirt fn_R
t data( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt get_running_jobs) s c C@ sp t j j | � } t j j j | d � �n } | j � } | j � | r[ | j | � } n9 y t j
| � Wn! t k
r� t j
d | � n Xd SWd QXt | t � s� d St j j j | d � s� y t j
| � Wn! t k
r� t j
d | � n Xd St | � sl| j d � } | r3t j d | � n y t j
| � Wn! t k
rgt j
d | � n Xd S| S( u0
Return a dict of JID metadata, or None
u rbu Unable to remove proc file %s.Nu pidu7 PID %s exists but does not appear to be a salt process.( t saltt payloadt Serialt utilst filest fopent readt closet loadsR t removeR R t debugR t
isinstancet dictt processt
os_is_runningt _check_cmdlinet gett warning( R
R t serialt fp_t bufR t pid( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR @ s<
c C@ s� t j j j � s t S| j d � } | s/ t St j j d � sE t St j j
d j | � � } t j j | � sv t Sy5 t j j
j | d � � } d | j � k SWd QXWn t t f k
r� t SXd S( u9
In some cases where there are an insane number of processes being created
on a system a PID can get recycled or assigned to a non-Salt process.
On Linux this fn checks to make sure the PID we are checking on is actually
a Salt process.
For non-Linux systems we punt and just return True
u pidu /procu /proc/{0}/cmdlineu rbR N( R R t platformt is_linuxt TrueR+ t FalseR R
R R t formatt isfileR R R! R R ( R R0 R
R. ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR* k s t MasterPillarUtilc B@ s� e Z d Z d d d e e e e d d � Z d � Z d � Z d � Z d d d � Z d � Z
d � Z d
� Z d � Z
d � Z d
� Z e e e d d � Z RS( u�
Helper utility for easy access to targeted minion grain and
pillar data, either from cached data on the master or retrieved
on demand, or (by default) both.
The minion pillar data returned in get_minion_pillar() is
compiled directly from salt.pillar.Pillar on the master to
avoid any possible 'pillar poisoning' from a compromised or
untrusted minion.
** However, the minion grains are still possibly entirely
supplied by the minion. **
Example use case:
For runner modules that need access minion pillar data,
MasterPillarUtil.get_minion_pillar should be used instead
of getting the pillar data by executing the "pillar" module
on the minions:
# my_runner.py
tgt = 'web*'
pillar_util = salt.utils.master.MasterPillarUtil(tgt, tgt_type='glob', opts=__opts__)
pillar_data = pillar_util.get_minion_pillar()
u u globc C@ s� t j d | j j � | d k rV t j d | j j � t d j | j j � � � n | | _ t j
j | j � | _ | | _
| | _ | | _ | | _ | | _ | | _ | | _ t j j | � | _ t j d | | | | | | | � d S( Nu New instance of %s created.u! %s: Missing master opts init arg.u" {0}: Missing master opts init arg.u� Init settings: tgt: '%s', tgt_type: '%s', saltenv: '%s', use_cached_grains: %s, use_cached_pillar: %s, grains_fallback: %s, pillar_fallback: %s( R R% t __class__t __name__R t errorR R5 R R R R R- t tgtt tgt_typet saltenvt use_cached_grainst use_cached_pillart grains_fallbackt pillar_fallbackt cachet factory( t selfR; R< R= R> R? R@ RA R ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt __init__� s*
c G@ s� t g | D] } | i f ^ q
� } | j j d t � rb | j j d t � rb t j d � | S| s} | j j d � } n xi | D]a } t j j
j | j | � s� q� n | j j d j
| � d � } t | t � r� | | | <q� q� W| S( Nu minion_data_cacheu enforce_mine_cacheuS Skipping cached mine data minion_data_cacheand enfore_mine_cache are both disabled.u minionsu minions/{0}u mine( R' R R+ R4 R R% RB t listR R t verifyt valid_idt fetchR5 R& ( RD t
minion_idst minion_idt mine_datat mdata( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _get_cached_mine_data� s %
c G@ s= t g | D] } | i f ^ q
� } | j � } | j j d t � s] t j d � | | f S| sx | j j d � } n x� | D]� } t j
j j | j | � s� q n | j j
d j | � d � } t | t � s� t j d t | � j | � q n d | k r| d | | <n d | k r | d | | <q q W| | f S( Nu minion_data_cacheu> Skipping cached data because minion_data_cache is not enabled.u minionsu minions/{0}u datauG cache.fetch should always return a dict. ReturnedType: %s, MinionId: %su grainsu pillar( R' t copyR R+ R4 R R% RB RF R R RG RH RI R5 R& R, t typeR9 ( RD RJ RK t grainst pillarsRM ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _get_cached_minion_data� s, %
c C@ s[ t j d | � t j j | j d � } | j d j | � d d | j d d d �} | S( Nu% Getting live grains for minions: "%s"u conf_fileu ,u grains.itemst timeoutu timeoutR< u list( R R% R t clientt get_local_clientR t cmdR ( RD RJ RU R ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _get_live_minion_grains� s
c C@ s� | d k r i S| s* t j d | � i St j d | � t j j | j | | | j | j d � } t j d | � | j � } | S( Nu2 Cannot get pillar data for %s: no grains supplied.u Getting live pillar for %su
ext_pillaru Compiling pillar for %s(
R R R, R% R t pillart PillarR R= t compile_pillar( RD RK t
minion_grainsRY R ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _get_live_minion_pillar� s" c
O@ s� i } | j d i � } i } i } | j rt g t j | � D] \ } } | r@ | | f ^ q@ � } g | D] } | | k rn | ^ qn } t j d | � | j r� | j | � } n t t t j t g | D] } | i f ^ q� � � � t | j
� � t | j
� � � } n� | j | � } g | D] } | | k r*| ^ q*} t j d | � | j r�t g t j | � D] \ } } | rt| | f ^ qt� } n t t t j t g | D] } | i f ^ q�� � � t | j
� � t | j
� � � } | S( Nu
cached_grainsu# Missed cached minion grains for: %su! Missed live minion grains for: %s( R+ R> R' R t iteritemsR R% R@ RX RF t items(
RD RJ t kwargsR t
cached_grainst crett lretRK t mcachet missed_minions( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _get_minion_grains s$ :% ]% =Zc O@ sj i } | j d i � } | j d i � } i } i } | j rTt g t j | � D] \ } } | rR | | f ^ qR � } g | D] } | | k r� | ^ q� }
t j d |
� | j r� t g |
D]* } | | j | | j | i � � f ^ q� � } n t t t j t g | D] } | i f ^ q
� � � t | j
� � t | j
� � � } nt g | D]* } | | j | | j | i � � f ^ q^� } g | D] } | | k r�| ^ q�}
t j d |
� | j r t g t j | � D] \ } } | r�| | f ^ q�� } n t t t j t g | D] } | i f ^ q"� � � t | j
� � t | j
� � � } | S( Nu grainsu
cached_pillaru$ Missed cached minion pillars for: %su" Missed live minion pillars for: %s( R+ R? R' R R^ R R% RA R] RF R_ ( RD RJ R` R RQ t
cached_pillarRb Rc RK Rd Re ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _get_minion_pillar( s& :% @]=% =Zc C@ s� g } t j j j | j � } | j | j | j � } | d } t | � d k ro t j
d | j | j � i St j
d | j | j | � | S( Nu minionsi u1 No minions matched for tgt="%s" and tgt_type="%s"u3 Matching minions for tgt="%s" and tgt_type="%s": %s( R R t minionst CkMinionsR t
check_minionsR; R< t lenR R% ( RD RJ t ckminionst _res( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt _tgt_to_listA s
c C@ s� i } i } | j � } t d � | j | j | j | j g D� � rk t j d � | j | � \ } } n i } i } t j d | � | j d | | � } t j d | � | j
d | d | | � } | S( u�
Get pillar data for the targeted minions, either by fetching the
cached minion data on the master, or by compiling the minion's
pillar data on the master.
For runner modules that need access minion pillar data, this
function should be used instead of getting the pillar data by
executing the pillar module on the minions.
By default, this function tries hard to get the pillar data:
- Try to get the cached minion grains and pillar if the
master has minion_data_cache: True
- If the pillar data for the minion is cached, use it.
- If there is no cached grains/pillar data for a minion,
then try to get the minion grains directly from the minion.
- Use the minion grains to compile the pillar directly from the
master using salt.pillar.Pillar
c s@ s | ] } | Vq d S( N( ( t .0t arg( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pys <genexpr>c s u Getting cached minion datau! Getting minion grain data for: %sRa u" Getting minion pillar data for: %sRQ Rg ( Ro t anyR> R? R@ RA R R% RS Rf Rh ( RD t minion_pillarsR\ RJ t cached_minion_grainst cached_minion_pillars( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt get_minion_pillarM s .
c C@ s� i } | j � } | s i St d � | j | j g D� � rc t j d � | j | � \ } } n i } t j d | � | j d | | � } | S( u5
Get grains data for the targeted minions, either by fetching the
cached minion data on the master, or by fetching the grains
directly on the minion.
By default, this function tries hard to get the grains data:
- Try to get the cached minion grains if the master
has minion_data_cache: True
- If the grains data for the minion is cached, use it.
- If there is no cached grains data for a minion,
then try to get the minion grains directly from the minion.
c s@ s | ] } | Vq d S( N( ( Rp Rq ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pys <genexpr>� s u Getting cached minion data.u! Getting minion grain data for: %sRa ( Ro Rr R> R@ R R% RS Rf ( RD R\ RJ Rt Ru ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt get_minion_grainst s
"
c C@ s5 i } | j � } t j d | � | j | � } | S( u@
Get cached mine data for the targeted minions.
u Getting cached mine data for: %s( Ro R R% RN ( RD RL RJ ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt get_cached_mine_data� s
c C@ s� g } | r | j d � n | r2 | j d � n | rH | j d � n | d k rm | j d j | � � n t | � s� t j d � t S| j � } t j d d j | � | � | | k r� i } i } n | j | � \ } } y�| j
j d � } x�| D]}}
t j
j j | j |
� s%q� n |
| k r7q� n d j |
� } | j |
t � } | j |
t � }
| rv| s�| r�|
s�| r�| r�| j
j | d
� nX | r�|
r�| j
j | d
i |
d 6� n, | r�| r�| j
j | d
i | d 6� n | r| j
j | d � q� | d k r� | j
j | d � } t | t � r{| j | t � rx| j
j | d | � qxq{q� q� WWn t t f k
r�t SXt S( uG
Clear the cached data/files for the targeted minions.
u pillaru grainsu mineu mine_func: '{0}'u, No cached data types specified for clearing.u Clearing cached %s data for: %su , u minionsu minions/{0}u dataN( R R R5 Rl R R% R4 Ro R RS RB RF R R RG RH R t popt flusht storeRI R& R' R R R3 ( RD t clear_pillart clear_grainst
clear_minet clear_mine_funct
clear_whatRJ RQ RR t c_minionsRK t bankt
minion_pillarR\ RL ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt clear_cached_minion_data� s`
'N( R9 t
__module__t __doc__R R3 RE RN RS RX R] Rf Rh Ro Rv Rw Rx R4 R� ( ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR7 � s. ' t
CacheTimerc B@ s e Z d Z d � Z d � Z RS( uw
A basic timer class the fires timer-events every second.
This is used for cleanup by the ConnectedCache()
c C@ sl t j | � | | _ | | _ t | _ t j j | j d d � � | _
t j j
| j d d � | _ d S( Nu serialu u sock_diru
con_timer.ipc( R RE R t stoppedR3 t daemonR R R R+ R- R R
R t
timer_sock( RD R t event( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyRE � s
!c C@ s� t j � } | j t j � } | j t j d � | j d | j � d } t j d � xN | j
j d � s� | j | j
j | � � | d 7} | d k r[ d } q[ q[ Wd S( u=
main loop that fires the event every second
id u ipc://i u ConCache-Timer startedi i< N( R t Contextt sockett PUBt
setsockoptt LINGERt bindR� R R% R� t waitt sendR- t dumps( RD t contextR� t count( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt run� s
( R9 R� R� RE R� ( ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� � s t CacheWorkerc B@ s2 e Z d Z d � Z d � Z d � Z d � Z RS( u�
Worker for ConnectedCache which runs in its
own process to prevent blocking of ConnectedCache
main-loop when refreshing minion-list
c K@ s# t t | � j | � | | _ d S( u<
Sets up the zmq-connection to the ConCache
N( t superR� RE R ( RD R R` ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyRE s c C@ s2 t | _ | j | d d | d d | d �d S( Nu optst log_queueu log_queuet log_queue_levelu log_queue_level( R3 t _is_childRE ( RD t state( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt __setstate__ s
c C@ s" i | j d 6| j d 6| j d 6S( Nu optsu log_queueu log_queue_level( R R� R� ( RD ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt __getstate__ s
c C@ s^ t t j j j | j � j � � } t | j � } | j � | j | g � t
j d � d S( uI
Gather currently connected minions and update the cache
u$ ConCache CacheWorker update finishedN( RF R R Ri Rj R t
connected_idst cache_clit
get_cachedt put_cacheR R% ( RD t new_minst cc( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� ! s
$
( R9 R� R� RE R� R� R� ( ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� s
t ConnectedCachec B@ sV e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
RS( u�
Provides access to all minions ids that the master has
successfully authenticated. The cache is cleaned up regularly by
comparing it to the IPs that have open connections to
the master publisher port.
c K@ s� t t | � j | � t j d � | | _ g | _ t j j | j d d � | _
t j j | j d d � | _ t j j | j d d � | _ | j
� t � | _ t | j | j � | _ | j j � t | _ d S( u=
starts the timer and inits the cache itself
u ConCache initializing...u sock_diru
con_cache.ipcu con_upd.ipcu
con_timer.ipcN( R� R� RE R R% R Ri R R
R t
cache_sockt update_sockt
upd_t_sockt cleanupR t
timer_stopR� t timert startR3 t running( RD R R` ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyRE 4 s
c C@ s2 t | _ | j | d d | d d | d �d S( Nu optsR� u log_queueR� u log_queue_level( R3 R� RE ( RD R� ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� Q s
c C@ s" i | j d 6| j d 6| j d 6S( Nu optsu log_queueu log_queue_level( R R� R� ( RD ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� Y s
c C@ s | j � d S( u-
handle signals and shutdown
N( t stop( RD t sigt frame( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt signal_handler` s c C@ s� t j d � t j j | j � r5 t j | j � n t j j | j � r] t j | j � n t j j | j � r� t j | j � n d S( u,
remove sockets on shutdown
u ConCache cleaning upN( R R% R R
t existsR� R$ R� R� ( RD ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� f s
c C@ s� t j d � t j j | j � r8 t j | j d � n t j j | j � rc t j | j d � n t j j | j � r� t j | j d � n d S( u9
secure the sockets for root-only access
u ConCache securing socketsi� N( R R% R R
R� R� t chmodR� R� ( RD ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt securer s
c C@ s= | j � | j r9 t | _ | j j � | j j � n d S( u(
shutdown cache process
N( R� R� R4 R� t setR� R ( RD ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� ~ s
c C@ sP t j � } | j t j � } | j t j d � | j d | j � | j t j � } | j t j d � | j t j d � | j d | j
� | j t j � } | j t j d � | j t j d � | j d | j � t j
� } | j | t j � | j | t j � | j | t j � t j j | j j d d � � } t j t j | j � | j � t j d � x�| j r y t | j d � � } WnT t k
r�| j � n: t j k
r�} t j! d � t j" | � | j � n X| j | � t j k ro| j# | j$ � � } t j% d | � t& | t' j( � ro| d
k rl| j) | j* � }
| j+ |
� qlqon | j | � t j k r�| j# | j$ � � } t& | t, � s�t j! d � ~ qsn y� t- | � d k r�t j% d
� wsn | d } t& | t' j( � r>| | j* k rft j% d | d � | j* j. | � qfn( t& | t, � rft j% d � | | _* n Wn! t/ k
r�t j% d � ~ n Xt j d t- | j* � � n | j | � t j k rs| j# | j$ � � }
t0 |
d � d k r t1 | j � } | j2 � q qsqsW| j � | j3 � | j3 � | j3 � | j4 � t j% d � d S( uw
Main loop of the ConCache, starts updates in intervals and
answers requests from the MWorkers
id u ipc://t u serialu u ConCache startedi u ConCache ZeroMQ-Error occurredu ConCache Received request: %su minionsu( ConCache Worker returned unusable resulti u% ConCache Got empty update from workeru" ConCache Adding minion %s to cacheu* ConCache Replacing minion list from workeru. ConCache Got malformed result dict from workeru ConCache %s entries in cachei u ConCache Shutting downN(5 R R� R� t REPR� R� R� R� t SUBt SUBSCRIBER� t connectR� t Pollert registert POLLINR R R R R+ t signalt SIGINTR� R� R t infoR� R' t pollt KeyboardInterruptR� t ZMQErrorR: t exceptionR# t recvR% R& R t string_typesR� Ri R� RF Rl R t
IndexErrort intR� R� R" t term( RD R� t creq_int cupd_int timer_int pollerR- t sockst zmq_errt msgt replyt
new_c_dataR t sec_eventt cw( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� � s� !
( R9 R� R� RE R� R� R� R� R� R� R� ( ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyR� , s c C@ si t j j � } | d rC t t j j j | � j � � } d } n d } d } | j | d d | �d S( Nu minion_data_cacheu listu *u globu test.pingR< ( R RU t LocalClientRF R Ri Rj R� RW ( R RU R; t form( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt ping_all_connected_minions s
! c C@ s | d k r< | j d d � d k r< | j d d � } q< n | j d � r` | j d d � } n t j j j � r� | j d d � } n t j j | d d j
| � � } t j j j | d | | � y/ t j j
j | d � � } | j � SWd QXWn t t f k
rd SXd S(
Nu rootu useru sudo_u \u _u cachediru .{0}_keyu ru ( R+ t
startswithR R R1 t
is_windowst replaceR R
R R5 RG t check_path_traversalR R R! R R ( t key_userR t skip_perm_errorst keyfilet key( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt get_master_key s" c C@ sG g } x: | D]2 } t j j j | | � r
| j | | � q
q
W| S( uL
Check a whitelist and/or blacklist to see if the value matches it.
( R R t stringutilst
expr_matcht extend( t pattern_dictt user_nameR t expr( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt get_values_of_matching_keys' s
u __main__u /etc/salt/master(6 R� t
__future__R R R t loggingR� t threadingR R t salt.logR t
salt.cachet salt.clientt salt.pillart salt.utils.atomicfilet salt.utils.filest salt.utils.minionst salt.utils.platformt salt.utils.stringutilst salt.utils.verifyt salt.payloadt salt.exceptionsR t salt.configt salt.utils.cacheR R� t salt.utils.processR t salt.extR t salt.utils.zeromqR t getLoggerR9 R R R R* t objectR7 R� R� R� R� R4 R� R� t configt
master_configR t concR� ( ( ( s5 /usr/lib/python2.7/site-packages/salt/utils/master.pyt <module> sL + � \!+�
Mini Shell