Current File : //lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyc
�
���^c @@ s" d Z d d l m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
Z
d d l Z d d l m
Z
d d l m Z d d l Z d d l m Z e j e � Z yF d d l Z e j e j d d d g � d
e �d k re � n Wn[ e k
rdy d d l Z Wqee k
r`d Z e j j d e � e j d
� qeXn Xd Z! d a" i d d 6d d 6d d 6d d 6d d 6d d 6d d 6d d 6Z# d � Z$ d � Z% d � Z& d � Z' d e j( e f d � � YZ) d! e j* f d" � � YZ+ d# e, f d$ � � YZ- d S(% uX
Fluent Logging Handler
======================
.. versionadded:: 2015.8.0
This module provides some fluentd_ logging handlers.
Fluent Logging Handler
----------------------
In the `fluent` configuration file:
.. code-block:: text
<source>
type forward
bind localhost
port 24224
</source>
Then, to send logs via fluent in Logstash format, add the
following to the salt (master and/or minion) configuration file:
.. code-block:: yaml
fluent_handler:
host: localhost
port: 24224
To send logs via fluent in the Graylog raw json format, add the
following to the salt (master and/or minion) configuration file:
.. code-block:: yaml
fluent_handler:
host: localhost
port: 24224
payload_type: graylog
tags:
- salt_master.SALT
The above also illustrates the `tags` option, which allows
one to set descriptive (or useful) tags on records being
sent. If not provided, this defaults to the single tag:
'salt'. Also note that, via Graylog "magic", the 'facility'
of the logged message is set to 'SALT' (the portion of the
tag after the first period), while the tag itself will be
set to simply 'salt_master'. This is a feature, not a bug :)
Note:
There is a third emitter, for the GELF format, but it is
largely untested, and I don't currently have a setup supporting
this config, so while it runs cleanly and outputs what LOOKS to
be valid GELF, any real-world feedback on its usefulness, and
correctness, will be appreciated.
Log Level
.........
The ``fluent_handler`` configuration section accepts an additional setting
``log_level``. If not set, the logging level used will be the one defined
for ``log_level`` in the global configuration file section.
.. admonition:: Inspiration
This work was inspired in `fluent-logger-python`_
.. _fluentd: http://www.fluentd.org
.. _`fluent-logger-python`: https://github.com/fluent/fluent-logger-python
i ( t absolute_importt print_functiont unicode_literalsN( t
LOG_LEVELS( t NewStyleClassMixIn( t sixi i i t use_listu [%(levelname)-8s] %(message)st
log_formatu7 Unable to import msgpack or msgpack_pure python modulesu fluentu EMERGu ALERTu CRITu ERRi u WARNINGi u NOTICEi u INFOi u DEBUGc K@ s@ | j d d � } | j d d � } t | d | d | �a d S( Nu hostu localhostu porti�^ t hostt port( t gett FluentSendert _global_sender( t tagt kwargsR R ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt setup� s c C@ s t S( N( R ( ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt get_global_sender� s c C@ s* t d t k g � s& t j d � t St S( Nu fluent_handleru� The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.( t anyt __opts__t logt tracet Falset __virtualname__( ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt __virtual__� s
c c@ s� d } } d t k rjt d j d d � } t d j d d � } t d j d d � } t d j d d g � } | r� | d n d } | d k r� d } n1 | d k r� d
} n d } t d j d d
� } | d k r� | d k r� t j d � qjt d | d | d | � } t | d | d | �} | j | � | j t t d j d t j d d � � � | Vn | d k r�| d k r�t
Vn d S( Nu fluent_handleru hostu portu payload_typeu tagsu salti u graylogu gelfg�������?u logstashu versioni u� The required 'fluent_handler' configuration keys, 'host' and/or 'port', are not properly configured. Not enabling the fluent logging handler.t payload_typet versiont tagsR R u log_levelu error( t NoneR R
R t debugt MessageFormattert
FluentHandlert setFormattert setLevelR R ( R R R R R
R t formattert fluent_handler( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt setup_handlers� s0
'R c B@ sG e Z d d d � Z d d � Z d � Z d � Z d � Z d � Z RS( c C@ s� | | _ | | _ | r" | d n d | _ | | _ | r@ | n | | _ | rU | n | | _ d j | | � j d d � } t | | � | _ t t
| � j d d d d � d S( Ni u saltu format_{0}_v{1}u .u _t fmtt datefmt(
R R R
R t msg_patht msg_typet formatt replacet getattrt superR t __init__R ( t selfR R R R' R& t format_func( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR, � s c C@ s7 | j d k r | j St j j | j � j � d d S( Nu gelfi����u Z( R t createdt datetimet utcfromtimestampt isoformat( R- t recordR% ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt
formatTime� s c C@ s
i | j � d 6| j | � d 6t j | j d � d 6| j d 6} | j ru | j | j � } | j i | d 6� n x� t j
| j � D]z \ } } | d k r� q� n t | t j
t t t t t t j f � r� | } n t | � } | j i | d j | � 6� q� W| S( u
Graylog 'raw' format is essentially the raw record, minimally munged to provide
the bare minimum that td-agent requires to accept and route the event. This is
well suited to a config where the client td-agents log directly to Graylog.
u messageu timestampu ALERTu levelu tagu full_messageu argsu asctimeu bracketlevelu bracketnameu bracketprocessu createdu exc_infou exc_textu idu levelnameu levelnou msecsu msgu relativeCreatedu versionu {0}( u argsu asctimeu bracketlevelu bracketnameu bracketprocessu createdu exc_infou exc_textu idu levelnameu levelnou msecsu msecsu messageu msgu relativeCreatedu version( t
getMessageR4 t
syslog_levelsR
t levelnameR
t exc_infot formatExceptiont updateR t iteritemst __dict__t
isinstancet string_typest boolt dictt floatt intt listt typest NoneTypet reprR( ( R- R3 t message_dictR8 t keyt valuet val( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt format_graylog_v0� s$
* !c C@ s' i | j d 6t j j j � d 6| j � d 6| j | � d 6t j | j d � d 6| j
d 6} | j r� | j | j � } | j
i | d 6� n x� t j | j � D]z \ } } | d k r� q� n t | t j t t t t t t j f � r� | } n t | � } | j
i | d j | � 6� q� W| S( u�
If your agent is (or can be) configured to forward pre-formed GELF to Graylog
with ZERO fluent processing, this function is for YOU, pal...
u versionu hostu
short_messageu timestampu ALERTu levelu _tagu full_messageu argsu asctimeu bracketlevelu bracketnameu bracketprocessu createdu exc_infou exc_textu idu levelnameu levelnou msecsu messageu msgu relativeCreatedu _{0}( u argsu asctimeu bracketlevelu bracketnameu bracketprocessu createdu exc_infou exc_textu idu levelnameu levelnou msecsu msecsu messageu msgu relativeCreatedu version( R t saltt utilst networkt get_fqhostnameR5 R4 R6 R
R7 R
R8 R9 R: R R; R< R= R> R? R@ RA RB RC RD RE RF R( ( R- R3 RG R8 RH RI RJ ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt format_gelf_v1_1� s(
* !c C@ s� t j j j � } i | j | � d 6i | j d 6| j d 6| j d 6| j d 6| j d 6| j
d 6| j d 6| j d 6d
6| j
� d 6d j | j | | j � d
6| d 6| j d 6| j d 6| j d 6} | j r� | j | j � | d
d <n x� t j | j � D]� \ } } | d! k r$qn | d" k rD| | d
| <qn t | t j t t t t t f � r|| | d
| <qn t | � | d
| <qW| S(# uG
Messages are formatted in logstash's expected format.
u
@timestampu levelnameu loggeru linenou pathnameu processu
threadNameu funcNameu processNameu @fieldsu @messageu
{0}://{1}/{2}u @sourceu @source_hostu @source_pathu @tagsu @typeu exc_infou argsu asctimeu createdu exc_textu filenameu idu levelnou moduleu msecsu messageu msgu nameu relativeCreatedu thread( u argsu asctimeu createdu exc_infou exc_textu filenameu funcNameu idu levelnameu levelnou linenou moduleu msecsu msecsu messageu msgu nameu pathnameu processu processNameu relativeCreatedu threadu
threadNameN( RL RM RN RO R4 R7 t namet linenot pathnamet processt
threadNamet funcNamet processNameR5 R( R' R& R R8 R9 R R; R< R R= R> R? R@ RA RB RC RF ( R- R3 R RG RH RI ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt format_logstash_v0 sN
$c C@ s\ i d d 6| j | � d 6t j j j � d 6| j d 6| j d 6| j d 6| j d 6| j d 6| j
d
6| j d 6| j d 6| j
� d
6| j d 6| j d 6} | j r� | j | j � | d <n x� t j | j � D]~ \ } } | d k r� q� n | d k r| | | <q� n t | t j t t t t t f � rD| | | <q� n t | � | | <q� W| S( uG
Messages are formatted in logstash's expected format.
i u @versionu
@timestampu hostu levelnameu loggeru linenou pathnameu processu
threadNameu funcNameu processNameu messageu tagsu typeu exc_infou argsu asctimeu createdu exc_textu filenameu idu levelnou moduleu msecsu msgu nameu relativeCreatedu thread( u argsu asctimeu createdu exc_infou exc_textu filenameu funcNameu idu levelnameu levelnou linenou moduleu msecsu msecsu messageu msgu nameu pathnameu processu processNameu relativeCreatedu threadu
threadNameN( R4 RL RM RN RO R7 RQ RR RS RT RU RV RW R5 R R' R8 R9 R R; R< R R= R> R? R@ RA RB RC RF ( R- R3 RG RH RI ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt format_logstash_v1R sB
$
N( t __name__t
__module__R R, R4 RK RP RX RY ( ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR � s ! # 7R c B@ s5 e Z d Z d d d e d � Z d � Z d � Z RS( u%
Logging Handler for fluent.
u localhosti�^ g @c
C@ sD | | _ t | d | d | d | d | �| _ t j j | � d S( NR R t timeoutt verbose( R
R t sendert loggingt HandlerR, ( R- R
R R R\ R] ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR, � s
c C@ s& | j | � } | j j d | � d S( N( R( R^ t emitR ( R- R3 t data( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRa � s c C@ s= | j � z! | j j � t j j | � Wd | j � Xd S( N( t acquireR^ t _closeR_ R` t closet release( R- ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRe � s
( RZ R[ t __doc__R R, Ra Re ( ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR � s R c B@ s_ e Z d d d d e d � Z d � Z d � Z d � Z d � Z d
� Z d � Z d � Z
RS( u localhosti�^ i i g @c C@ s� | | _ | | _ | | _ | | _ | | _ | | _ d | _ d | _ t j
� | _ y | j � Wn t
k
r� | j � n Xd S( N( R
R R t bufmaxR\ R] R t sockett pendingst threadingt Lockt lockt
_reconnectt ExceptionRd ( R- R
R R Rh R\ R] ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR, � s
c C@ s) t t j � � } | j | | | � d S( N( RB t timet emit_with_time( R- t labelRb t cur_time( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRa � s c C@ s& | j | | | � } | j | � d S( N( t _make_packett _send( R- Rr t timestampRb t bytes_( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRq � s c C@ s\ | r! d j | j | f � } n | j } | | | f } | j rO t | � n t j | � S( Nu .( t joinR
R] t printt msgpackt packb( R- Rr Rv Rb R
t packet( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRt � s
c C@ s3 | j j � z | j | � Wd | j j � Xd S( N( Rm Rc t _send_internalRf ( R- Rw ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRu � s
c C@ s� | j r$ | j | 7_ | j } n y' | j � | j j | � d | _ WnQ t k
r� | j � | j r� t | j � | j k r� d | _ q� | | _ n Xd S( N( Rj Rn Ri t sendallR Ro Rd t lenRh ( R- Rw ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR} � s
! c C@ s� | j s� | j j d � r` t j t j t j � } | j | j � | j | j t d � � nA t j t j t j � } | j | j � | j | j | j
f � | | _ n d S( Nu unix://( Ri R t
startswitht AF_UNIXt SOCK_STREAMt
settimeoutR\ t connectR t AF_INETR ( R- t sock( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRn � s c C@ s& | j r | j j � n d | _ d S( N( Ri Re R ( R- ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyRd � s i i ( RZ R[ R R, Ra Rq Rt Ru R} Rn Rd ( ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyR � s
(. Rg t
__future__R R R R_ t logging.handlersRp R0 Ri Rk RD t salt.log.setupR t salt.log.mixinsR t salt.utils.networkRL t salt.extR t getLoggerRZ R Rz t loadst dumpst TrueR t ImportErrort msgpack_puret
LOG_FORMATt setup_console_loggert fatalR R R6 R R R R# t FormatterR R` R t objectR ( ( ( s@ /usr/lib/python2.7/site-packages/salt/log/handlers/fluent_mod.pyt <module>J sT -
' �
Mini Shell