Current File : //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/processor.cpython-37.pyc
B
r�b}% � @ s� d Z ddlZddlZddlZddlZddlZddlZddlmZmZm Z ddl
m Z m Z m
Z
ddlmZmZ ddlmZ ddlmZ dd lmZ dd
lmZ d d lmZ d d
lm Z d dl m Z m Z m Z e G dd� de��Z!dS )z-
This module contains RequestProcessor class
� N)�datetime� timedelta�date)�Thread�RLock�current_thread)�Callable�Any� )�
AutoTracer)�Common)�
DecisionMaker)�StatisticsSender� )� storage_file)�SSAError)� singleton� url_split�switch_schedstatsc s e Zd ZdZ� fdd�Zedd� �Zed�dd�Zd d
�d d �Z d d
�d
d�Z
e e d�dd�Z
d-ed�dd�Zdd� Zed
�dd�Zeeed�dd ��Zd d
�d d �Zeed d �d d!�Zeed d"�d#d$�Zeeeeed d%�d&d'�Zed d(�d)d*�Zd+d,� Z � Z S ).�RequestProcessorz^
SSA Request processor implementation.
Only one instance is allowed to be created
c st t � �� t�d�| _| j�dt� tdd� | �� \| _ }|sH| �
� t � | _ t
� | _t� | _t� | _| �� d S )NZ
req_processorzProcessor enabled: %sT)�enabled)�super�__init__�logging� getLogger�logger�info� __package__r Z
load_stats� total_struct�cleanupr �_lockr
�decision_makerr �senderr � auto_tracer�start_background_routine)�selfZtotal_struct_date)� __class__� �F/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/processor.pyr ) s
zRequestProcessor.__init__c C s
| j d S )zp
Return config file value multiplied by 1000000,
as we receive duration in microseconds
i@B )Zrequests_duration)r% r' r' r( �configured_duration8 s z$RequestProcessor.configured_duration)�reportc
C sJ y| j �|� W n4 tk
rD } z| j�dt|�� W dd}~X Y nX dS )z(
Call Statistics Sender
zStatisticsSender failed: %sN)r" �sendr r �error�str)r% r* �er' r' r( �
send_stats@ s zRequestProcessor.send_statsN)�returnc C s* t | jdd�}|�� | j�d|j� dS )zA
Start dumper|DecisionMaker thread in background
T)�target�daemonz[%s] Routine startedN)r �background_routine�startr r �name)r% �tr' r' r( r$ I s z)RequestProcessor.start_background_routinec C s� x�t �� }|jdkr�| j�dt� j|� | �� |jdkr�| j�dt� j|� | � | j
� | j�dt� j|� | � | j �}| � � | � | j
|� tdd� | �d� q| �|j� qW dS ) z�
Dumps collected stats to file once an hour.
Runs DecisionMaker once a day
Cleanup storage after DecisionMaker run
r z'[%s] Routine thread launching dump (%s)z-[%s] Routine thread launching AutoTracer (%s)z0[%s] Routine thread launching DecisionMaker (%s)T)r �< N)r �now�minuter r r r5 � dump_collected_stats_to_file�hour�
_safe_execr# r! r r/ r �
_simple_sleep�_sleep_till_next_hour)r% Ztickr* r' r' r( r3 Q s(
z#RequestProcessor.background_routine)�actionr0 c G sR y||� S t k
rL t�� \}}}| jjdt|�||dt�� id� Y nX dS )zACall requested Callable with given args and capture any exceptionz %s failed with exception %s, %sZorig_traceback)�extraN)� Exception�sys�exc_infor �criticalr- � traceback�
format_exc)r% r? �args�etZev�_r' r' r( r< o s
zRequestProcessor._safe_exec� )�to_sleepc C s" | j �dt� j|� t�|� dS )zP
Log and sleep given number of seconds or 15 minutes by default
z%[%s] Routine thread sleeping for (%s)N)r r r r5 �time�sleep)r% rK r' r' r( r= y s z RequestProcessor._simple_sleepc C s* t dd�t |d� �� }| �t|�� dS )zF
Sleep the number of minutes remaining till next hour
r
)�hours)�minutesN)r �
total_secondsr= �int)r% Z start_minuteZ sleep_forr' r' r( r> � s
z&RequestProcessor._sleep_till_next_hourc C s� | j �� | j�dt� j� | �� yLt�t��8}x | j �
� D ]\}}|||< q<W dd� |�
� D �}W dQ R X W n@ t k
r� } z"| jj ddt
|�id� t� }W dd}~X Y nX W dQ R X | j�dt� j� |S ) z.
Dump collected stats to file
z [%s] Acquires lock to dump statsc S s i | ]\}}||�qS r' r' )�.0�k�vr' r' r( �
<dictcomp>� s zARequestProcessor.dump_collected_stats_to_file.<locals>.<dictcomp>NzFailed to dump data�err)r@ z [%s] Released lock to dump stats)r r �debugr r5 �add_current_date�shelve�openr r �items�OSErrorr, r- �dict)r% �db�item�valueZ dump_resultr. r' r' r( r: � s"
"
z-RequestProcessor.dump_collected_stats_to_file)� timestampr0 c C s t �| �jS )zN
Takes an hour of a day, to which the given timestamp belongs
)r �
fromtimestampr; )ra r' r' r( �get_interval_for� s z!RequestProcessor.get_interval_forc C s t �� | jd< dS )zW
Adds current date to the 'total struct' dict before dumping to a file
Z current_dateN)r �todayr )r% r' r' r( rX � s z!RequestProcessor.add_current_date)r5 �is_wpr0 c C s> || j kr:| j�dt� j|� ttdgd �|d�| j |< dS )zH
Add new domain sub-struct if it is not already present
z [%s] New domain received: %sr � )�domain_total_reqsZis_a_wordpress_domainN)r r rW r r5 r] �list)r% r5 re r' r' r( �
add_domain� s
zRequestProcessor.add_domain)�domain�urlr0 c C s` || j | kr\| j�dt� j|� ttdgd �tdgd �tdgd �t� d�| j | |< dS )zE
Add new URL sub-struct if it is not already present
z[%s] New URL received: %sr rf )�url_total_reqs�
url_slow_reqs�url_throttled_reqs� durationsN)r r rW r r5 r] rh )r% rj rk r' r' r( �add_url� s zRequestProcessor.add_url)rj rk ra �duration� throttledr0 c C s | � |�}| j| d | d7 < | j| | d | d7 < || jkrj| j| | d | d7 < |r�d| j| | kr�tdgd �| j| | d< | j| | d | d7 < | j| | d �|� | j�d t� j|� | j� d
t� j| j� d S ) za
Update request counters for given domain and url,
save request duration
rg r
rl rm rn r rf ro z [%s] Request to %s processedz[%s] %sN)
rc r r) rh �appendr r r r5 rW )r% rj rk ra rq rr �intervalr' r' r( � update_data� s
z RequestProcessor.update_data)�datar0 c
C s� |� d�}| �|�r&| j�d|� dS t|�\}}| j�n | j�dt� j� | j|t |� d��d� | �
||� | j ||t |� d��t
|� d��t |� d ��d
� W dQ R X | j�d t� j� dS ) z,
Process given request data
rk z
%s ignoredNz-[%s] Acquires lock to handle request countersZ wordpress)re ra rq Zhitting_limits)rj rk ra rq rr z-[%s] Released lock to handle request counters)�getZ
is_ignoredr rW r r r r5 ri �boolrp ru rQ �float)r% rv rk rj �urir' r' r( �handle� s$
zRequestProcessor.handlec C sJ | j �d� yt�t� W n tk
r: | j �d� Y nX | j�� dS )z2
Cleanup storage and total_struct
zRequestProcessor cleanup...zAlready no storage fileN)r r �os�unlinkr r\ r �clear)r% r' r' r( r � s zRequestProcessor.cleanup)rJ ) �__name__�
__module__� __qualname__�__doc__r �propertyr) r] r/ r$ r3 r r r< rQ r= r> r: � staticmethodrc rX r- rx ri rp ry ru r{ r �
__classcell__r' r' )r& r( r " s&
r )"r� r r| rY rB rL rE r r r � threadingr r r �typingr r Z
autotracerr �commonr r! r
Z stat_senderr �internal.constantsr �internal.exceptionsr �internal.utilsr r r r r' r' r' r( �<module>
s$
Mini Shell