Current File : //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/autotracer.cpython-37.pyc
B
r�bjJ � @ s� d Z dZddlZddlZddlmZ ddlmZmZm Z ddl
m
Z
m Z ddl m
Z
ddlmZmZmZmZ dd lmZ ddlZd
d lmZ d d
lmZmZm Z d dl m Z d dl m Z d dl!m"Z" d dl#m$Z$m%Z% yddl&m'Z( W n e)k
r� dZ(Y nX eG dd� d��Z*eG dd� d��Z+eG dd� d��Z,G dd� de�Z-e.e-d�dd �Z/G d d � d e�Z0e1d k�r�e%� ej2d ej3d!� e0� Z4e4� dS )"z'
This module contains Autotracer class
z ssa.modules� N)� defaultdict)� dataclass�field�asdict)�datetime� timedelta)� fnmatchcase)�Iterator�Optional�
NamedTuple�List)�urlparse� )�Common� )�status�disabled_users�is_edition_supported)�
load_tunables)�autotracing_tunables_schema)�autotracing_stats_file)�is_xray_version_supported� sentry_initc @ sJ e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S )
�URLzURL data container�uri� avg_duration� max_duration�total_slow_reqs�
total_reqs�
belongs_to�belongs_to_userN)�__name__�
__module__� __qualname__�__doc__�str�__annotations__�int� r( r( �G/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/autotracer.pyr ( s
r c @ s~ e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < dZ
eed
< dZeed < d S )
� RulesRejectszRejects counters containerr �non_wp� throttled�density�slowness�disabled�nginx� frequency� server_limit� domain_limitN)r! r" r# r$ r+ r' r& r, r- r. r/ r0 r1 r2 r3 r( r( r( r) r* 4 s
r* c @ sD e Zd ZU dZeed< dZeed< dZeed< e e
d�Z e
ed< dS ) �Statsz AutoTracer statistics container�
rules_versionr �urls_processed�
urls_selected)�default_factory�rejectsN) r! r" r# r$ r% r& r6 r' r7 r r* r9 r( r( r( r) r4 B s
r4 c @ s" e Zd ZU dZeed< eed< dS )�URLSz"
Representation of an URL
� domain_name�uri_pathN)r! r" r# r$ r% r& r( r( r( r) r: K s
r: )�url�returnc C s\ t | �}|jrd|j� �nd}|jr2|j� |� �nd}|j�dd�}|�d�d }t||�S )z�
Split URL into domain_name and uripath including query string
:param url: URL of format protocol://domain/path;parameters?query#fragment
:return: namedtuple URL(domain_name, uripath)
�?� �/zwww.�:r )r
�query�path�netloc�replace�splitr: )r= � fragments�qsr Z_no_www_netlocZ_no_port_netlocr( r( r) � url_splitS s rJ c sp e Zd ZdZ� fdd�Z� fdd�Zeed�dd��Zee e
e d �d
d ��Z e e
d �d
d�Zed�dd�Zd6e
e ed�dd�Ze e
d�dd�Zeee
d�dd�Zee
d�d d �Ze
e ee
d �d d �Ze
e
e
d!�d"d#�Zeee d$�d%d&�Zd7e
e ee d'�d(d)�Zee d�d*d+�Z e e d�d,d-�Z dd�d.d/�Z dd�d0d1�Z ed�d2d3�Z!ed�d4d5�Z"� Z#S )8�
AutoTracerz0
SSA autotracing module implementation.
c s> t � �� t�d�| _| j�dt� t� | _t� | _ d | _
d S )NZ auto_tracerzAutoTracer enabled: %s) �super�__init__�logging� getLogger�logger�info� __package__�listr �
tasks_list�stats)�self)� __class__r( r) rM f s
zAutoTracer.__init__c s t � s| j�d� d S t� s,| j�d� d S td krD| j�d� d S tj}|d k �r| j�d� t� �� | j�d| j � t
| j �| _ xz| �
� D ]n}| j�d|j� | j jd7 _y||j| jd� W q� tk
r� } z| j�d t|�� W d d }~X Y q�X q�W | �� d S )
Nz*AutoTracer skipped: unsupported CL editionz-AutoTracer skipped: unsupported X-Ray versionz*AutoTracer skipped: unable to import X-RayzAutoTracer startedz AutoTracer loaded config: %szStarting auto task for %sr )r= Z
tracing_countzFailed to start task: %s)r rP rQ r �xray_libZstart_autotracingrL rM �debug�configr4 r5 rU �urls_scheduledr r7 �request_number� Exception�errorr% �save_iteration_stats)rV Z
start_toolr= �e)rW r( r) �__call__n s.
(zAutoTracer.__call__)r> c C s
t dt�S )z$
Load configuration
zautotracing.json)r r r( r( r( r) � load_conf� s zAutoTracer.load_conf)r; r> c C s t dkr dS t �| �S )z;
Gets domain info for the specified domain
N)rX � domain_info)r; r( r( r) �gets_domaininfo� s zAutoTracer.gets_domaininfo)� user_namer> c C s" | j r tdkrdS t�|�jS dS )z>
Says if nginx is enabled for a specific user
NF)Z
skip_nginxrX ZNginxUserCache�
is_enabled)rV re r( r( r) �nginx_is_enabled� s
zAutoTracer.nginx_is_enabledc C s, t dd� | jD ��}| j�d|� | j| S )z>Recalculate limit per server taking into account running tasksc S s( g | ] }|d dkr|d dkr|�qS )r �running�userz
*autotracing*r( )�.0�taskr( r( r) �
<listcomp>� s z<AutoTracer.per_server_limit_recalculated.<locals>.<listcomp>z&Number of running autotracing tasks %s)�lenrT rP rY Zper_server_limit)rV Z
running_countr( r( r) � per_server_limit_recalculated� s z(AutoTracer.per_server_limit_recalculatedN)� full_listr> c sD |dkr t �� }| j�d|� t�� t| jd� � � fdd�|D �S )zQ
Excludes tasks older than N days from the general list of tasks
NzTask list loaded %s)�daysc s8 g | ]0}|� d �dk r |� d �ndt� �� �kr|�qS )Z
createtimeNr )�getr' � timestamp)rj rk )�
n_days_agor( r) rl � s z1AutoTracer.excludes_old_tasks.<locals>.<listcomp>)rX rT rP rY r �nowr Zmin_retracing_interval)rV ro r( )rs r) �excludes_old_tasks� s z AutoTracer.excludes_old_tasks)� current_urlr> c C s| t |�}xn| jD ]d}t |d �}|j|jko6|j|jk}t|j|j�oRt|j|j�}|s\|r| j�d||d � dS qW dS )z�
Excludes url from the list if it completely matches the current url
or if domain names match and "*" follows the domain name in the list
r= z2Skipped: URL %s was traced recently. Matched by %sTF)rJ rT r; r< r rP rY )rV rv �cZ task_data�tZ direct_matchZwildcard_matchr( r( r) �exclude_thesame_urls� s z AutoTracer.exclude_thesame_urls)�url_total_reqs�domain_total_reqsr> c C s4 | j r0t�t�||��}| j�d|� || jkS dS )z-Check that URL density passes given thresholdzCalculated density %sT)r- �npZaminZcorrcoefrP rY Zdensity_threshold)rV rz r{ Z url_densityr( r( r) �pass_by_density� s
zAutoTracer.pass_by_density)�
url_durationsr> c C s0 d| j }t�||�}| j�d||� || jkS )z�
The measure of "slowness" for URL is:
at least N% of its requests take more than X seconds.
N% -- self.slow_duration_percentage
X -- self.slow_duration_threshold
�d z Calculated %sth percentile %s)Zslow_duration_percentager| Z
percentilerP rY Zslow_duration_threshold)rV r~ Zreversed_percentileZreversed_percentile_valuer( r( r) �pass_by_slowness_percentile� s
z&AutoTracer.pass_by_slowness_percentile)�url_throttled_reqsrz r> c C s8 |dkr dS t |�t |� d }| j�d|� || jkS )zY
Check that percent of throttled requests per URL passes given threshold
NFr z Calculated throttled percent %s)�sumrP rY Z allowed_throttling_percentage)rV r� rz Zthrottled_percentr( r( r) �pass_by_allowed_throttling� s
z%AutoTracer.pass_by_allowed_throttling)� wp_statusr> c C s | j rt|�S dS )zx
Check that URLs of a particular domain should be analyzed.
For now we skip non-wordpress sites
T)Zonly_wp�bool)rV r� r( r( r) �pass_by_engine� s zAutoTracer.pass_by_engine)�collected_statsr> c # s� �x�|� � D �]�\}}d}� �|�d��sp� j�d� t� fdd�|D ��}� jj j|7 _� j j |7 _ q �xD|� � D �]6\}}|� j
kr�q|� j�d|� � j j d7 _ � � |�d�|d �s� j�d
� � jj j d7 _ q|� �
|d |d ��s � j�d � � jj jd7 _q|� �|d
��sL� j�d� � jj jd7 _q||dk�rv� �|�}|dk�rpd}n|j}t|tt�|d
��t|d �t|d �t|d
�||d�V q|W q W dS )a
Select all URLs suitable for auto tracing by very basic rules:
- WP site
- suitable throttling
- suitable density
- measure of "slow" URL
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
NZis_a_wordpress_domainz Skipped by engine: non-wordpressc s g | ]}|� j kr|�qS r( )�non_url_fields)rj �u)rV r( r) rl s z<AutoTracer.urls_computationally_filtered.<locals>.<listcomp>zProcessing URL %sr r� rz z Skipped by throttled percentr{ zSkipped by densityZ durationsz Skipped by slowness percentile�unknownZ
url_slow_reqs)r r r r r r )�itemsr� rq rP rY rm rU r9 r+ r6 r� r� r, r} r- r� r. rd ri r r' r| Zmeanr� �max) rV r� �domainZ url_statsZ domain_ownerZ
skipped_countr= �datarc r( )rV r) � urls_computationally_filtered sP
z(AutoTracer.urls_computationally_filtered)�stats_collectedr> c c s� |dkr"| � � \}}| j�d|� | �� xx| �|�D ]j}|j| jkrj| j�d|j� | jj j d7 _ q6| �
|j�r�| j�d|j� | jj j d7 _ q6|V q6W dS )z�
From selected by computed thresholds URLs take those for which:
- autotracing enabled
- nginx disabled
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
NzAutoTracer loaded data: %sz'Skipped: autotracing is disabled for %sr z Skipped: nginx is enabled for %s) Z
load_statsrP rY �fill_in_disabled_usersr� r r rU r9 r/ rg r0 )rV r� �_r= r( r( r) r7 H s zAutoTracer.urls_selectedc c s� d}t t�}| �� }| j�d|� | �� | _| �� }x�|D ]�}| �|j �r^| j
j j d7 _ q:||k r�||j
| jk r�|d7 }||j
d7 < |V q�| j�d|j | j� | j
j jd7 _q:| j�d|j |� | j
j jd7 _q:W dS )a=
Schedule autotracing by sorted list taking into account the limits:
- no same task for 10 days
- limit per server
- limit per domain
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Return resulting list of URLs scheduled for auto tracing
r zSorted scheduled list %sr z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r' � urls_sortedrP rY ru rT rn ry r rU r9 r1 r Zper_domain_limitr3 r2 )rV Zgeneral_tasks_counterZtasks_counter_per_domainZ sorted_urlsZper_server_smart_limitr= r( r( r) r[ h s.
zAutoTracer.urls_scheduledc C s, t t| �� �dd� dd�}t |dd� dd�S )zd
Sort URLs by total number of requests first
and by average duration second
c S s | j S )N)r )r� r( r( r) �<lambda>� � z(AutoTracer.urls_sorted.<locals>.<lambda>T)�key�reversec S s | j S )N)r )r� r( r( r) r� � r� )�sortedrS r7 )rV Z first_serier( r( r) r� � s
zAutoTracer.urls_sortedc C s t � | _ dS )z6
Fill internal list of disabled users
N)r )rV r( r( r) r� � s z!AutoTracer.fill_in_disabled_usersc
C sp | j dk rly,ttd��}t�t| j �|� W dQ R X W n4 tk
rj } z| j�dt |�� W dd}~X Y nX dS )zI
Save collected statistics for current iteration to file
N�wz*Unable to save iteration stats to file: %s)
rU �openr �json�dumpr �OSErrorrP �warningr% )rV �
stats_filer` r( r( r) r_ � s
z AutoTracer.save_iteration_statsc
C sv yFt t��4}t�|�}t|d |d |d tf |d ��}W dQ R X W n& ttjtfk
rl t| j �}Y nX t
|�S )z@
Load statistics for latest iteration from file
r5 r6 r7 r9 N) r� r r� �loadr4 r* r� �JSONDecodeError�KeyErrorr5 r )rV r� �_dataZ stat_datar( r( r) �load_iteration_stats� s
z AutoTracer.load_iteration_statsc C s* | � � }|�tt� d tt� �d�� |S )r@ r )r Zdisabled_users_quantity)r� �update�dictr rm r )rV Z stats_loadedr( r( r) � get_stats� s
zAutoTracer.get_stats)N)N)$r! r" r# r$ rM ra � staticmethodr� rb r% r
�objectrd r� rg r' rn rS ru ry r} r� r� r� r r r� r7 r[ r r� r� r_ r� r� �
__classcell__r( r( )rW r) rK a s2
H ,
rK �__main__zauto_tracer_standalone.log)�filename�level)5r$ rR r� rN � collectionsr Z dataclassesr r r r r �fnmatchr �typingr r
r r � urllib.parser
Znumpyr| �commonr � autotracingr r r �
configurationr Zconfiguration.schemesr �internal.constantsr �internal.utilsr r Zxray.shared_libraryZshared_libraryrX � ImportErrorr r* r4 r: r% rJ rK r! � basicConfig�DEBUGrx r( r( r( r) �<module>
sD
k
Mini Shell