Current File : //proc/227033/root/opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyc
�
+��`c @ s, d d l Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z y d d l Z Wn e k
r� d d l
Z n Xd d l Z d d l m
Z
d d l m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z m Z d d l m Z m! Z! m" Z" d d l# m$ Z$ m% Z% d d l& m' Z' m( Z( m) Z) m* Z* m+ Z+ m, Z, m- Z- m. Z. m/ Z/ d d l0 m1 Z1 m2 Z2 d d
l3 m4 Z4 m5 Z5 e j6 e7 � Z8 e j9 d � Z: e j9 d e j; � Z< e j9 d
� Z= d Z> e? d � Z@ d e f d � � YZA d eB f d � � YZC d eC f d � � YZD d eC f d � � YZE d eB f d � � YZF d eC f d � � YZG d eC f d � � YZH d eC f d � � YZI d eC f d! � � YZJ d" eC f d# � � YZK eK eI � eG d$ d% d& �d' d( �ZL eL jM ZM e j9 d) � ZN d* eB f d+ � � YZO d S(, i����N( t BytesIOi ( t DistlibException( t urljoint urlparset
urlunparset url2pathnamet pathname2urlt queuet quotet unescapet string_typest build_openert HTTPRedirectHandlert text_typet Requestt HTTPErrort URLError( t Distributiont DistributionPatht make_dist( t Metadatat MetadataInvalidError( t cached_propertyt parse_credentialst ensure_slasht split_filenamet get_project_datat parse_requirementt parse_name_and_versiont ServerProxyt normalize_name( t
get_schemet UnsupportedVersionError( t Wheelt
is_compatibles ^(\w+)=([a-f0-9]+)s ;\s*charset\s*=\s*(.*)\s*$s text/html|application/x(ht)?mls https://pypi.org/pypic C sJ | d k r t } n t | d d �} z | j � SWd | d � � Xd S( s�
Return all distribution names known by an index.
:param url: The URL of the index.
:return: A list of all known distribution names.
t timeoutg @Nt close( t Nonet
DEFAULT_INDEXR t
list_packages( t urlt client( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt get_all_distribution_names) s t RedirectHandlerc B s% e Z d Z d � Z e Z Z Z RS( sE
A class to work around a bug in some Python 3.2.x releases.
c C s� d } x( d D] } | | k r
| | } Pq
q
W| d k rA d St | � } | j d k r� t | j � | � } t | d � r� | j | | � q� | | | <n t j | | | | | | � S( Nt locationt urit t replace_header( R, R- ( R% R t schemeR t get_full_urlt hasattrR/ t BaseRedirectHandlert http_error_302( t selft reqt fpt codet msgt headerst newurlt keyt urlparts( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR4 @ s
( t __name__t
__module__t __doc__R4 t http_error_301t http_error_303t http_error_307( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR+ 7 s t Locatorc B s� e Z d Z d Z d Z d Z d Z e d Z d d � Z d
� Z
d � Z d � Z d � Z
d � Z e e
e � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z e d � Z RS(! sG
A base class for locators - things that locate distributions.
s .tar.gzs .tar.bz2s .tars .zips .tgzs .tbzs .eggs .exes .whls .pdft defaultc C s@ i | _ | | _ t t � � | _ d | _ t j � | _ d S( s^
Initialise an instance.
:param scheme: Because locators look for most recent versions, they
need to know the version scheme to use. This specifies
the current PEP-recommended scheme - use ``'legacy'``
if you need to support existing distributions on PyPI.
N(
t _cacheR0 R R+ t openerR% t matcherR t Queuet errors( R5 R0 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt __init__f s
c C sm g } x` | j j � sh y# | j j t � } | j | � Wn | j j k
rW q n X| j j � q W| S( s8
Return any errors which have occurred.
( RJ t emptyt gett Falset appendt Emptyt task_done( R5 t resultt e( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt
get_errorsy s c C s | j � d S( s>
Clear any errors which may have been logged.
N( RT ( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt clear_errors� s c C s | j j � d S( N( RF t clear( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt clear_cache� s c C s | j S( N( t _scheme( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _get_scheme� s c C s
| | _ d S( N( RX ( R5 t value( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _set_scheme� s c C s t d � � d S( s=
For a given project, get a dictionary mapping available versions to Distribution
instances.
This should be implemented in subclasses.
If called from a locate() request, self.matcher will be set to a
matcher for the requirement to satisfy, otherwise it will be None.
s Please implement in the subclassN( t NotImplementedError( R5 t name( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _get_project� s
c C s t d � � d S( sJ
Return all the distribution names known to this locator.
s Please implement in the subclassN( R\ ( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt get_distribution_names� s c C sj | j d k r! | j | � } nE | | j k r@ | j | } n& | j � | j | � } | | j | <| S( s�
For a given project, get a dictionary mapping available versions to Distribution
instances.
This calls _get_project to do all the work, and just implements a caching layer on top.
N( RF R% R^ RU ( R5 R] RR ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt get_project� s
c C s� t | � } t j | j � } t } | j d � } | j | j � } | rf t t | � | j � } n | j
d k d | j k | | | | f S( su
Give an url a score which can be used to choose preferred URLs
for a given project release.
s .whlt httpss pypi.org( R t posixpatht basenamet patht Truet endswitht downloadable_extensionsR" R! t
wheel_tagsR0 t netloc( R5 R( t tRc t
compatiblet is_wheelt is_downloadable( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt score_url� s c C s{ | } | rw | j | � } | j | � } | | k r? | } n | | k ra t j d | | � qw t j d | | � n | S( s{
Choose one of two URLs where both are candidates for distribution
archives for the same version of a distribution (for example,
.tar.gz vs. zip).
The current implementation favours https:// URLs over http://, archives
from PyPI over those from other locations, wheel compatibility (if a
wheel) and then the archive name.
s Not replacing %r with %rs Replacing %r with %r( Rn t loggert debug( R5 t url1t url2RR t s1t s2( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt
prefer_url� s
c C s
t | | � S( sZ
Attempt to split a filename in project name, version and Python version.
( R ( R5 t filenamet project_name( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR � s c C s d � } d } t | � \ } } } } } }
|
j � j d � rX t j d | |
� n t j |
� } | r� | j � \ } }
n d \ } }
| } | r� | d d k r� | d } n | j d � r�y� t
| � } t | | j � s� t j d | � n� | d k rt
} n | | j | � } | r�i | j d 6| j d 6| j d
6t | | | | | d f � d 6d
j g | j D] } d j t | d � � ^ qw� d 6} n Wq�t k
r�} t j d | � q�Xn| j | j � s�t j d | � n� t j | � } } x� | j D]� } | j | � r| t | � } | j | | � } | sct j d | � nu | \ } } } | s�| | | � r�i | d 6| d 6| d
6t | | | | | d f � d 6} | r�| | d <q�n PqqW| r�| r�|
| d | <n | S( s
See if a URL is a candidate for a download URL for a project (the URL
has typically been scraped from an HTML page).
If it is, a dictionary is returned with keys "name", "version",
"filename" and "url"; otherwise, None is returned.
c S s t | � t | � k S( N( R ( t name1t name2( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt same_project� s s egg=s %s: version hint in fragment: %ri����t /s .whls Wheel not compatible: %sR] t versionRv R. R( s , t .i s python-versions invalid path for wheel: %ss Not downloadable: %ss No match for project/version: %ss %s_digestN( NN( R% R t lowert
startswithRo Rp t HASHER_HASHt matcht groupsRf R! R" Rh Re R] R| Rv R t joint pyvert listt Exceptiont warningRg Rb Rc t lenR ( R5 R( Rw Rz RR R0 Ri Rd t paramst queryt fragt mt algot digestt origpatht wheelt includet vRS Rv t extRj R] R| R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt convert_url_to_download_info� sn
= c C s� d } d | k rP | d } x1 d D]& } | | k r# | | | f } Pq# q# Wn | s� x; d D]0 } d | } | | k r] | | | f } Pq] q] Wn | S( s�
Get a digest from a dictionary by looking at a "digests" dictionary
or keys of the form 'algo_digest'.
Returns a 2-tuple (algo, digest) if found, else None. Currently
looks only for SHA256, then MD5.
t digestst sha256t md5s %s_digestN( R� R� ( R� R� ( R% ( R5 t infoRR R� R� R< ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _get_digest1 s
c C s� | j d � } | j d � } | | k r@ | | } | j } n! t | | d | j �} | j } | j | � | _ } | d } | | d | <| j | d k r� | j | j | � | _ | d j | t � � j
| � n | | _ | | | <d S( s�
Update a result dictionary (the final result from _get_project) with a
dictionary for a specific version, which typically holds information
gleaned from a filename or URL for an archive for the distribution.
R] R| R0 R( R� t urlsN( t popt metadataR R0 R� R� t
source_urlRu t
setdefaultt sett addt locator( R5 RR R� R] R| t distt mdR� R( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _update_version_dataH s
# c C sU d
} t | � } | d
k r1 t d | � � n t | j � } | j | j � | _ } t j d | t | � j
� | j | j � } t
| � d k r�g } | j } x� | D]� }
|
d k r� q� n ye | j |
� s� t j d | |
� n< | s| |
� j r| j |
� n t j d |
| j � Wq� t k
rRt j d | |
� q� Xq� Wt
| � d k r�t | d
| j �} n | r�t j d | � | d } | | } q�n | rH| j r�| j | _ n | j d i � j | t � � | _ i } | j d i � }
x. | j D]# } | |
k r|
| | | <qqW| | _ n d
| _ | S( s
Find the most recent distribution which matches the given
requirement.
:param requirement: A requirement of the form 'foo (1.0)' or perhaps
'foo (>= 1.0, < 2.0, != 1.3)'
:param prereleases: If ``True``, allow pre-release versions
to be located. Otherwise, pre-release versions
are not returned.
:return: A :class:`Distribution` instance, or ``None`` if no such
distribution could be located.
s Not a valid requirement: %rs matcher: %s (%s)i R� R� s %s did not match %rs% skipping pre-release version %s of %ss error matching %s with %ri R< s sorted list: %si����N( s urlss digests( R% R R R R0 RH t requirementRo Rp t typeR> R` R] R� t
version_classR� t
is_prereleaseRO R� R� t sortedR< t extrasRM R� t
download_urlsR� ( R5 R� t prereleasesRR t rR0 RH t versionst slistt vclst kR| t dt sdR( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt locate_ sT
$ ( s .tar.gzs .tar.bz2s .tars .zips .tgzs .tbz( s .eggs .exes .whl( s .pdfN( s .whl( R> R? R@ t source_extensionst binary_extensionst excluded_extensionsR% Rh Rg RK RT RU RW RY R[ t propertyR0 R^ R_ R` Rn Ru R R� R� R� RN R� ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRD V s.
J t PyPIRPCLocatorc B s) e Z d Z d � Z d � Z d � Z RS( s�
This locator uses XML-RPC to locate distributions. It therefore
cannot be used with simple mirrors (that only mirror file content).
c K s8 t t | � j | � | | _ t | d d �| _ d S( s�
Initialise an instance.
:param url: The URL to use for XML-RPC.
:param kwargs: Passed to the superclass constructor.
R# g @N( t superR� RK t base_urlR R) ( R5 R( t kwargs( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK � s c C s t | j j � � S( sJ
Return all the distribution names known to this locator.
( R� R) R' ( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR_ � s c C sv i i d 6i d 6} | j j | t � } xF| D]>} | j j | | � } | j j | | � } t d | j � } | d | _ | d | _ | j d � | _
| j d g � | _ | j d � | _ t
| � } | r0 | d } | d
| _ | j | � | _ | | _ | | | <xX | D]M } | d
}
| j | � } | d j | t � � j |
� | | d |
<qWq0 q0 W| S( NR� R� R0 R] R| t licenset keywordst summaryi R( ( R) t package_releasesRe t release_urlst release_dataR R0 R] R| RM R� R� R� R R� R� R� R� R� R� R� ( R5 R] RR R� R� R� t dataR� R� R� R( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ � s0
( R> R? R@ RK R_ R^ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� � s t PyPIJSONLocatorc B s) e Z d Z d � Z d � Z d � Z RS( sw
This locator uses PyPI's JSON interface. It's very limited in functionality
and probably not worth using.
c K s) t t | � j | � t | � | _ d S( N( R� R� RK R R� ( R5 R( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK � s c C s t d � � d S( sJ
Return all the distribution names known to this locator.
s Not available from this locatorN( R\ ( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR_ � s c C s� i i d 6i d 6} t | j d t | � � } yE| j j | � } | j � j � } t j | � } t d | j
� } | d } | d | _ | d | _ | j
d � | _ | j
d g � | _ | j
d
� | _ t | � } | | _ | d } | | | j <x| | d D]p }
|
d } | j j | � | j |
� | j | <| d j | j t � � j | � | j |
� | d | <q Wx� | d j � D]� \ } } | | j k r�q�n t d | j
� }
| j |
_ | |
_ t |
� } | | _ | | | <xu | D]m }
|
d } | j j | � | j |
� | j | <| d j | t � � j | � | j |
� | d | <q�Wq�WWn9 t k
r�} | j j t | � � t j d
| � n X| S( NR� R� s %s/jsonR0 R� R] R| R� R� R� R( t releasess JSON fetch failed: %s( R R� R RG t opent readt decodet jsont loadsR R0 R] R| RM R� R� R� R R� R� R� R� R� R� R� t itemsR� RJ t putR
Ro t exception( R5 R] RR R( t respR� R� R� R� R� R� R| t infost omdt odistRS ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ � sT
#
# ( R> R? R@ RK R_ R^ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� � s t Pagec B sz e Z d Z e j d e j e j Be j B� Z e j d e j e j B� Z d � Z
e j d e j � Z e d � � Z
RS( s4
This class represents a scraped HTML page.
s�
(rel\s*=\s*(?:"(?P<rel1>[^"]*)"|'(?P<rel2>[^']*)'|(?P<rel3>[^>\s
]*))\s+)?
href\s*=\s*(?:"(?P<url1>[^"]*)"|'(?P<url2>[^']*)'|(?P<url3>[^>\s
]*))
(\s+rel\s*=\s*(?:"(?P<rel4>[^"]*)"|'(?P<rel5>[^']*)'|(?P<rel6>[^>\s
]*)))?
s! <base\s+href\s*=\s*['"]?([^'">]+)c C sM | | _ | | _ | _ | j j | j � } | rI | j d � | _ n d S( sk
Initialise an instance with the Unicode page contents and the URL they
came from.
i N( R� R� R( t _baset searcht group( R5 R� R( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK s
s [^a-z0-9$&+,/:;=?@.#%_\\|-]c C s d � } t � } x� | j j | j � D]� } | j d � } | d pv | d pv | d pv | d pv | d pv | d } | d p� | d
p� | d } t | j | � } t | � } | j j d � | � } | j
| | f � q( Wt | d
d � d t �} | S( s�
Return the URLs of all the links on a page together with information
about their "rel" attribute, for determining which ones to treat as
downloads and which ones to queue for further scraping.
c S s@ t | � \ } } } } } } t | | t | � | | | f � S( s Tidy up an URL.( R R R ( R( R0 Ri Rd R� R� R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt clean4 s R. t rel1t rel2t rel3t rel4t rel5t rel6Rq Rr t url3c S s d t | j d � � S( Ns %%%2xi ( t ordR� ( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt <lambda>B s R< c S s | d S( Ni ( ( Rj ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� F s t reverse(
R� t _hreft finditerR� t groupdictR R� R t _clean_ret subR� R� Re ( R5 R� RR R� R� t relR( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt links- s ( R> R? R@ t ret compilet It St XR� R� RK R� R R� ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� s t SimpleScrapingLocatorc B s� e Z d Z i e j d 6d � d 6d � d 6Z d d d � Z d � Z d � Z d
� Z
e j d e j
� Z d � Z d
� Z d � Z d � Z d � Z e j d � Z d � Z RS( s�
A locator which scrapes HTML pages to locate downloads for a distribution.
This runs multiple threads to do the I/O; performance is at least as good
as pip's PackageFinder, which works in an analogous fashion.
t deflatec C s t j d t t � � j � S( Nt fileobj( t gzipt GzipFileR R� R� ( t b( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� T s R� c C s | S( N( ( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� U s t nonei
c K s� t t | � j | � t | � | _ | | _ i | _ t � | _ t j
� | _ t � | _ t
| _ | | _ t j � | _ t j � | _ t
| _ d S( s�
Initialise an instance.
:param url: The root URL to use for scraping.
:param timeout: The timeout, in seconds, to be applied to requests.
This defaults to ``None`` (no timeout specified).
:param num_workers: The number of worker threads you want to do I/O,
This defaults to 10.
:param kwargs: Passed to the superclass.
N( R� R� RK R R� R# t _page_cacheR� t _seenR RI t _to_fetcht
_bad_hostsRN t skip_externalst num_workerst threadingt RLockt _lockt _gplockt platform_check( R5 R( R# R� R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK X s
c C sc g | _ xS t | j � D]B } t j d | j � } | j t � | j � | j j | � q Wd S( s�
Threads are created only when get_project is called, and terminate
before it returns. They are there primarily to parallelise I/O (i.e.
fetching web pages).
t targetN(
t _threadst rangeR� R� t Threadt _fetcht setDaemonRe t startRO ( R5 t iRj ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _prepare_threadss s
c C sO x! | j D] } | j j d � q
Wx | j D] } | j � q. Wg | _ d S( su
Tell all the threads to terminate (by sending a sentinel value) and
wait for them to do so.
N( R R� R� R% R� ( R5 Rj ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt
_wait_threads� s
c C s� i i d 6i d 6} | j �� | | _ | | _ t | j d t | � � } | j j � | j j � | j � z1 t
j d | � | j j
| � | j j � Wd | j � X| ` Wd QX| S( NR� R� s %s/s Queueing %s( R� RR Rw R R� R R� RV R� R Ro Rp R� R� R� R ( R5 R] RR R( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ � s
s< \b(linux_(i\d86|x86_64|arm\w+)|win(32|_amd64)|macosx_?\d+)\bc C s | j j | � S( sD
Does an URL refer to a platform-specific download?
( t platform_dependentR� ( R5 R( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _is_platform_dependent� s c C sy | j r! | j | � r! d } n | j | | j � } t j d | | � | ru | j � | j | j | � Wd QXn | S( s%
See if an URL is a suitable download for a project.
If it is, register information in the result dictionary (for
_get_project) about the specific version it's for.
Note that the return value isn't actually used other than as a boolean
value.
s process_download: %s -> %sN(
R� R
R% R� Rw Ro Rp R� R� RR ( R5 R( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt _process_download� s
c
C s t | � \ } } } } } } | j | j | j | j � rG t } n� | j rl | j | j � rl t } n� | j | j � s� t } ny | d k r� t } nd | d k r� t } nO | j | � r� t } n7 | j
d d � d } | j � d k r� t } n t } t
j d
| | | | � | S(
s�
Determine whether a link URL from a referring page and with a
particular "rel" attribute should be queued for scraping.
t homepaget downloadt httpRa t ftpt :i i t localhosts# should_queue: %s (%s) from %s -> %s( R R
( R s httpsR ( R Rf R� R� R� RN R� R R� R
t splitR~ Re Ro Rp (
R5 t linkt referrerR� R0 Ri Rd t _RR t host( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt
_should_queue� s*
c C s8 x1t r3| j j � } z� y� | r� | j | � } | d k rE w n x� | j D]� \ } } | | j k rO y_ | j j | � | j | � r� | j | | | � r� t
j d | | � | j j | � n Wq� t
k
r� q� XqO qO Wn Wn) t k
r} | j j t | � � n XWd | j j � X| s Pq q Wd S( s�
Get a URL to fetch from the work queue, get the HTML page, examine its
links for download candidates and candidates for further scraping.
This is a handy method to run in a thread.
s Queueing %s from %sN( Re R� RM t get_pageR% R� R� R� R R Ro Rp R� R R� RJ R
RQ ( R5 R( t pageR R� RS ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR � s.
c C s� t | � \ } } } } } } | d k rZ t j j t | � � rZ t t | � d � } n | | j k r� | j | } t j d | | � nK| j
d d � d } d } | | j k r� t j d | | � n
t
| d i d d
6�} z�y7t j d | � | j j | d | j �} t j d
| � | j � }
|
j d d � } t j | � r | j � } | j � }
|
j d � } | r�| j | } | |
� }
n d } t j | � } | r�| j d � } n y |
j | � }
Wn t k
r�|
j d � }
n Xt |
| � } | | j | <n Wn� t k
rZ} | j d k r�t j d | | � q�nk t! k
r�} t j d | | � | j" � | j j# | � Wd QXn& t$ k
r�} t j d | | � n XWd | | j | <X| S( s
Get the HTML for an URL, possibly from an in-memory cache.
XXX TODO Note: this cache is never actually cleared. It's assumed that
the data won't get stale over the lifetime of a locator instance (not
necessarily true for the default_locator).
t files
index.htmls Returning %s from cache: %sR i i s Skipping %s due to bad host %sR: t identitys Accept-encodings Fetching %sR# s
Fetched %ss Content-TypeR. s Content-Encodings utf-8s latin-1i� s Fetch failed: %s: %sN(% R t osRd t isdirR R R R� Ro Rp R R% R� R RG R� R# R� RM t HTML_CONTENT_TYPER� t geturlR� t decoderst CHARSETR� R� R� t UnicodeErrorR� R R8 R� R R� R� R� ( R5 R( R0 Ri Rd R RR R R6 R� R: t content_typet final_urlR� t encodingt decoderR� RS ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR � s\ $
s <a href=[^>]*>([^<]+)<c C sq t � } | j | j � } | s7 t d | j � � n x3 | j j | j � D] } | j | j d � � qM W| S( sJ
Return all the distribution names known to this locator.
s Unable to get %si ( R� R R� R t _distname_reR� R� R� R� ( R5 RR R R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR_ 7 s N( R> R? R@ t zlibt
decompressR R% RK R R R^ R� R� R� R R
R R R R R' R_ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR� J s$
! ;t DirectoryLocatorc B s2 e Z d Z d � Z d � Z d � Z d � Z RS( s?
This class locates distributions in a directory tree.
c K so | j d t � | _ t t | � j | � t j j | � } t j j | � sb t
d | � � n | | _ d S( s�
Initialise an instance.
:param path: The root of the directory tree to search.
:param kwargs: Passed to the superclass constructor,
except for:
* recursive - if True (the default), subdirectories are
recursed into. If False, only the top-level directory
is searched,
t recursives Not a directory: %rN( R� Re R+ R� R* RK R Rd t abspathR R t base_dir( R5 Rd R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK H s
c C s | j | j � S( s�
Should a filename be considered as a candidate for a distribution
archive? As well as the filename, the directory which contains it
is provided, though not used by the current implementation.
( Rf Rg ( R5 Rv t parent( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt should_includeY s c C s� i i d 6i d 6} x� t j | j � D]� \ } } } x� | D]� } | j | | � r= t j j | | � } t d d t t j j | � � d d d f � } | j | | � } | r� | j
| | � q� q= q= W| j s' Pq' q' W| S( NR� R� R R. ( R t walkR- R/ Rd R� R R R, R� R� R+ ( R5 R] RR t roott dirst filest fnR( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ a s "
c C s� t � } x� t j | j � D]� \ } } } x� | D]� } | j | | � r2 t j j | | � } t d d t t j j | � � d d d f � } | j
| d � } | r� | j | d � q� q2 q2 W| j
s Pq q W| S( sJ
Return all the distribution names known to this locator.
R R. R] N( R� R R0 R- R/ Rd R� R R R, R� R% R� R+ ( R5 RR R1 R2 R3 R4 R( R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR_ q s "
( R> R? R@ RK R/ R^ R_ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR* C s
t JSONLocatorc B s e Z d Z d � Z d � Z RS( s
This locator uses special extended metadata (not available on PyPI) and is
the basis of performant dependency resolution in distlib. Other locators
require archive downloads before dependencies can be determined! As you
might imagine, that can be slow.
c C s t d � � d S( sJ
Return all the distribution names known to this locator.
s Not available from this locatorN( R\ ( R5 ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR_ � s c C sB i i d 6i d 6} t | � } | r>x| j d g � D]� } | d d k s9 | d d k re q9 n t | d | d d
| j d
d � d | j �} | j } | d
| _ d | k r� | d r� d | d f | _ n | j d i � | _ | j d i � | _ | | | j <| d j
| j t � � j | d
� q9 Wn | S( NR� R� R3 t ptypet sdistt pyversiont sourceR] R| R� s Placeholder for summaryR0 R( R� R� t requirementst exports(
R RM R R0 R� R� R� t dependenciesR; R| R� R� R� ( R5 R] RR R� R� R� R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ � s&
.( R> R? R@ R_ R^ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR5 � s t DistPathLocatorc B s e Z d Z d � Z d � Z RS( s�
This locator finds installed distributions in a path. It can be useful for
adding to an :class:`AggregatingLocator`.
c K s8 t t | � j | � t | t � s+ t � | | _ d S( ss
Initialise an instance.
:param distpath: A :class:`DistributionPath` instance to search.
N( R� R= RK t
isinstanceR t AssertionErrort distpath( R5 R@ R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK � s c C s� | j j | � } | d k r5 i i d 6i d 6} nG i | | j 6i t | j g � | j 6d 6i t d g � | j 6d 6} | S( NR� R� ( R@ t get_distributionR% R| R� R� ( R5 R] R� RR ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ � s
( R> R? R@ RK R^ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR= � s
t AggregatingLocatorc B sP e Z d Z d � Z d � Z d � Z e e j j e � Z d � Z
d � Z RS( sI
This class allows you to chain and/or merge a list of locators.
c O s8 | j d t � | _ | | _ t t | � j | � d S( s�
Initialise an instance.
:param locators: The list of locators to search.
:param kwargs: Passed to the superclass constructor,
except for:
* merge - if False (the default), the first successful
search from any of the locators is returned. If True,
the results from all locators are merged (this can be
slow).
t mergeN( R� RN RC t locatorsR� RB RK ( R5 RD R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK � s c C s5 t t | � j � x | j D] } | j � q Wd S( N( R� RB RW RD ( R5 R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRW � s c C s* | | _ x | j D] } | | _ q Wd S( N( RX RD R0 ( R5 RZ R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR[ � s c C s] i } xP| j D]E} | j | � } | r | j r� | j d i � } | j d i � } | j | � | j d � } | r� | r� xF | j � D]5 \ } } | | k r� | | c | O<q� | | | <q� Wn | j d � }
| rR|
rR|
j | � qRqU| j d k rt } n3 t } x* | D]" } | j j
| � r t } Pq q W| rU| } PqUq q W| S( NR� R� ( RD R` RC RM t updateR� RH R% Re RN R� ( R5 R] RR R� R� R3 R� t dfR� R� t ddt found( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR^ � s8
c C sI t � } x9 | j D]. } y | | j � O} Wq t k
r@ q Xq W| S( sJ
Return all the distribution names known to this locator.
( R� RD R_ R\ ( R5 RR R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyR_ s
( R> R? R@ RK RW R[ R� RD R0 t fgetR^ R_ ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRB � s ,s https://pypi.org/simple/R# g @R0 t legacys1 (?P<name>[\w-]+)\s*\(\s*(==\s*)?(?P<ver>[^)]+)\)$t DependencyFinderc B sV e Z d Z d d � Z d � Z d � Z d � Z d � Z d � Z d e
d � Z RS( s0
Locate dependencies for distributions.
c C s( | p t | _ t | j j � | _ d S( sf
Initialise an instance, using the specified locator
to locate distributions.
N( t default_locatorR� R R0 ( R5 R� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK 1 s c C s� t j d | � | j } | | j | <| | j | | j f <x^ | j D]S } t | � \ } } t j d | | | � | j j | t
� � j | | f � qF Wd S( s�
Add a distribution to the finder. This will update internal information
about who provides what.
:param dist: The distribution to add.
s adding distribution %ss Add to provided: %s, %s, %sN( Ro Rp R< t
dists_by_namet distsR| t providesR t providedR� R� R� ( R5 R� R] t pR| ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt add_distribution9 s
c C s� t j d | � | j } | j | =| j | | j f =xl | j D]a } t | � \ } } t j d | | | � | j | } | j | | f � | s@ | j | =q@ q@ Wd S( s�
Remove a distribution from the finder. This will update internal
information about who provides what.
:param dist: The distribution to remove.
s removing distribution %ss Remove from provided: %s, %s, %sN(
Ro Rp R< RM RN R| RO R RP t remove( R5 R� R] RQ R| t s( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt remove_distributionH s
c C sP y | j j | � } Wn3 t k
rK | j � d } | j j | � } n X| S( s�
Get a version matcher for a requirement.
:param reqt: The requirement
:type reqt: str
:return: A version matcher (an instance of
:class:`distlib.version.Matcher`).
i ( R0 RH R R ( R5 t reqtRH R] ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt get_matcherZ s
c C s� | j | � } | j } t � } | j } | | k r� x_ | | D]P \ } } y | j | � } Wn t k
ry t } n X| rA | j | � PqA qA Wn | S( s�
Find the distributions which can fulfill a requirement.
:param reqt: The requirement.
:type reqt: str
:return: A set of distribution which can fulfill the requirement.
( RW R< R� RP R� R RN R� ( R5 RV RH R] RR RP R| t providerR� ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt find_providersj s
c C s� | j | } t � } x? | D]7 } | j | � } | j | j � s | j | � q q W| r� | j d | | t | � f � t } nZ | j | � | j | =x- | D]% } | j j | t � � j | � q� W| j
| � t } | S( s�
Attempt to replace one provider with another. This is typically used
when resolving dependencies from multiple sources, e.g. A requires
(B >= 1.0) while C requires (B >= 1.1).
For successful replacement, ``provider`` must meet all the requirements
which ``other`` fulfills.
:param provider: The provider we are trying to replace with.
:param other: The provider we're trying to replace.
:param problems: If False is returned, this will contain what
problems prevented replacement. This is currently
a tuple of the literal string 'cantreplace',
``provider``, ``other`` and the set of requirements
that ``provider`` couldn't fulfill.
:return: True if we can replace ``other`` with ``provider``, else
False.
t cantreplace( t reqtsR� RW R� R| R� t frozensetRN RU R� RR Re ( R5 RX t othert problemst rlistt unmatchedRT RH RR ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt try_to_replace� s"
#
c
C s! i | _ i | _ i | _ i | _ t | p0 g � } d | k rk | j d � | t d d d g � O} n t | t � r� | } } t j d | � nK | j
j | d | �} } | d k r� t
d | � � n t j d | � t | _ t � } t | g � } t | g � } x�| r�| j � } | j } | | j k rO| j | � n/ | j | }
|
| k r~| j | |
| � n | j | j B} | j } t � }
| r�| | k r�xA d D]6 } d | } | | k r�|
t | d
| � O}
q�q�Wn | | B|
B} x�| D]�} | j | � } | s1t j d | � | j
j | d | �} | d k r|| r|| j
j | d t �} n | d k r�t j d | � | j d | f � q1| j | j } } | | f | j k r�| j | � n | j | � | | k r1| | k r1| j | � t j d | j � q1n xw | D]o } | j } | | j k rx| j j | t � � j | � q8| j | }
|
| k r8| j | |
| � q8q8Wq WqWt | j j � � } x< | D]4 } | | k | _ | j r�t j d | j � q�q�Wt j d | � | | f S( s�
Find a distribution and all distributions it depends on.
:param requirement: The requirement specifying the distribution to
find, or a Distribution instance.
:param meta_extras: A list of meta extras such as :test:, :build: and
so on.
:param prereleases: If ``True``, allow pre-release versions to be
returned - otherwise, don't return prereleases
unless they're all that's available.
Return a set of :class:`Distribution` instances and a set of
problems.
The distributions returned should be such that they have the
:attr:`required` attribute set to ``True`` if they were
from the ``requirement`` passed to ``find()``, and they have the
:attr:`build_time_dependency` attribute set to ``True`` unless they
are post-installation dependencies of the ``requirement``.
The problems should be a tuple consisting of the string
``'unsatisfied'`` and the requirement which couldn't be satisfied
by any distribution known to the locator.
s :*:s :test:s :build:s :dev:s passed %s as requirementR� s Unable to locate %rs
located %st testt buildt devs :%s:s %s_requiress No providers found for %rs Cannot satisfy %rt unsatisfieds Adding %s to install_distss# %s is a build-time dependency only.s find done for %sN( Rb s buildRd ( RP RN RM R[ R� RS R> R Ro Rp R� R� R% R Re t requestedR� R< RR Ra t run_requirest
meta_requirest build_requirest getattrRY R� R| t name_and_versionR� t valuest build_time_dependency( R5 R� t meta_extrasR� R� R� R^ t todot
install_distsR] R] t ireqtst sreqtst ereqtsR< RS t all_reqtsR� t providersRX t nR� RQ RN ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt find� s�
!
"
"
N( R> R? R@ R% RK RR RU RW RY Ra RN Rw ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyRK , s ((P R� t ioR R� t loggingR Rb R� R� t ImportErrort dummy_threadingR( R. R t compatR R R R R R R R R
R R R3 R
R R R t databaseR R R R� R R t utilR R R R R R R R R R| R R R� R! R" t getLoggerR> Ro R� R� R� R! R R& R% R* R+ t objectRD R� R� R� R� R* R5 R= RB RL R� t NAME_VERSION_RERK ( ( ( sM /opt/alt/python37/lib/python3.7/site-packages/pip/_vendor/distlib/locators.pyt <module> sV
d @ � F0E:�A& [
Mini Shell