Python：Scrapy
先查看PIP的版本：
|
1 2 3 |
[root@master01 ~]# pip3 -V pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6) [root@master01 ~]# |
使用PIP安装：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
[root@master01 ~]# pip3 install scrapy WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. Collecting scrapy Using cached https://files.pythonhosted.org/packages/22/26/497ef936b54ae00e24694934cb25d524061722bb0d8582da33430e3e7608/Scrapy-2.5.1-py2.py3-none-any.whl Collecting h2<4.0,>=3.0 (from scrapy) Using cached https://files.pythonhosted.org/packages/25/de/da019bcc539eeab02f6d45836f23858ac467f584bfec7a526ef200242afe/h2-3.2.0-py2.py3-none-any.whl Collecting w3lib>=1.17.0 (from scrapy) Downloading https://files.pythonhosted.org/packages/a3/59/b6b14521090e7f42669cafdb84b0ab89301a42f1f1a82fcf5856661ea3a7/w3lib-1.22.0-py2.py3-none-any.whl Collecting itemadapter>=0.1.0 (from scrapy) Downloading https://files.pythonhosted.org/packages/7f/a9/017e825977a8a6447ef3e3144c11fc70ffadb0d998ade8ed5419b9fa0447/itemadapter-0.4.0-py3-none-any.whl Collecting cryptography>=2.0 (from scrapy) Using cached https://files.pythonhosted.org/packages/10/91/90b8d4cd611ac2aa526290ae4b4285aa5ea57ee191c63c2f3d04170d7683/cryptography-35.0.0.tar.gz Complete output from command python setup.py egg_info: =============================DEBUG ASSISTANCE========================== If you are seeing an error here please try the following to successfully install cryptography: Upgrade to the latest pip and try again. This will fix errors for most users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip =============================DEBUG ASSISTANCE========================== Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-jnx0nipy/cryptography/setup.py", line 14, in <module> from setuptools_rust import RustExtension ModuleNotFoundError: No module named 'setuptools_rust' ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jnx0nipy/cryptography/ [root@master01 ~]# |
遇到该报错的解决方法，升级PIP：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
[root@master01 ~]# pip3 install --upgrade pip WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. Cache entry deserialization failed, entry ignored Collecting pip Downloading https://files.pythonhosted.org/packages/90/a9/1ea3a69a51dcc679724e3512fc2aa1668999eed59976f749134eb02229c8/pip-21.3-py3-none-any.whl (1.7MB) 100% |████████████████████████████████| 1.7MB 231kB/s Installing collected packages: pip Successfully installed pip-21.3 [root@master01 ~]# [root@master01 ~]# pip3 -V WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. pip 21.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6) [root@master01 ~]# |
再次安装：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
[root@master01 ~]# pip3 install scrapy WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. Collecting scrapy Using cached Scrapy-2.5.1-py2.py3-none-any.whl (254 kB) Requirement already satisfied: lxml>=3.5.0 in /usr/lib64/python3.6/site-packages (from scrapy) (4.2.5) Collecting itemloaders>=1.0.1 Downloading itemloaders-1.0.4-py3-none-any.whl (11 kB) Collecting cryptography>=2.0 Downloading cryptography-35.0.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.7 MB) |████████████████████████████████| 3.7 MB 55 kB/s Collecting queuelib>=1.4.2 Downloading queuelib-1.6.2-py2.py3-none-any.whl (13 kB) Collecting zope.interface>=4.1.3 Downloading zope.interface-5.4.0-cp36-cp36m-manylinux2010_x86_64.whl (251 kB) |████████████████████████████████| 251 kB 38 kB/s Collecting PyDispatcher>=2.0.5 Downloading PyDispatcher-2.0.5.zip (47 kB) |████████████████████████████████| 47 kB 50 kB/s Preparing metadata (setup.py) ... done Collecting parsel>=1.5.0 Using cached parsel-1.6.0-py2.py3-none-any.whl (13 kB) Collecting Twisted[http2]>=17.9.0 Using cached Twisted-21.7.0-py3-none-any.whl (3.1 MB) Collecting h2<4.0,>=3.0 Using cached h2-3.2.0-py2.py3-none-any.whl (65 kB) Collecting itemadapter>=0.1.0 Using cached itemadapter-0.4.0-py3-none-any.whl (10 kB) Collecting service-identity>=16.0.0 Using cached service_identity-21.1.0-py2.py3-none-any.whl (12 kB) Collecting protego>=0.1.15 Downloading Protego-0.1.16.tar.gz (3.2 MB) |████████████████████████████████| 3.2 MB 71 kB/s Preparing metadata (setup.py) ... done Collecting cssselect>=0.9.1 Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB) Collecting pyOpenSSL>=16.2.0 Downloading pyOpenSSL-21.0.0-py2.py3-none-any.whl (55 kB) |████████████████████████████████| 55 kB 91 kB/s Collecting w3lib>=1.17.0 Using cached w3lib-1.22.0-py2.py3-none-any.whl (20 kB) Collecting cffi>=1.12 Downloading cffi-1.15.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (405 kB) |████████████████████████████████| 405 kB 62 kB/s Collecting hyperframe<6,>=5.2.0 Downloading hyperframe-5.2.0-py2.py3-none-any.whl (12 kB) Collecting hpack<4,>=3.0 Downloading hpack-3.0.0-py2.py3-none-any.whl (38 kB) Collecting jmespath>=0.9.5 Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) Collecting six>=1.6.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting attrs>=19.1.0 Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB) |████████████████████████████████| 53 kB 75 kB/s Collecting pyasn1-modules Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) |████████████████████████████████| 155 kB 74 kB/s Collecting pyasn1 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Collecting incremental>=21.3.0 Downloading incremental-21.3.0-py2.py3-none-any.whl (15 kB) Collecting constantly>=15.1 Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB) Collecting typing-extensions>=3.6.5 Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB) Collecting Automat>=0.8.0 Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB) Collecting hyperlink>=17.1.1 Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB) |████████████████████████████████| 74 kB 56 kB/s Collecting priority<2.0,>=1.1.0 Downloading priority-1.3.0-py2.py3-none-any.whl (11 kB) Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from zope.interface>=4.1.3->scrapy) (39.2.0) Collecting pycparser Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB) |████████████████████████████████| 112 kB 61 kB/s Collecting idna>=2.5 Downloading idna-3.3-py3-none-any.whl (61 kB) |████████████████████████████████| 61 kB 62 kB/s Using legacy 'setup.py install' for protego, since package 'wheel' is not installed. Using legacy 'setup.py install' for PyDispatcher, since package 'wheel' is not installed. Installing collected packages: six, pycparser, idna, attrs, zope.interface, w3lib, typing-extensions, pyasn1, incremental, hyperlink, hyperframe, hpack, cssselect, constantly, cffi, Automat, Twisted, pyasn1-modules, priority, parsel, jmespath, itemadapter, h2, cryptography, service-identity, queuelib, pyOpenSSL, PyDispatcher, protego, itemloaders, scrapy Running setup.py install for PyDispatcher ... done Running setup.py install for protego ... done Successfully installed Automat-20.2.0 PyDispatcher-2.0.5 Twisted-21.7.0 attrs-21.2.0 cffi-1.15.0 constantly-15.1.0 cryptography-35.0.0 cssselect-1.1.0 h2-3.2.0 hpack-3.0.0 hyperframe-5.2.0 hyperlink-21.0.0 idna-3.3 incremental-21.3.0 itemadapter-0.4.0 itemloaders-1.0.4 jmespath-0.10.0 parsel-1.6.0 priority-1.3.0 protego-0.1.16 pyOpenSSL-21.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 queuelib-1.6.2 scrapy-2.5.1 service-identity-21.1.0 six-1.16.0 typing-extensions-3.10.0.2 w3lib-1.22.0 zope.interface-5.4.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [root@master01 ~]# |
验证安装：
|
1 2 3 4 5 |
[root@master01 ~]# python -V Python 2.7.5 [root@master01 ~]# [root@master01 ~]# python3 -c "import scrapy" [root@master01 ~]# |
