[Ubuntu 16.04] Scrapy 설치 (without virtualenv)

아래의 순서로 설치

1. 필수 lib 설치하기.

apt-get install libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev  apt-get install libffi-dev

1. pip 설치하기.

root@DevGuru-Linux:~# apt-get install python-pip 패키지 목록을 읽는 중입니다... 완료 의존성 트리를 만드는 중입니다       상태 정보를 읽는 중입니다... 완료 The following additional packages will be installed:   libexpat1-dev libpython-all-dev libpython-dev libpython2.7-dev python-all python-all-dev python-dev python-pip-whl python-pkg-resources   python-setuptools python-wheel python2.7-dev 제안하는 패키지:   python-setuptools-doc 다음 새 패키지를 설치할 것입니다:   libexpat1-dev libpython-all-dev libpython-dev libpython2.7-dev python-all python-all-dev python-dev python-pip python-pip-whl python-pkg-resources   python-setuptools python-wheel python2.7-dev 0개 업그레이드, 13개 새로 설치, 0개 제거 및 5개 업그레이드 안 함. 29.8 M바이트 아카이브를 받아야 합니다. 이 작업 후 45.1 M바이트의 디스크 공간을 더 사용하게 됩니다. 계속 하시겠습니까? [Y/n] y 받기:1 http://kr.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libexpat1-dev amd64 2.1.0-7ubuntu0.16.04.2 [115 kB] 받기:2 http://kr.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpython2.7-dev amd64 2.7.12-1ubuntu0~16.04.1 [27.8 MB] 받기:3 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 libpython-dev amd64 2.7.11-1 [7,728 B] 받기:4 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 libpython-all-dev amd64 2.7.11-1 [992 B] 받기:5 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 python-all amd64 2.7.11-1 [978 B] 받기:6 http://kr.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python2.7-dev amd64 2.7.12-1ubuntu0~16.04.1 [276 kB] 받기:7 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 python-dev amd64 2.7.11-1 [1,160 B] 받기:8 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 python-all-dev amd64 2.7.11-1 [1,000 B] 받기:9 http://kr.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-pip-whl all 8.1.1-2ubuntu0.4 [1,110 kB] 받기:10 http://kr.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-pip all 8.1.1-2ubuntu0.4 [144 kB] 받기:11 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 python-pkg-resources all 20.7.0-1 [108 kB] 받기:12 http://kr.archive.ubuntu.com/ubuntu xenial/main amd64 python-setuptools all 20.7.0-1 [169 kB] 받기:13 http://kr.archive.ubuntu.com/ubuntu xenial/universe amd64 python-wheel all 0.29.0-1 [48.0 kB] 내려받기 29.8 M바이트, 소요시간 4초 (6,956 k바이트/초) Selecting previously unselected package libexpat1-dev:amd64. (데이터베이스 읽는중 ...현재 210478개의 파일과 디렉터리가 설치되어 있습니다.) Preparing to unpack .../libexpat1-dev_2.1.0-7ubuntu0.16.04.2_amd64.deb ... Unpacking libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.2) ... Selecting previously unselected package libpython2.7-dev:amd64. Preparing to unpack .../libpython2.7-dev_2.7.12-1ubuntu0~16.04.1_amd64.deb ... Unpacking libpython2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.1) ... Selecting previously unselected package libpython-dev:amd64. Preparing to unpack .../libpython-dev_2.7.11-1_amd64.deb ... Unpacking libpython-dev:amd64 (2.7.11-1) ... Selecting previously unselected package libpython-all-dev:amd64. Preparing to unpack .../libpython-all-dev_2.7.11-1_amd64.deb ... Unpacking libpython-all-dev:amd64 (2.7.11-1) ... Selecting previously unselected package python-all. Preparing to unpack .../python-all_2.7.11-1_amd64.deb ... Unpacking python-all (2.7.11-1) ... Selecting previously unselected package python2.7-dev. Preparing to unpack .../python2.7-dev_2.7.12-1ubuntu0~16.04.1_amd64.deb ... Unpacking python2.7-dev (2.7.12-1ubuntu0~16.04.1) ... Selecting previously unselected package python-dev. Preparing to unpack .../python-dev_2.7.11-1_amd64.deb ... Unpacking python-dev (2.7.11-1) ... Selecting previously unselected package python-all-dev. Preparing to unpack .../python-all-dev_2.7.11-1_amd64.deb ... Unpacking python-all-dev (2.7.11-1) ... Selecting previously unselected package python-pip-whl. Preparing to unpack .../python-pip-whl_8.1.1-2ubuntu0.4_all.deb ... Unpacking python-pip-whl (8.1.1-2ubuntu0.4) ... Selecting previously unselected package python-pip. Preparing to unpack .../python-pip_8.1.1-2ubuntu0.4_all.deb ... Unpacking python-pip (8.1.1-2ubuntu0.4) ... Selecting previously unselected package python-pkg-resources. Preparing to unpack .../python-pkg-resources_20.7.0-1_all.deb ... Unpacking python-pkg-resources (20.7.0-1) ... Selecting previously unselected package python-setuptools. Preparing to unpack .../python-setuptools_20.7.0-1_all.deb ... Unpacking python-setuptools (20.7.0-1) ... Selecting previously unselected package python-wheel. Preparing to unpack .../python-wheel_0.29.0-1_all.deb ... Unpacking python-wheel (0.29.0-1) ... Processing triggers for doc-base (0.10.7) ... Processing 1 added doc-base file... Processing triggers for man-db (2.7.5-1) ... libexpat1-dev:amd64 (2.1.0-7ubuntu0.16.04.2) 설정하는 중입니다 ... libpython2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.1) 설정하는 중입니다 ... libpython-dev:amd64 (2.7.11-1) 설정하는 중입니다 ... libpython-all-dev:amd64 (2.7.11-1) 설정하는 중입니다 ... python-all (2.7.11-1) 설정하는 중입니다 ... python2.7-dev (2.7.12-1ubuntu0~16.04.1) 설정하는 중입니다 ... python-dev (2.7.11-1) 설정하는 중입니다 ... python-all-dev (2.7.11-1) 설정하는 중입니다 ... python-pip-whl (8.1.1-2ubuntu0.4) 설정하는 중입니다 ... python-pip (8.1.1-2ubuntu0.4) 설정하는 중입니다 ... python-pkg-resources (20.7.0-1) 설정하는 중입니다 ... python-setuptools (20.7.0-1) 설정하는 중입니다 ... python-wheel (0.29.0-1) 설정하는 중입니다 ...

1. pip 버전 확인하기.

oot@DevGuru-Linux:~# pip -V pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

1. pip 버전 Upgrade. 

root@DevGuru-Linux:~# python -m pip install -U pip Collecting pip   Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)     100% |████████████████████████████████| 1.3MB 603kB/s Installing collected packages: pip   Found existing installation: pip 8.1.1     Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr Successfully installed pip-9.0.1

1. lxml 설치하기.

root@DevGuru-Linux:~# pip install lxml==3.7.0 Collecting lxml==3.7.0   Downloading lxml-3.7.0.tar.gz (3.8MB)     100% |████████████████████████████████| 3.8MB 266kB/s Building wheels for collected packages: lxml   Running setup.py bdist_wheel for lxml ... done   Stored in directory: /root/.cache/pip/wheels/b1/6c/fb/5453b393a084672d8b42e59aec63eafeae4a2c2e1c5dd2485f Successfully built lxml Installing collected packages: lxml Successfully installed lxml-3.7.0

1. Scrapy 설치하기.

root@DevGuru-Linux:~# pip install Scrapy Collecting Scrapy   Downloading Scrapy-1.2.2-py2.py3-none-any.whl (295kB)     100% |████████████████████████████████| 296kB 1.6MB/s Collecting queuelib (from Scrapy)   Downloading queuelib-1.4.2-py2.py3-none-any.whl Collecting pyOpenSSL (from Scrapy)   Downloading pyOpenSSL-16.2.0-py2.py3-none-any.whl (43kB)     100% |████████████████████████████████| 51kB 6.7MB/s Collecting Twisted>=10.0.0 (from Scrapy)   Downloading Twisted-16.6.0.tar.bz2 (3.0MB)     100% |████████████████████████████████| 3.0MB 298kB/s Collecting w3lib>=1.15.0 (from Scrapy)   Downloading w3lib-1.16.0-py2.py3-none-any.whl Collecting six>=1.5.2 (from Scrapy)   Downloading six-1.10.0-py2.py3-none-any.whl Collecting service-identity (from Scrapy)   Downloading service_identity-16.0.0-py2.py3-none-any.whl Collecting cssselect>=0.9 (from Scrapy)   Downloading cssselect-1.0.0-py2.py3-none-any.whl Requirement already satisfied: lxml in /usr/local/lib/python2.7/dist-packages (from Scrapy) Collecting parsel>=0.9.3 (from Scrapy)   Downloading parsel-1.1.0-py2.py3-none-any.whl Collecting PyDispatcher>=2.0.5 (from Scrapy)   Downloading PyDispatcher-2.0.5.tar.gz Collecting cryptography>=1.3.4 (from pyOpenSSL->Scrapy)   Downloading cryptography-1.7.1.tar.gz (420kB)     100% |████████████████████████████████| 430kB 1.8MB/s Collecting zope.interface>=3.6.0 (from Twisted>=10.0.0->Scrapy)   Downloading zope.interface-4.3.3.tar.gz (150kB)     100% |████████████████████████████████| 153kB 4.8MB/s Collecting constantly>=15.1 (from Twisted>=10.0.0->Scrapy)   Downloading constantly-15.1.0-py2.py3-none-any.whl Collecting incremental>=16.10.1 (from Twisted>=10.0.0->Scrapy)   Downloading incremental-16.10.1-py2.py3-none-any.whl Collecting pyasn1 (from service-identity->Scrapy)   Downloading pyasn1-0.1.9-py2.py3-none-any.whl Collecting pyasn1-modules (from service-identity->Scrapy)   Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl Collecting attrs (from service-identity->Scrapy)   Downloading attrs-16.3.0-py2.py3-none-any.whl Collecting idna>=2.0 (from cryptography>=1.3.4->pyOpenSSL->Scrapy)   Downloading idna-2.1-py2.py3-none-any.whl (54kB)     100% |████████████████████████████████| 61kB 8.0MB/s Requirement already satisfied: setuptools>=11.3 in /usr/lib/python2.7/dist-packages (from cryptography>=1.3.4->pyOpenSSL->Scrapy) Collecting enum34 (from cryptography>=1.3.4->pyOpenSSL->Scrapy)   Downloading enum34-1.1.6-py2-none-any.whl Collecting ipaddress (from cryptography>=1.3.4->pyOpenSSL->Scrapy)   Downloading ipaddress-1.0.17-py2-none-any.whl Collecting cffi>=1.4.1 (from cryptography>=1.3.4->pyOpenSSL->Scrapy)   Downloading cffi-1.9.1-cp27-cp27mu-manylinux1_x86_64.whl (387kB)     100% |████████████████████████████████| 389kB 2.4MB/s Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3.4->pyOpenSSL->Scrapy)   Downloading pycparser-2.17.tar.gz (231kB)     100% |████████████████████████████████| 235kB 2.0MB/s Building wheels for collected packages: Twisted, PyDispatcher, cryptography, zope.interface, pycparser   Running setup.py bdist_wheel for Twisted ... done   Stored in directory: /root/.cache/pip/wheels/ba/4d/6b/258f71d2e3d3c4018ff833bcba82fa482e3563157ebd368f66   Running setup.py bdist_wheel for PyDispatcher ... done   Stored in directory: /root/.cache/pip/wheels/86/02/a1/5857c77600a28813aaf0f66d4e4568f50c9f133277a4122411   Running setup.py bdist_wheel for cryptography ... done   Stored in directory: /root/.cache/pip/wheels/35/c3/d6/cc2e097314f1a505e80e232cca8818242ec903f7d9fe727d05   Running setup.py bdist_wheel for zope.interface ... done   Stored in directory: /root/.cache/pip/wheels/00/aa/8b/f1d1eb398423e59894b45ee151344e243808156c2d182c9f4e   Running setup.py bdist_wheel for pycparser ... done   Stored in directory: /root/.cache/pip/wheels/a8/0b/41/dc95621f9d3a0da7bc191b8a71f0e8182ffd3cc5f33ac55005 Successfully built Twisted PyDispatcher cryptography zope.interface pycparser Installing collected packages: queuelib, idna, pyasn1, six, enum34, ipaddress, pycparser, cffi, cryptography, pyOpenSSL, zope.interface, constantly, incremental, Twisted, w3lib, pyasn1-modules, attrs, service-identity, cssselect, parsel, PyDispatcher, Scrapy Successfully installed PyDispatcher-2.0.5 Scrapy-1.2.2 Twisted-16.6.0 attrs-16.3.0 cffi-1.9.1 constantly-15.1.0 cryptography-1.7.1 cssselect-1.0.0 enum34-1.1.6 idna-2.1 incremental-16.10.1 ipaddress-1.0.17 parsel-1.1.0 pyOpenSSL-16.2.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.17 queuelib-1.4.2 service-identity-16.0.0 six-1.10.0 w3lib-1.16.0 zope.interface-4.3.3

1. scrapy 실행 및 확인.

root@DevGuru-Linux:~# scrapy Scrapy 1.2.2 - no active project Usage:   scrapy <command> [options] [args] Available commands:   bench         Run quick benchmark test   commands         fetch         Fetch a URL using the Scrapy downloader   genspider     Generate new spider using pre-defined templates   runspider     Run a self-contained spider (without creating a project)   settings      Get settings values   shell         Interactive scraping console   startproject  Create new project   version       Print Scrapy version   view          Open URL in browser, as seen by Scrapy   [ more ]      More commands available when run from project directory Use "scrapy <command> -h" to see more info about a command