Python requests模塊基礎使用方法實例及高級應用(自動登陸,抓取網頁源碼)實例詳解

發布時間：2020-09-16 13:58:00 來源：腳本之家閱讀：135 作者：返回主頁小L 欄目：開發技術

1、Python requests模塊說明

requests是使用Apache2 licensed 許可證的HTTP庫。

用python編寫。

比urllib2模塊更簡潔。

Request支持HTTP連接保持和連接池，支持使用cookie保持會話，支持文件上傳，支持自動響應內容的編碼，支持國際化的URL和POST數據自動編碼。

在python內置模塊的基礎上進行了高度的封裝，從而使得python進行網絡請求時，變得人性化，使用Requests可以輕而易舉的完成瀏覽器可有的任何操作。

現代，國際化，友好。

requests會自動實現持久連接keep-alive

2、Python requests模塊基礎入門

1）導入模塊

import requests

2）發送請求的簡潔

示例代碼：獲取一個網頁（個人github）

import requests
r = requests.get('https://github.com/Ranxf')    # 最基本的不帶參數的get請求
r1 = requests.get(url='http://dict.baidu.com/s', params={'wd': 'python'})   # 帶參數的get請求

我們還可以使用requests模塊其它請求方法

1 requests.get(‘https://github.com/timeline.json') # GET請求

2 requests.post(“http://httpbin.org/post”) # POST請求

3 requests.put(“http://httpbin.org/put”) # PUT請求

4 requests.delete(“http://httpbin.org/delete”) # DELETE請求

5 requests.head(“http://httpbin.org/get”) # HEAD請求

6 requests.options(“http://httpbin.org/get” ) # OPTIONS請求

3）為url傳遞參數

>>> url_params = {'key':'value'}    #  字典傳遞參數，如果值為None的鍵不會被添加到url中
>>> r = requests.get('your url',params = url_params)
>>> print(r.url)

　　your url?key=value

4）響應的內容

r.encoding #獲取當前的編碼

r.encoding = 'utf-8' #設置編碼

r.text #以encoding解析返回內容。字符串方式的響應體，會自動根據響應頭部的字符編碼進行解碼。

r.content #以字節形式（二進制）返回。字節方式的響應體，會自動為你解碼 gzip 和 deflate 壓縮。

r.headers #以字典對象存儲服務器響應頭，但是這個字典比較特殊，字典鍵不區分大小寫，若鍵不存在則返回None

r.status_code #響應狀態碼

r.raw #返回原始響應體，也就是 urllib 的 response 對象，使用 r.raw.read()

r.ok # 查看r.ok的布爾值便可以知道是否登陸成功

#*特殊方法*#

r.json() #Requests中內置的JSON解碼器，以json形式返回,前提返回的內容確保是json格式的，不然解析出錯會拋異常

r.raise_for_status() #失敗請求(非200響應)拋出異常

post發送json請求：

import requests
import json
 
r = requests.post('https://api.github.com/some/endpoint', data=json.dumps({'some': 'data'}))

print(r.json())

5）定制頭和cookie信息

header = {'user-agent': 'my-app/0.0.1''}
cookie = {'key':'value'}
 r = requests.get/post('your url',headers=header,cookies=cookie) 
data = {'some': 'data'}
headers = {'content-type': 'application/json',
      'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0'}
 
r = requests.post('https://api.github.com/some/endpoint', data=data, headers=headers)
print(r.text)

6）響應狀態碼

使用requests方法后，會返回一個response對象，其存儲了服務器響應的內容，如上實例中已經提到的 r.text、r.status_code……

獲取文本方式的響應體實例：當你訪問 r.text 之時，會使用其響應的文本編碼進行解碼，并且你可以修改其編碼讓 r.text 使用自定義的編碼進行解碼。

r = requests.get('http://www.itwhy.org')
print(r.text, '\n{}\n'.format('*'*79), r.encoding)
r.encoding = 'GBK'
print(r.text, '\n{}\n'.format('*'*79), r.encoding)

示例代碼：

import requests

r = requests.get('https://github.com/Ranxf')    # 最基本的不帶參數的get請求
print(r.status_code)                # 獲取返回狀態
r1 = requests.get(url='http://dict.baidu.com/s', params={'wd': 'python'})   # 帶參數的get請求
print(r1.url)
print(r1.text)    # 打印解碼后的返回數據

運行結果：

/usr/bin/python3.5 /home/rxf/python3_1000/1000/python3_server/python3_requests/demo1.py

200

http://dict.baidu.com/s?wd=python

…………

Process finished with exit code 0

r.status_code #如果不是200，可以使用 r.raise_for_status() 拋出異常

7）響應

r.headers #返回字典類型,頭信息

r.requests.headers #返回發送到服務器的頭信息

r.cookies #返回cookie

r.history #返回重定向信息,當然可以在請求是加上allow_redirects = false 阻止重定向

8）超時

r = requests.get('url',timeout=1)      #設置秒數超時，僅對于連接有效

9)會話對象，能夠跨請求保持某些參數

s = requests.Session()
s.auth = ('auth','passwd')
s.headers = {'key':'value'}
r = s.get('url')
r1 = s.get('url1')

10）代理

proxies = {'http':'ip1','https':'ip2' }
requests.get('url',proxies=proxies)

匯總：

# HTTP請求類型
# get類型
r = requests.get('https://github.com/timeline.json')
# post類型
r = requests.post("http://m.ctrip.com/post")
# put類型
r = requests.put("http://m.ctrip.com/put")
# delete類型
r = requests.delete("http://m.ctrip.com/delete")
# head類型
r = requests.head("http://m.ctrip.com/head")
# options類型
r = requests.options("http://m.ctrip.com/get")

# 獲取響應內容
print(r.content) #以字節的方式去顯示，中文顯示為字符
print(r.text) #以文本的方式去顯示

#URL傳遞參數
payload = {'keyword': '香港', 'salecityid': '2'}
r = requests.get("http://m.ctrip.com/webapp/tourvisa/visa_list", params=payload) 
print（r.url） #示例為http://m.ctrip.com/webapp/tourvisa/visa_list?salecityid=2&keyword=香港

#獲取/修改網頁編碼
r = requests.get('https://github.com/timeline.json')
print （r.encoding）


#json處理
r = requests.get('https://github.com/timeline.json')
print（r.json()） # 需要先import json  

# 定制請求頭
url = 'http://m.ctrip.com'
headers = {'User-Agent' : 'Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19'}
r = requests.post(url, headers=headers)
print （r.request.headers)

#復雜post請求
url = 'http://m.ctrip.com'
payload = {'some': 'data'}
r = requests.post(url, data=json.dumps(payload)) #如果傳遞的payload是string而不是dict，需要先調用dumps方法格式化一下

# post多部分編碼文件
url = 'http://m.ctrip.com'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)

# 響應狀態碼
r = requests.get('http://m.ctrip.com')
print(r.status_code)
  
# 響應頭
r = requests.get('http://m.ctrip.com')
print (r.headers)
print (r.headers['Content-Type'])
print (r.headers.get('content-type')) #訪問響應頭部分內容的兩種方式
  
# Cookies
url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies['example_cookie_name']  #讀取cookies
  
url = 'http://m.ctrip.com/cookies'
cookies = dict(cookies_are='working')
r = requests.get(url, cookies=cookies) #發送cookies

#設置超時時間
r = requests.get('http://m.ctrip.com', timeout=0.001)

#設置訪問代理
proxies = {
      "http": "http://10.10.1.10:3128",
      "https": "http://10.10.1.100:4444",
     }
r = requests.get('http://m.ctrip.com', proxies=proxies)


#如果代理需要用戶名和密碼，則需要這樣：
proxies = {
  "http": "http://user:pass@10.10.1.10:3128/",
}

# HTTP請求類型
# get類型
r = requests.get('https://github.com/timeline.json')
# post類型
r = requests.post("http://m.ctrip.com/post")
# put類型
r = requests.put("http://m.ctrip.com/put")
# delete類型
r = requests.delete("http://m.ctrip.com/delete")
# head類型
r = requests.head("http://m.ctrip.com/head")
# options類型
r = requests.options("http://m.ctrip.com/get")

# 獲取響應內容
print(r.content) #以字節的方式去顯示，中文顯示為字符
print(r.text) #以文本的方式去顯示

#URL傳遞參數
payload = {'keyword': '香港', 'salecityid': '2'}
r = requests.get("http://m.ctrip.com/webapp/tourvisa/visa_list", params=payload) 
print（r.url） #示例為http://m.ctrip.com/webapp/tourvisa/visa_list?salecityid=2&keyword=香港

#獲取/修改網頁編碼
r = requests.get('https://github.com/timeline.json')
print （r.encoding）


#json處理
r = requests.get('https://github.com/timeline.json')
print（r.json()） # 需要先import json  

# 定制請求頭
url = 'http://m.ctrip.com'
headers = {'User-Agent' : 'Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19'}
r = requests.post(url, headers=headers)
print （r.request.headers)

#復雜post請求
url = 'http://m.ctrip.com'
payload = {'some': 'data'}
r = requests.post(url, data=json.dumps(payload)) #如果傳遞的payload是string而不是dict，需要先調用dumps方法格式化一下

# post多部分編碼文件
url = 'http://m.ctrip.com'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)

# 響應狀態碼
r = requests.get('http://m.ctrip.com')
print(r.status_code)
  
# 響應頭
r = requests.get('http://m.ctrip.com')
print (r.headers)
print (r.headers['Content-Type'])
print (r.headers.get('content-type')) #訪問響應頭部分內容的兩種方式
  
# Cookies
url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies['example_cookie_name']  #讀取cookies
  
url = 'http://m.ctrip.com/cookies'
cookies = dict(cookies_are='working')
r = requests.get(url, cookies=cookies) #發送cookies

#設置超時時間
r = requests.get('http://m.ctrip.com', timeout=0.001)

#設置訪問代理
proxies = {
      "http": "http://10.10.1.10:3128",
      "https": "http://10.10.1.100:4444",
     }
r = requests.get('http://m.ctrip.com', proxies=proxies)


#如果代理需要用戶名和密碼，則需要這樣：
proxies = {
  "http": "http://user:pass@10.10.1.10:3128/",
}

3、示例代碼

GET請求

# 1、無參數實例
 
import requests
 
ret = requests.get('https://github.com/timeline.json')
 
print(ret.url)
print(ret.text)
 
 
 
# 2、有參數實例
 
import requests
 
payload = {'key1': 'value1', 'key2': 'value2'}
ret = requests.get("http://httpbin.org/get", params=payload)
 
print(ret.url)
print(ret.text)

POST請求

# 1、基本POST實例
 
import requests
 
payload = {'key1': 'value1', 'key2': 'value2'}
ret = requests.post("http://httpbin.org/post", data=payload)
 
print(ret.text)
 
 
# 2、發送請求頭和數據實例
 
import requests
import json
 
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}
headers = {'content-type': 'application/json'}
 
ret = requests.post(url, data=json.dumps(payload), headers=headers)
 
print(ret.text)
print(ret.cookies)

請求參數

def request(method, url, **kwargs):

"""Constructs and sends a :class:`Request <Request>`.

:param method: method for the new :class:`Request` object.

:param url: URL for the new :class:`Request` object.

:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.

:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.

:param json: (optional) json data to send in the body of the :class:`Request`.

:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.

:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.

:param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.

``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``

or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string

defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers

to add for the file.

:param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.

:param timeout: (optional) How long to wait for the server to send data

before giving up, as a float, or a :ref:`(connect timeout, read

timeout) <timeouts>` tuple.

:type timeout: float or tuple

:param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.

:type allow_redirects: bool

:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.

:param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.

:param stream: (optional) if ``False``, the response content will be immediately downloaded.

:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.

:return: :class:`Response <Response>` object

:rtype: requests.Response

Usage::

>>> import requests

>>> req = requests.request('GET', 'http://httpbin.org/get')

<Response [200]>

參數示例代碼

def param_method_url():
  # requests.request(method='get', url='http://127.0.0.1:8000/test/')
  # requests.request(method='post', url='http://127.0.0.1:8000/test/')
  pass


def param_param():
  # - 可以是字典
  # - 可以是字符串
  # - 可以是字節（ascii編碼以內）

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params={'k1': 'v1', 'k2': '水電費'})

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params="k1=v1&k2=水電費&k3=v3&k3=vv3")

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8'))

  # 錯誤
  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=水電費&k3=v3&k3=vv3", encoding='utf8'))
  pass


def param_data():
  # 可以是字典
  # 可以是字符串
  # 可以是字節
  # 可以是文件對象

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data={'k1': 'v1', 'k2': '水電費'})

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1; k2=v2; k3=v3; k3=v4"
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1;k2=v2;k3=v3;k3=v4",
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件內容是：k1=v1;k2=v2;k3=v3;k3=v4
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )
  pass


def param_json():
  # 將json中對應的數據進行序列化成一個字符串，json.dumps(...)
  # 然后發送到服務器端的body中，并且Content-Type是 {'Content-Type': 'application/json'}
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水電費'})


def param_headers():
  # 發送請求頭到服務器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水電費'},
           headers={'Content-Type': 'application/x-www-form-urlencoded'}
           )


def param_cookies():
  # 發送Cookie到服務器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies={'cook1': 'value1'},
           )
  # 也可以使用CookieJar（字典形式就是在此基礎上封裝）
  from http.cookiejar import CookieJar
  from http.cookiejar import Cookie

  obj = CookieJar()
  obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None,
             discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False,
             port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False)
          )
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies=obj)


def param_files():
  # 發送文件
  # file_dict = {
  # 'f1': open('readme', 'rb')
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 發送文件，定制文件名
  # file_dict = {
  # 'f1': ('test.txt', open('readme', 'rb'))
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 發送文件，定制文件名
  # file_dict = {
  # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf")
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 發送文件，定制文件名
  # file_dict = {
  #   'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'})
  # }
  # requests.request(method='POST',
  #         url='http://127.0.0.1:8000/test/',
  #         files=file_dict)

  pass


def param_auth():
  from requests.auth import HTTPBasicAuth, HTTPDigestAuth

  ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))
  print(ret.text)

  # ret = requests.get('http://192.168.1.1',
  # auth=HTTPBasicAuth('admin', 'admin'))
  # ret.encoding = 'gbk'
  # print(ret.text)

  # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass'))
  # print(ret)
  #


def param_timeout():
  # ret = requests.get('http://google.com/', timeout=1)
  # print(ret)

  # ret = requests.get('http://google.com/', timeout=(5, 1))
  # print(ret)
  pass


def param_allow_redirects():
  ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False)
  print(ret.text)


def param_proxies():
  # proxies = {
  # "http": "61.172.249.96:80",
  # "https": "http://61.185.219.126:3128",
  # }

  # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}

  # ret = requests.get("http://www.proxy#/Proxy", proxies=proxies)
  # print(ret.headers)


  # from requests.auth import HTTPProxyAuth
  #
  # proxyDict = {
  # 'http': '77.75.105.165',
  # 'https': '77.75.105.165'
  # }
  # auth = HTTPProxyAuth('username', 'mypassword')
  #
  # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth)
  # print(r.text)

  pass


def param_stream():
  ret = requests.get('http://127.0.0.1:8000/test/', stream=True)
  print(ret.content)
  ret.close()

  # from contextlib import closing
  # with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
  # # 在此處理響應。
  # for i in r.iter_content():
  # print(i)


def requests_session():
  import requests

  session = requests.Session()

  ### 1、首先登陸任何頁面，獲取cookie

  i1 = session.get(url="http://dig.chouti.com/help/service")

  ### 2、用戶登陸，攜帶上一次的cookie，后臺對cookie中的 gpsd 進行授權
  i2 = session.post(
    url="http://dig.chouti.com/login",
    data={
      'phone': "8615131255089",
      'password': "xxxxxx",
      'oneMonth': ""
    }
  )

  i3 = session.post(
    url="http://dig.chouti.com/link/vote?linksId=8589623",
  )
  print(i3.text)

json請求：

#! /usr/bin/python3
import requests
import json


class url_request():
  def __init__(self):
    ''' init '''

if __name__ == '__main__':
  heard = {'Content-Type': 'application/json'}
  payload = {'CountryName': '中國',
        'ProvinceName': '四川省',
        'L1CityName': 'chengdu',
        'L2CityName': 'yibing',
        'TownName': '',
        'Longitude': '107.33393',
        'Latitude': '33.157131',
        'Language': 'CN'}
  r = requests.post("http://www.xxxxxx.com/CityLocation/json/LBSLocateCity", heards=heard, data=payload)
  data = r.json()
  if r.status_code!=200:
    print('LBSLocateCity API Error' + str(r.status_code))
  print(data['CityEntities'][0]['CityID']) # 打印返回json中的某個key的value
  print(data['ResponseStatus']['Ack'])
  print(json.dump(data, indent=4, sort_keys=True, ensure_ascii=False)) # 樹形打印json，ensure_ascii必須設為False否則中文會顯示為unicode

Xml請求：

#! /usr/bin/python3
import requests

class url_request():
  def __init__(self):
    """init"""

if __name__ == '__main__':
  heards = {'Content-type': 'text/xml'}
  XML = '<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><Request xmlns="http://tempuri.org/"><jme><JobClassFullName>WeChatJSTicket.JobWS.Job.JobRefreshTicket,WeChatJSTicket.JobWS</JobClassFullName><Action>RUN</Action><Param>1</Param><HostIP>127.0.0.1</HostIP><JobInfo>1</JobInfo><NeedParallel>false</NeedParallel></jme></Request></soap:Body></soap:Envelope>'
  url = 'http://jobws.push.mobile.xxxxxxxx.com/RefreshWeiXInTokenJob/RefreshService.asmx'
  r = requests.post(url=url, heards=heards, data=XML)
  data = r.text
  print(data)

狀態異常處理

import requests

URL = 'http://ip.taobao.com/service/getIpInfo.php' # 淘寶IP地址庫API
try:
  r = requests.get(URL, params={'ip': '8.8.8.8'}, timeout=1)
  r.raise_for_status() # 如果響應狀態碼不是 200，就主動拋出異常
except requests.RequestException as e:
  print(e)
else:
  result = r.json()
  print(type(result), result, sep='\n')

上傳文件

使用request模塊，也可以上傳文件，文件的類型會自動進行處理：

import requests
 
url = 'http://127.0.0.1:8080/upload'
files = {'file': open('/home/rxf/test.jpg', 'rb')}
#files = {'file': ('report.jpg', open('/home/lyb/sjzl.mpg', 'rb'))}   #顯式的設置文件名
 
r = requests.post(url, files=files)
print(r.text)

request更加方便的是，可以把字符串當作文件進行上傳：

import requests
 
url = 'http://127.0.0.1:8080/upload'
files = {'file': ('test.txt', b'Hello Requests.')}   #必需顯式的設置文件名
 
r = requests.post(url, files=files)
print(r.text)

身份驗證

基本身份認證(HTTP Basic Auth)

import requests
from requests.auth import HTTPBasicAuth
 
r = requests.get('https://httpbin.org/hidden-basic-auth/user/passwd', auth=HTTPBasicAuth('user', 'passwd'))
# r = requests.get('https://httpbin.org/hidden-basic-auth/user/passwd', auth=('user', 'passwd'))  # 簡寫
print(r.json())

另一種非常流行的HTTP身份認證形式是摘要式身份認證，Requests對它的支持也是開箱即可用的:

requests.get(URL, auth=HTTPDigestAuth('user', 'pass')

Cookies與會話對象

如果某個響應中包含一些Cookie，你可以快速訪問它們：

import requests
 
r = requests.get('http://www.google.com.hk/')
print(r.cookies['NID'])
print(tuple(r.cookies))

要想發送你的cookies到服務器，可以使用 cookies 參數：

import requests
 
url = 'http://httpbin.org/cookies'
cookies = {'testCookies_1': 'Hello_Python3', 'testCookies_2': 'Hello_Requests'}
# 在Cookie Version 0中規定空格、方括號、圓括號、等于號、逗號、雙引號、斜杠、問號、@，冒號，分號等特殊符號都不能作為Cookie的內容。
r = requests.get(url, cookies=cookies)
print(r.json())

會話對象讓你能夠跨請求保持某些參數，最方便的是在同一個Session實例發出的所有請求之間保持cookies，且這些都是自動處理的，甚是方便。

下面就來一個真正的實例，如下是快盤簽到腳本：

import requests
 
headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'Accept-Encoding': 'gzip, deflate, compress',
      'Accept-Language': 'en-us;q=0.5,en;q=0.3',
      'Cache-Control': 'max-age=0',
      'Connection': 'keep-alive',
      'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0'}
 
s = requests.Session()
s.headers.update(headers)
# s.auth = ('superuser', '123')
s.get('https://www.kuaipan.cn/account_login.htm')
 
_URL = 'http://www.kuaipan.cn/index.php'
s.post(_URL, params={'ac':'account', 'op':'login'},
    data={'username':'****@foxmail.com', 'userpwd':'********', 'isajax':'yes'})
r = s.get(_URL, params={'ac':'zone', 'op':'taskdetail'})
print(r.json())
s.get(_URL, params={'ac':'common', 'op':'usersign'})

requests模塊抓取網頁源碼并保存到文件示例

這是一個基本的文件保存操作，但這里有幾個值得注意的問題：

1.安裝requests包，命令行輸入pip install requests即可自動安裝。很多人推薦使用requests，自帶的urllib.request也可以抓取網頁源碼

2.open方法encoding參數設為utf-8，否則保存的文件會出現亂碼。

3.如果直接在cmd中輸出抓取的內容，會提示各種編碼錯誤，所以保存到文件查看。

4.with open方法是更好的寫法，可以自動操作完畢后釋放資源

Python requests模塊抽屜自動登錄

#! /urs/bin/python3
import requests

'''requests模塊抓取網頁源碼并保存到文件示例'''
html = requests.get("http://www.baidu.com")
with open('test.txt', 'w', encoding='utf-8') as f:
  f.write(html.text)
  
'''讀取一個txt文件，每次讀取一行，并保存到另一個txt文件中的示例'''
ff = open('testt.txt', 'w', encoding='utf-8')
with open('test.txt', encoding="utf-8") as f:
  for line in f:
    ff.write(line)
    ff.close()

因為在命令行中打印每次讀取一行的數據，中文會出現編碼錯誤，所以每次讀取一行并保存到另一個文件，這樣來測試讀取是否正常。（注意open的時候制定encoding編碼方式）

Python requests模塊自動登陸實例：

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import requests


# ############## 方式一 ##############
"""
# ## 1、首先登陸任何頁面，獲取cookie
i1 = requests.get(url="http://dig.chouti.com/help/service")
i1_cookies = i1.cookies.get_dict()

# ## 2、用戶登陸，攜帶上一次的cookie，后臺對cookie中的 gpsd 進行授權
i2 = requests.post(
  url="http://dig.chouti.com/login",
  data={
    'phone': "8615131255089",
    'password': "xxooxxoo",
    'oneMonth': ""
  },
  cookies=i1_cookies
)

# ## 3、點贊（只需要攜帶已經被授權的gpsd即可）
gpsd = i1_cookies['gpsd']
i3 = requests.post(
  url="http://dig.chouti.com/link/vote?linksId=8589523",
  cookies={'gpsd': gpsd}
)

print(i3.text)
"""


# ############## 方式二 ##############
"""
import requests

session = requests.Session()
i1 = session.get(url="http://dig.chouti.com/help/service")
i2 = session.post(
  url="http://dig.chouti.com/login",
  data={
    'phone': "8615131255089",
    'password': "xxooxxoo",
    'oneMonth': ""
  }
)
i3 = session.post(
  url="http://dig.chouti.com/link/vote?linksId=8589523"
)
print(i3.text)

"""

Python requests模塊github自動登錄

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests
from bs4 import BeautifulSoup

# ############## 方式一 ##############
#
# # 1. 訪問登陸頁面，獲取 authenticity_token
# i1 = requests.get('https://github.com/login')
# soup1 = BeautifulSoup(i1.text, features='lxml')
# tag = soup1.find(name='input', attrs={'name': 'authenticity_token'})
# authenticity_token = tag.get('value')
# c1 = i1.cookies.get_dict()
# i1.close()
#
# # 1. 攜帶authenticity_token和用戶名密碼等信息，發送用戶驗證
# form_data = {
# "authenticity_token": authenticity_token,
#   "utf8": "",
#   "commit": "Sign in",
#   "login": "wupeiqi@live.com",
#   'password': 'xxoo'
# }
#
# i2 = requests.post('https://github.com/session', data=form_data, cookies=c1)
# c2 = i2.cookies.get_dict()
# c1.update(c2)
# i3 = requests.get('https://github.com/settings/repositories', cookies=c1)
#
# soup3 = BeautifulSoup(i3.text, features='lxml')
# list_group = soup3.find(name='div', class_='listgroup')
#
# from bs4.element import Tag
#
# for child in list_group.children:
#   if isinstance(child, Tag):
#     project_tag = child.find(name='a', class_='mr-1')
#     size_tag = child.find(name='small')
#     temp = "項目:%s(%s); 項目路徑:%s" % (project_tag.get('href'), size_tag.string, project_tag.string, )
#     print(temp)



# ############## 方式二 ##############
# session = requests.Session()
# # 1. 訪問登陸頁面，獲取 authenticity_token
# i1 = session.get('https://github.com/login')
# soup1 = BeautifulSoup(i1.text, features='lxml')
# tag = soup1.find(name='input', attrs={'name': 'authenticity_token'})
# authenticity_token = tag.get('value')
# c1 = i1.cookies.get_dict()
# i1.close()
#
# # 1. 攜帶authenticity_token和用戶名密碼等信息，發送用戶驗證
# form_data = {
#   "authenticity_token": authenticity_token,
#   "utf8": "",
#   "commit": "Sign in",
#   "login": "wupeiqi@live.com",
#   'password': 'xxoo'
# }
#
# i2 = session.post('https://github.com/session', data=form_data)
# c2 = i2.cookies.get_dict()
# c1.update(c2)
# i3 = session.get('https://github.com/settings/repositories')
#
# soup3 = BeautifulSoup(i3.text, features='lxml')
# list_group = soup3.find(name='div', class_='listgroup')
#
# from bs4.element import Tag
#
# for child in list_group.children:
#   if isinstance(child, Tag):
#     project_tag = child.find(name='a', class_='mr-1')
#     size_tag = child.find(name='small')
#     temp = "項目:%s(%s); 項目路徑:%s" % (project_tag.get('href'), size_tag.string, project_tag.string, )
#     print(temp)

Python requests模塊知乎自動登錄

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import time

import requests
from bs4 import BeautifulSoup

session = requests.Session()

i1 = session.get(
  url='https://www.zhihu.com/#signin',
  headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
  }
)

soup1 = BeautifulSoup(i1.text, 'lxml')
xsrf_tag = soup1.find(name='input', attrs={'name': '_xsrf'})
xsrf = xsrf_tag.get('value')

current_time = time.time()
i2 = session.get(
  url='https://cache.yisu.com/upload/information/20200622/113/12088.gif',
  params={'r': current_time, 'type': 'login'},
  headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
  })

with open('zhihu.gif', 'wb') as f:
  f.write(i2.content)

captcha = input('請打開zhihu.gif文件，查看并輸入驗證碼：')
form_data = {
  "_xsrf": xsrf,
  'password': 'xxooxxoo',
  "captcha": 'captcha',
  'email': '424662508@qq.com'
}
i3 = session.post(
  url='https://www.zhihu.com/login/email',
  data=form_data,
  headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
  }
)

i4 = session.get(
  url='https://www.zhihu.com/settings/profile',
  headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
  }
)

soup4 = BeautifulSoup(i4.text, 'lxml')
tag = soup4.find(id='rename-section')
nick_name = tag.find('span',class_='name').string
print(nick_name)

Python requests模塊博客園自動登錄

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import re
import json
import base64

import rsa
import requests


def js_encrypt(text):
  b64der = 'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCp0wHYbg/NOPO3nzMD3dndwS0MccuMeXCHgVlGOoYyFwLdS24Im2e7YyhB0wrUsyYf0/nhzCzBK8ZC9eCWqd0aHbdgOQT6CuFQBMjbyGYvlVYU2ZP7kG9Ft6YV6oc9ambuO7nPZh+bvXH0zDKfi02prknrScAKC0XhadTHT3Al0QIDAQAB'
  der = base64.standard_b64decode(b64der)

  pk = rsa.PublicKey.load_pkcs1_openssl_der(der)
  v1 = rsa.encrypt(bytes(text, 'utf8'), pk)
  value = base64.encodebytes(v1).replace(b'\n', b'')
  value = value.decode('utf8')

  return value


session = requests.Session()

i1 = session.get('https://passport.cnblogs.com/user/signin')
rep = re.compile("'VerificationToken': '(.*)'")
v = re.search(rep, i1.text)
verification_token = v.group(1)

form_data = {
  'input1': js_encrypt('wptawy'),
  'input2': js_encrypt('asdfasdf'),
  'remember': False
}

i2 = session.post(url='https://passport.cnblogs.com/user/signin',
         data=json.dumps(form_data),
         headers={
           'Content-Type': 'application/json; charset=UTF-8',
           'X-Requested-With': 'XMLHttpRequest',
           'VerificationToken': verification_token}
         )

i3 = session.get(url='https://i.cnblogs.com/EditDiary.aspx')

print(i3.text)

Python requests模塊拉勾網自動登錄

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import requests


# 第一步：訪問登陸頁,拿到X_Anti_Forge_Token，X_Anti_Forge_Code
# 1、請求url:https://passport.lagou.com/login/login.html
# 2、請求方法:GET
# 3、請求頭:
#  User-agent
r1 = requests.get('https://passport.lagou.com/login/login.html',
         headers={
           'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
         },
         )

X_Anti_Forge_Token = re.findall("X_Anti_Forge_Token = '(.*?)'", r1.text, re.S)[0]
X_Anti_Forge_Code = re.findall("X_Anti_Forge_Code = '(.*?)'", r1.text, re.S)[0]
print(X_Anti_Forge_Token, X_Anti_Forge_Code)
# print(r1.cookies.get_dict())
# 第二步：登陸
# 1、請求url:https://passport.lagou.com/login/login.json
# 2、請求方法:POST
# 3、請求頭:
#  cookie
#  User-agent
#  Referer:https://passport.lagou.com/login/login.html
#  X-Anit-Forge-Code:53165984
#  X-Anit-Forge-Token:3b6a2f62-80f0-428b-8efb-ef72fc100d78
#  X-Requested-With:XMLHttpRequest
# 4、請求體：
# isValidate:true
# username:15131252215
# password:ab18d270d7126ea65915c50288c22c0d
# request_form_verifyCode:''
# submit:''
r2 = requests.post(
  'https://passport.lagou.com/login/login.json',
  headers={
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
    'Referer': 'https://passport.lagou.com/login/login.html',
    'X-Anit-Forge-Code': X_Anti_Forge_Code,
    'X-Anit-Forge-Token': X_Anti_Forge_Token,
    'X-Requested-With': 'XMLHttpRequest'
  },
  data={
    "isValidate": True,
    'username': '15131255089',
    'password': 'ab18d270d7126ea65915c50288c22c0d',
    'request_form_verifyCode': '',
    'submit': ''
  },
  cookies=r1.cookies.get_dict()
)
print(r2.text)

更多關于Python requests模塊基礎使用方法請查看下面的相關鏈接

向AI問一下細節

91超碰碰碰碰久久久久久综合_超碰av人澡人澡人澡人澡人掠_国产黄大片在线观看画质优化_txt小说免费全本

Python requests模塊基礎使用方法實例及高級應用(自動登陸,抓取網頁源碼)實例詳解

1、Python requests模塊說明

2、Python requests模塊基礎入門

1）導入模塊

2）發送請求的簡潔

3）為url傳遞參數

4）響應的內容

post發送json請求：

5）定制頭和cookie信息

6）響應狀態碼

7）響應

8）超時

9)會話對象，能夠跨請求保持某些參數

10）代理

3、示例代碼

GET請求

POST請求

json請求：

Xml請求：

狀態異常處理

上傳文件

身份驗證

Cookies與會話對象

requests模塊抓取網頁源碼并保存到文件示例

猜你喜歡

91超碰碰碰碰久久久久久综合_超碰av人澡人澡人澡人澡人掠_国产黄大片在线观看画质优化_txt小说免费全本

Python requests模塊基礎使用方法實例及高級應用(自動登陸,抓取網頁源碼)實例詳解

1、Python requests模塊說明

2、Python requests模塊基礎入門

1）導入模塊

2）發送請求的簡潔

3）為url傳遞參數

4）響應的內容

post發送json請求：

5）定制頭和cookie信息

6）響應狀態碼

7）響應

8）超時

9)會話對象，能夠跨請求保持某些參數

10）代理

3、示例代碼

GET請求

POST請求

json請求：

Xml請求：

狀態異常處理

上傳文件

身份驗證

Cookies與會話對象

requests模塊抓取網頁源碼并保存到文件示例

猜你喜歡

最新資訊

相關推薦

相關標簽