在Python中實現并發上千個請求有多種方法。以下是一些常用的方法:
threading
模塊創建和管理多個線程,并發發送請求。每個線程可以負責發送一個請求。可以使用線程池來管理和控制線程的數量。import threading
import requests
def send_request(url):
response = requests.get(url)
print(response.text)
urls = [...] # 存儲要發送請求的URL列表
threads = []
for url in urls:
thread = threading.Thread(target=send_request, args=(url,))
thread.start()
threads.append(thread)
for thread in threads:
thread.join()
asyncio
模塊和aiohttp
庫來實現并發請求。協程是一種輕量級的線程,可以在單個線程中實現并發。通過使用async
和await
關鍵字,可以創建異步函數,并發執行請求。import asyncio
import aiohttp
async def send_request(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
data = await response.text()
print(data)
urls = [...] # 存儲要發送請求的URL列表
loop = asyncio.get_event_loop()
tasks = [send_request(url) for url in urls]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
grequests
或gevent
,來實現并發請求。這些庫可以在單個線程中并發執行多個請求。使用grequests
庫的示例:
import grequests
urls = [...] # 存儲要發送請求的URL列表
requests = [grequests.get(url) for url in urls]
responses = grequests.map(requests)
for response in responses:
print(response.text)
使用gevent
庫的示例:
import gevent
import requests
def send_request(url):
response = requests.get(url)
print(response.text)
urls = [...] # 存儲要發送請求的URL列表
greenlets = [gevent.spawn(send_request, url) for url in urls]
gevent.joinall(greenlets)
無論選擇哪種方法,都要注意控制并發請求的數量,以避免過多的資源消耗或服務器超載。