溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
這篇文章將為大家詳細講解有關利用Python如何實現根據URL地址下載并保存文件至對應目錄，文章內容質量較高，因此小編分享給大家做個參考，希望大家閱讀完這篇文章后對相關知識有一定的了解。
引言
在編程中經常會遇到圖片等數據集將圖片等數據以URL形式存儲在txt文檔中，為便于后續的分析，需要將其下載下來，并按照文件夾分類存儲。本文以Github中Alexander Kim提供的圖片分類數據集為例，下載其提供的圖片樣本并分類保存
Python 3.6.5，Anaconda， VSCode
1. 下載數據集文件
建立項目文件夾，下載上述Github項目中的raw_data文件夾，并保存至項目目錄中。
2. 獲取樣本文件位置
編寫get_doc_path.py，根據根目錄位置，獲取目錄及其子目錄所有數據集文件
import os
def get_file(root_path, all_files={}):
'''
遞歸函數，遍歷該文檔目錄和子目錄下的所有文件，獲取其path
'''
files = os.listdir(root_path)
for file in files:
if not os.path.isdir(root_path + '/' + file): # not a dir
all_files[file] = root_path + '/' + file
else: # is a dir
get_file((root_path+'/'+file), all_files)
return all_files
if __name__ == '__main__':
path = './raw_data'
print(get_file(path))3. 下載文件
3.1 讀取url列表并
for filename, path in paths.items():
print('reading file: {}'.format(filename))
with open(path, 'r') as f:
lines = f.readlines()
url_list = []
for line in lines:
url_list.append(line.strip('\n'))
print(url_list)3.2 創建文件夾
foldername = "./picture_get_by_url/pic_download/{}".format(filename.split('.')[0])
if not os.path.exists(folder_path):
print("Selected folder not exist, try to create it.")
os.makedirs(folder_path)3.3 下載圖片
def get_pic_by_url(folder_path, lists):
if not os.path.exists(folder_path):
print("Selected folder not exist, try to create it.")
os.makedirs(folder_path)
for url in lists:
print("Try downloading file: {}".format(url))
filename = url.split('/')[-1]
filepath = folder_path + '/' + filename
if os.path.exists(filepath):
print("File have already exist. skip")
else:
try:
urllib.request.urlretrieve(url, filename=filepath)
except Exception as e:
print("Error occurred when downloading file, error message:")
print(e)4. 完整源碼
4.1 get_doc_path.py
import os
def get_file(root_path, all_files={}):
'''
遞歸函數，遍歷該文檔目錄和子目錄下的所有文件，獲取其path
'''
files = os.listdir(root_path)
for file in files:
if not os.path.isdir(root_path + '/' + file): # not a dir
all_files[file] = root_path + '/' + file
else: # is a dir
get_file((root_path+'/'+file), all_files)
return all_files
if __name__ == '__main__':
path = './raw_data'
print(get_file(path))4.2 get_pic.py
import get_doc_path
import os
import urllib.request
def get_pic_by_url(folder_path, lists):
if not os.path.exists(folder_path):
print("Selected folder not exist, try to create it.")
os.makedirs(folder_path)
for url in lists:
print("Try downloading file: {}".format(url))
filename = url.split('/')[-1]
filepath = folder_path + '/' + filename
if os.path.exists(filepath):
print("File have already exist. skip")
else:
try:
urllib.request.urlretrieve(url, filename=filepath)
except Exception as e:
print("Error occurred when downloading file, error message:")
print(e)
if __name__ == "__main__":
root_path = './picture_get_by_url/raw_data'
paths = get_doc_path.get_file(root_path)
print(paths)
for filename, path in paths.items():
print('reading file: {}'.format(filename))
with open(path, 'r') as f:
lines = f.readlines()
url_list = []
for line in lines:
url_list.append(line.strip('\n'))
foldername = "./picture_get_by_url/pic_download/{}".format(filename.split('.')[0])
get_pic_by_url(foldername, url_list)4.3 運行結果
執行get_pic.py
當程序意外停止或再次執行時，程序會自動跳過文件夾中已下載的文件，繼續下載未下載的內容
{‘urls_drawings.txt': ‘./picture_get_by_url/raw_data/drawings/urls_drawings.txt', ‘urls_hentai.txt': ‘./picture_get_by_url/raw_data/hentai/urls_hentai.txt', ‘urls_neutral.txt': ‘./picture_get_by_url/raw_data/neutral/urls_neutral.txt', ‘urls_porn.txt': ‘./picture_get_by_url/raw_data/porn/urls_porn.txt', ‘urls_sexy.txt': ‘./picture_get_by_url/raw_data/sexy/urls_sexy.txt'}
reading file: urls_drawings.txt
Try downloading file: http://41.media.tumblr.com/xxxxxx.jpg
Try downloading file: http://41.media.tumblr.com/xxxxxx.jpg
Try downloading file: http://ak1.polyvoreimg.com/cgi/img-thing/size/l/tid/xxxxxx.jpg
Error occurred when downloading file, error message:
HTTP Error 502: No data received from server or forwarder
Try downloading file: http://akicocotte.weblike.jp/gaugau/xxxxxx.jpg
Try downloading file: http://animewriter.files.wordpress.com/2009/01/nagisa-xxxxxx-xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
Try downloading file: http://cdn.awwni.me/xxxxxx.jpg
關于利用Python如何實現根據URL地址下載并保存文件至對應目錄就分享到這里了，希望以上內容可以對大家有一定的幫助，可以學到更多知識。如果覺得文章不錯，可以把它分享出去讓更多的人看到。
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
