您好,登錄后才能下訂單哦!
本文實例為大家分享了python爬蟲獲取小區經緯度、地址的具體代碼,供大家參考,具體內容如下
通過小區名稱利用百度api可以獲取小區的地址以及經緯度,但是由于api返回的值中的地址形式不同,所以可以首先利用小區名稱進行一輪爬蟲,獲取小區的經緯度,然后再利用經緯度Reverse到小區的結構化的地址。另外小區名稱如果是'...號‘,可以在爬蟲開始之前在'號‘之后加一個'院‘,得到的精確度更高。這次寫到程序更加便于二次利用,只需要給程序傳遞一個dataframe就可以坐等結果了。現在程序已經寫好了,就等接下來在工作中看看效果如何了。
class GetAddressInfo: def __init__(self,df): import pandas assert type(df) == pandas.core.frame.DataFrame and ('city' in df.columns) and ('name' in df.columns),\ 'The dataframe is not vailid' from bs4 import BeautifulSoup from urllib import request import re import pandas as pd import numpy as np import urllib.parse as urp self.__data__ = df def get_address(self): import numpy as np self.__data__['小區經度'] = np.nan self.__data__['小區緯度'] = np.nan self.__data__['小區地址'] = np.nan for i in self.__data__.index: self.__data__.loc[i,'小區緯度'],self.__data__.loc[i,'小區經度'],self.__data__.loc[i,'小區地址'] =\ self.__get_neigbour_address__(self.__data__.loc[i,'name'],\ self.__data__.loc[i,'city']) return self.__data__ def __lat__(self,res): try: return pd.to_numeric(re.findall('"lat":(.*)',res)[0].split(',')[0]) except: return 0 def __lng__(self,res): try: return pd.to_numeric(re.findall('"lng":(.*)',res)[0]) except: return 0 def __address__(self,res): try: return re.findall('"address":"(.*)",',res)[0] except: return 'None' def __get_neigbour_address__(self,name,city): my_ak = ##替換自己的ak qurey = urp.quote(name) tag = urp.quote('住宅區') try: url = 'http://api.map.baidu.com/place/v2/search?query='+qurey+'&tag='+tag+'®ion='+urp.quote(city)+'&output=json&ak='+my_ak req = request.urlopen(url) res = req.read().decode() lat = self.__lat__(res) lng = self.__lng__(res) address = self.__address__(res) return lat,lng,address except: return 0,0,'None' class ReverseGetAddress: def __init__(self,data): assert ('小區緯度' in data.columns) and ('小區經度' in data.columns) and ('name' in data.columns),\ 'The DataFrame is not vailid' from bs4 import BeautifulSoup from urllib import request import re import pandas as pd import numpy as np import urllib.parse as urp self.__data__ = data def __get_address1__(self,url): try: req = request.urlopen(url) res = req.read().decode() address = re.findall('address":"(.*?)"',res)[0] return address except: return 'None1' def __to_string__(self,arr): return str(arr) def __get_address2__(self): my_ak = ##替換自己的Ak base_url1 = 'http://api.map.baidu.com/geocoder/v2/?callback=renderReverse' base_url2 = '&location=' base_url3 = '&pois=0&radius=1&output=json&pois=1&ak=' url = base_url1+base_url2+self.__data__['小區緯度'].apply(self.__to_string__)+','\ +self.__data__['小區經度'].apply(self.__to_string__)+base_url3+my_ak return url def get_address(self): url = self.__get_address2__() self.__data__['小區地址'] = url.apply(self.__get_address1__) return self.__data__
以上就是本文的全部內容,希望對大家的學習有所幫助,也希望大家多多支持億速云。
免責聲明:本站發布的內容(圖片、視頻和文字)以原創、轉載和分享為主,文章觀點不代表本網站立場,如果涉及侵權請聯系站長郵箱:is@yisu.com進行舉報,并提供相關證據,一經查實,將立刻刪除涉嫌侵權內容。