标签：印象 oat 保存 pre 回滚 rom 设置 解析 gem
恰逢诺兰导演的新片《敦刻尔克》即将在中国上映，作为诺兰导演的铁粉，印象中他的很多部电影都进入了IMDB TOP250的榜单，但是具体是多少部呢？他是不是IMDB TOP250 中作品最多的导演呢？哪些演员在这些电影中出镜最多呢？在这些问题的启发下，我准备写一个简单的爬虫脚本来获取我想要的数据。
首先需要对工作的流程进行一个简单的分析。我们的目标是获取以下的数据：
要得到以上的数据，我们需要的原始数据包括：
让我们先来看一下数据的来源，IMDB TOP250的网页。
可以看到在页面HTML文件中，我们可以得到的数据有电影的评分，电影的名字，电影的年份。但是导演和演员的数据呢？可以发现在页面上点击电影的名字，可以到达电影的详情页，而这个link也在HTML文件中。
我们接着观察电影的详情页。在HTML中我们可以获取到导演的信息
同时在Cast 的表中还可以获取到主要演员的信息
这样一来我们需要的数据就都有了。
要实现这种类型数据的排名和统计，关系型数据库更加合适。在这里，我的设计是用5个不同的表来记录不同的数据。同时我使用的是开源的MySQL数据库。
imdb_movie schematop_250_movies 用于存储电影的信息：电影名称 name， 电影的发行年份 year， 电影的评分 rate.tt0111161的部分，所以我猜测0111161就是这部电影在IMDB中的UUID，所以我决定用这个值作为这个表的id值。CREATE TABLE `top_250_movies` ( `id` int(11) NOT NULL, `name` varchar(45) NOT NULL, `year` int(11) DEFAULT NULL, `rate` float NOT NULL, PRIMARY KEY (`id`) )
actors 和 directors来保存演员和导演的信息。id 和演员的 name. 而演员／导演的ID和前面的电影ID的思路类似，通过演员详情页链接中的ID来设置。CREATE TABLE `actors` ( `id` int(11) NOT NULL, `name` varchar(45) DEFAULT NULL, PRIMARY KEY (`id`) ) REATE TABLE `directors` ( `id` int(11) NOT NULL, `name` varchar(45) NOT NULL, PRIMARY KEY (`id`) )
cast_in_movie来保存演员出演电影的信息。cast_id来标示每一个出演的关系，这个表中的每一行数据记录了一个演员参演了一部电影。同时是分别使用actor_id和movie_id为Foreign Key与actors和top_250_movies关联。CREATE TABLE `cast_in_movie` ( `cast_id` int(11) NOT NULL AUTO_INCREMENT, `actor_id` int(11) NOT NULL, `movie_id` int(11) NOT NULL, PRIMARY KEY (`cast_id`), KEY `actor_id_idx` (`actor_id`), KEY `movie_id_idx` (`movie_id`), CONSTRAINT `actor_id` FOREIGN KEY (`actor_id`) REFERENCES `actors` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION, CONSTRAINT `movie_id` FOREIGN KEY (`movie_id`) REFERENCES `top_250_movies` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION )
direct_movie。CREATE TABLE `direct_movie` ( `id` int(11) NOT NULL AUTO_INCREMENT, `director_id` int(11) NOT NULL, `movie_id` int(11) NOT NULL, PRIMARY KEY (`id`), KEY `director_id_idx` (`director_id`), KEY `movie_id_idx` (`movie_id`), CONSTRAINT `director_id` FOREIGN KEY (`director_id`) REFERENCES `directors` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION )
在理清了工作流程之后，可以开始实现脚本了。
import re import pymysql import requests from bs4 import BeautifulSoup from requests.exceptions import RequestException
代码中主要使用BeautifulSoup对HTML文件进行解析和搜索，获取需要的数据。
另外需要注意的是使用正则表达式来获取电影的ID
id_pattern = re.compile(r‘(?<=tt)\d+(?=/?)‘)
获取的是tt开头 /结尾的字符串，但是不包含tt和/，这部分数字就是我们想要的ID。
这个方法是一个生成器，每次的返回是一部电影的数据。
def get_top250_movies_list(): url = "http://www.imdb.com/chart/top" try: response = requests.get(url) if response.status_code == 200: html = response.text soup = BeautifulSoup(html, ‘lxml‘) movies = soup.select(‘tbody tr‘) for movie in movies: poster = movie.select_one(‘.posterColumn‘) score = poster.select_one(‘span[name="ir"]‘)[‘data-value‘] movie_link = movie.select_one(‘.titleColumn‘).select_one(‘a‘)[‘href‘] year_str = movie.select_one(‘.titleColumn‘).select_one(‘span‘).get_text() year_pattern = re.compile(‘\d{4}‘) year = int(year_pattern.search(year_str).group()) id_pattern = re.compile(r‘(?<=tt)\d+(?=/?)‘) movie_id = int(id_pattern.search(movie_link).group()) movie_name = movie.select_one(‘.titleColumn‘).select_one(‘a‘).string yield { ‘movie_id‘: movie_id, ‘movie_name‘: movie_name, ‘year‘: year, ‘movie_link‘: movie_link, ‘movie_rate‘: float(score) } else: print("Error when request URL") except RequestException: print("Request Failed") return None
db =pymysql.connect("localhost","testuser01","111111","imdb_movie" ) cursor = db.cursor()
def store_movie_data_to_db(movie_data): print(movie_data) sel_sql = "SELECT * FROM top_250_movies WHERE id = %d" % (movie_data[‘movie_id‘]) try: cursor.execute(sel_sql) result = cursor.fetchall() except: print("Failed to fetch data") if result.__len__() == 0: sql = "INSERT INTO top_250_movies (id, name, year, rate) VALUES (‘%d‘, ‘%s‘, ‘%d‘, ‘%f‘)" % (movie_data[‘movie_id‘], movie_data[‘movie_name‘], movie_data[‘year‘], movie_data[‘movie_rate‘]) try: cursor.execute(sql) db.commit() print("movie data ADDED to DB table top_250_movies!") except: # 发生错误时回滚 db.rollback() else: print("This movie ALREADY EXISTED!!!")
接着利用上面的得到的movie_data 来获取电影详情页的信息。包括导演信息和演员信息。
def get_movie_detail_data(movie_data): url = "http://www.imdb.com" + movie_data[‘movie_link‘] try: response = requests.get(url) if response.status_code == 200: soup = BeautifulSoup(response.text, ‘lxml‘) # Parse Director‘s info director = soup.select_one(‘span[itemprop="director"]‘) person_link = director.select_one(‘a‘)[‘href‘] director_name = director.select_one(‘span[itemprop="name"]‘) id_pattern = re.compile(r‘(?<=nm)\d+(?=/?)‘) person_id = int(id_pattern.search(person_link).group()) movie_data[‘director_id‘] = person_id movie_data[‘director_name‘] = director_name.string store_director_data_in_db(movie_data) #parse Cast‘s data cast = soup.select(‘table.cast_list tr[class!="castlist_label"]‘) for actor in get_cast_data(cast): store_actor_data_to_db(actor, movie_data) else: print("GET url of movie Do Not 200 OK!") except RequestException: print("Get Movie URL failed!") return None
获取演员信息的方法：
def get_cast_data(cast): for actor in cast: actor_data = actor.select_one(‘td[itemprop="actor"] a‘) person_link = actor_data[‘href‘] id_pattern = re.compile(r‘(?<=nm)\d+(?=/)‘) person_id = int(id_pattern.search(person_link).group()) actor_name = actor_data.get_text().strip() yield { ‘actor_id‘: person_id, ‘actor_name‘: actor_name }
4. 把导演信息存入数据库
这里需要在两个table中插入数据。首先在directors中插入导演的数据，同样检查记录是否已经存在。接着在 direct_movie插入数据，插入前也检查是否已经存在相同的数据。
def store_director_data_in_db(movie): sel_sql = "SELECT * FROM directors WHERE id = %d" % (movie[‘director_id‘]) try: # 执行sql语句 cursor.execute(sel_sql) # 执行sql语句 result = cursor.fetchall() except: print("Failed to fetch data") if result.__len__() == 0: sql = "INSERT INTO directors (id, name) VALUES (‘%d‘, ‘%s‘)" % (movie[‘director_id‘], movie[‘director_name‘]) try: # 执行sql语句 cursor.execute(sql) # 执行sql语句 db.commit() print("Director data ADDED to DB table directors!", movie[‘director_name‘] ) except: # 发生错误时回滚 db.rollback() else: print("This Director ALREADY EXISTED!!") sel_sql = "SELECT * FROM direct_movie WHERE director_id = %d AND movie_id = %d" % (movie[‘director_id‘], movie[‘movie_id‘]) try: # 执行sql语句 cursor.execute(sel_sql) # 执行sql语句 result = cursor.fetchall() except: print("Failed to fetch data") if result.__len__() == 0: sql = "INSERT INTO direct_movie (director_id, movie_id) VALUES (‘%d‘, ‘%d‘)" % (movie[‘director_id‘], movie[‘movie_id‘]) try: # 执行sql语句 cursor.execute(sql) # 执行sql语句 db.commit() print("Director direct movie data ADD to DB table direct_movie!") except: # 发生错误时回滚 db.rollback() else: print("This Director direct movie ALREADY EXISTED!!!")
这里需要在两个table中插入数据。首先在actors中插入演员的数据，同样检查记录是否已经存在。接着在cast_in_movie插入数据，插入前也检查是否已经存在相同的数据。
def store_actor_data_to_db(actor, movie): sel_sql = "SELECT * FROM actors WHERE id = %d" % (actor[‘actor_id‘]) try: # 执行sql语句 cursor.execute(sel_sql) # 执行sql语句 result = cursor.fetchall() except: print("Failed to fetch data") if result.__len__() == 0: sql = "INSERT INTO actors (id, name) VALUES (‘%d‘, ‘%s‘)" % (actor[‘actor_id‘], actor[‘actor_name‘]) try: # 执行sql语句 cursor.execute(sql) # 执行sql语句 db.commit() print("actor data ADDED to DB table actors!") except: # 发生错误时回滚 db.rollback() else: print("This actor has been saved already") sel_sql = "SELECT * FROM cast_in_movie WHERE actor_id = %d AND movie_id = %d" % (actor[‘actor_id‘], movie[‘movie_id‘]) try: # 执行sql语句 cursor.execute(sel_sql) # 执行sql语句 result = cursor.fetchall() except: print("Failed to fetch data") if result.__len__() == 0: sql = "INSERT INTO cast_in_movie (actor_id, movie_id) VALUES (‘%d‘, ‘%d‘)" % (actor[‘actor_id‘], movie[‘movie_id‘]) try: # 执行sql语句 cursor.execute(sql) # 执行sql语句 db.commit() print("actor casted in movie data ADDED to DB table cast_in_movie!") except: # 发生错误时回滚 db.rollback() else: print("This actor casted in movie data ALREADY EXISTED")
这里需要注意的是在操作完成或者出错的情况下都要关闭数据库连接。
def main(): try: for movie in get_top250_movies_list(): store_movie_data_to_db(movie) get_movie_detail_data(movie) finally: db.close() if __name__ == ‘__main__‘: main()
运行脚本完成数据获取之后，我们通过SQL语句来获取我们最终想要的数据
SELECT dm.director_id, d.name, count(dm.id) as direct_count FROM imdb_movie.direct_movie as dm JOIN imdb_movie.directors as d ON d.id = dm.director_id group by dm.director_id order by direct_count desc
SELECT cm.actor_id, a.name, count(cm.actor_id) as count_of_act FROM imdb_movie.cast_in_movie as cm JOIN imdb_movie.actors as a ON a.id = cm.actor_id group by cm.actor_id order by count_of_act desc
最终的答案是什么呢？各位同学可以自己来揭晓。
标签：印象 oat 保存 pre 回滚 rom 设置 解析 gem
原文地址：http://www.cnblogs.com/batkid/p/7710543.html
