标签：types rust rdbms enc ora miss mis difficult storage
1. veracity (quality)
how correct the data is, shows if we can trust the data
challenging因为易于发生，影响巨大且难以控制
2. variability
variety指same data, different object
如冰淇淋有各种不同的口味
variability指same data, different meaning
如两句相同的话在不同的时间有不同的意义
3. visibility
capture and properly present the characteristics of data
common types: charts, tables, graphs, maps, infographics, dashboards
难度体现在选择最合适的方式体现数据特征，需要结合数据特征以及目的；同时数据视图化本身也是有难度的（对于高维度的要先降维；数据本身没有结构的如区分文中积极与消极的语气，可以标注成不同的颜色；scalability可伸展性，如很多点集中在一起是如何分辨；动态数据）
4. value
value from other V‘s
5. in general
fundamental V‘s: volume, variety, velocity
characteristics/difficulties: veracity, variability
tools: visibility
objective: value
6. big data management is to server the purpose of big data analytics
7. data acquisition
application oriented: 确定什么样子的信息是问题所需要的
comprehensive: 尽可能全面的收集信息
handle data: 处理来源不同种类不同的信息
8. data storage
a) traditional way: 为structured data设计的, disk-oriented，大数据不适用
b) big data era
b.1) RDBMS -- SAP HANA
b.2) NoSQL -- HBase, Hive, MongoDB
b.3) Distributed file systems -- HDFS
9. data preparation
a) data exploration: understand your data
b) data pre-processing
data cleaning -- veracity
data integration -- variety
10. data explore
trends, correlations, outliers, statistics(mean, mode, median, standard deviation, dange: 可用来数据处理，如身高中大部分都是180，175，一个17的数据就可以被认为是dirty data)
11. data cleaning
dirty data types:
miss values/records: remove the record
invalid data; use another data as replacement
inconsistency: do additional works
duplicate: merge
outliers
12. data integration
merge data from multiple, complex and heterogenous resources to perfrom a unified view of data
13. data curation
data curation includes all the processes needed for principled and controlled data creation, maintenance, and management, together with the capacity to add value to data
数据策划包括原则性和受控数据创建，维护和管理所需的所有过程，以及为数据增值的能力
标签：types rust rdbms enc ora miss mis difficult storage
原文地址：https://www.cnblogs.com/eleni/p/13049568.html
