FIU Libraries: GIS/RS Data Services: Big Data Collection

Data Collection

It is common to define BIG DATA in terms of the "3 V's": Volume, Velocity and Variety.

Big data involves a huge volume of data, such as are now becoming available from sensors, radars, surveys, online publication, web-mapping applications, social media and so on. The community society says that today we are experiencing a data volume in the order of 1018 (peta) bytes. Data is also becoming available much faster. Real-data positions and crowed-sourcing apps are already really popular.

Velocity raises and is a layer of complexity regards to Big Data. Our capacity to produce GIS data is far greater than our capacity to store, examine, analyze, visualize. In the past data production has followed its own timetable. Now a days data are available from satellites, ground-based sensors, and social media in near-real time, offering the potential of almost immediate discoveries. Geographical data science, then, shifts the emphasis towards problem-solving and predictions since the data is available in a short space-temporal frame.

Variety is, perhaps, the most promising and disruptive of the 3 V's. GIS data used to be relied to one authoritative source of data: human on census, physical on remote sensing and Digital elevation models on topography survey. Big data offers a completely new data sources that have never been available before. The smart phones, web 2.0, allowed every single person produce a whole set of data. The problem wih this new data source is quality, or the lack of it.

free best big data source top

One important part of the big data production is the collaboration of people using crowd-sourcing applications in the Web 2.0.

The Social Web - Web 2.0 - is well known by the high interactivity and sociability producing a broad public crowd-sourcing information and resources. People can co-create, co-tag, reply in a different variety of online-mediated ways. Individuals can share a wide range of interests, experiences and resources.

Social media platforms are a crucial aspect of the social web and also for the crowed-sourcing data collection. Emails, instant messaging, short message services, video calls, social networking sites (facebook, LinkedIn), content sharing sites (Flickr, Instagram, Pinterest, and YouTube), blogs, wikis, microblogs (twitter), online encyclopedias, collaborative online work sites.

Imagem relacionada

In many cases, the data shared in this social medias may be public or private. Public data can be accessible thought web/mobile interfaces, API, Web Services (WFS/WMS/WCS), HTML-crawler. Some of the web/mobile application allows download, at least, part of it.

In other cases, the possibility of developing a crowed-sourcing application in order to have a great amount of people helping you collect data of your interesting can be a good strategy. The challenge with crowed-sourcing apps in general ay any level is motivating the user. That's one reason that access and explore the available data from the platforms that are already collecting a massive amount of data sounds a good option.

APIs

Stands for "Application Programming Interface." An API is a set of commands, functions, protocols, and objects that programmers can use to create software or interact with an external system. It provides developers with standard commands for performing common operations so they do not have to write the code from scratch.

Web Services

It is a service offered by an electronic device to another one to communicate with each other via the World Wide Web. There are two basic service sets – the Web Feature Services (WFS) and the Web Map Services (WMS). The WFS is concerned with direct access to your data – reading, writing, and updating your features. The WMS is concerned with transforming your data into a map (image).

The Open Geospatial Consortium (OGC) has defined Web Mapping Service (WMS) specification protocols for the transfer of geospatial data from servers to client applications.

WMS gives clients access to an image of data without storing a local copy of the dataset. Using a specially structured http request, you can call a dataset (eg. orthoimagery, highways) into your GIS software application.

Web/Mobile Mapping Interfaces

A web-mapping application or WebGIS, is web based, interactive map that allows query and display layers. It is a platform that end-user can run it in a browsers or smart phones. It’s powered by web services—standard services that deliver data and capabilities, and connect components. Usually allows to visualize and, in some cases, download, at least, some portion of the data.

Web crawler

A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner.

This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.

Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code.

Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting data.

Scripts

Scripts wrote in R and Python are becoming quite famous on gathering social media data. There are many libraries and packages that facilitates getting the public data from twitter, facebook and instagram.

Copyright is a legal right created by the law of a country that grants the creator of an original work exclusive rights for its use and distribution. All data, table, application available are subjected to the terms of copyright, license agreement and usage.

Whenever you download, use, distribute geospatial data, you may check the applicable terms.

Learn more about:

Open data licensing issues here.

Spatial Data in the Public Domain here

Floor		Resources
One		Center for Academic Success: University Learning/Testing Center Electronic Classroom Broadcast Video Productions Resource for Education Technology UTS - Media Equipment Services Distance Learning Electronic Classroom Instructional Photography & Graphics
Two		Information & Research Services Curriculum & Juvenile Collection LACiC - Latin American and Caribbean Center Government Documents GIS-RS Center & Teaching Lab Maps & Imagery User Statistics
Three		Periodicals & Newspapers Laptop Lounge Honors College Student Resource Center College of Medicine Library Copiers Systems
Four		College of Medicine Special Collections & Archivese Academic Space Management School of Social Work Study Area
Five		Audiovisual Collection Diaz Ayala Collection Architecture and Fine Arts (General Collection and Periodical Section N) Music (General Collection and Periodical Section M) Music Listening Room Music CAI Center Spanish Resource Center Group Study Rooms
Six		Quiet Study Space Group Study Rooms General Collection HJ-L & P-Z Oversized Books
Seven		General Collection A-HG Quiet Study Floor Group Study Rooms
Eight		Library Administration Collection Management Digital Collections Center

Floor		Resources
One		Circulation Course Reserves Intercampus/Interlibrary Loan Information and Research Services Center for Excellence in Writing Communication Arts Studio Computer Labs/Copiers
Two		Microforms Periodicals Juvenile Collection Oversize Collection Video Collection Government Documents Reference Workroom Reference Collection Copiers
Three		General Collection, A-Z Library Administration

GIS/RS Data Services

Data Collection

APIs

Web Services

Web/Mobile Mapping Interfaces

Web crawler

Scripts

Information

Libraries

Contact