science

Web data integration

Web data integration is a science topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Web data integration rather than just read about it. In short: Web data integration (WDI) is the process of aggregating and managing data from different websites into a single, homogeneous workflow. This process includes data access, transformation, mapping, quality assurance and fusion of data.

Key takeaways

Web data integration belongs to science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Web data integration to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Web data integration from memory before moving on to harder problems.

Reference excerpt

Web data integration (WDI) is the process of aggregating and managing data from different websites into a single, homogeneous workflow. This process includes data access, transformation, mapping, quality assurance and fusion of data. Data that is sourced and structured from websites is referred to as "web data". WDI is an extension and specialization of data integration that views the web as a collection of heterogeneous databases. Data integration techniques in the context of the web, forms the foundation for businesses taking advantage of data available on the ever-increasing number of publicly-accessible websites. Corporate spending on this area amounted to about USD 2.5bn in 2017, and it is expected that by 2020 the market will reach almost USD 7bn.

Sources Web data integration extends and specializes data integration to see the web as a collection of views of databases accessible over the web protocols, including, but not limited to:

Open data catalogs Government data catalogs Web applications and sites UI (web scraping) API The semantic web (SPARQL) HTML embedded structured data HTML data tables Spreadsheets PDFs Online encyclopedias

Data access and transformation WDI has technical challenges different from data integration due to the data access and transformation required for the web data sources being often unstructured or semi-structured data without a standard query mechanism.

Data quality Understanding the quality and veracity of data is even more important in WDI than in data integration, as the data is generally less implicitly trusted and of lower quality than that which is collected from a trusted source. There are attempts to try to automate a trust rating for web data. Data quality in data integration can generally happen after data access and transformation, but in WDI quality may need to be monitored as data is collected, due to both the time and the cost of re-collecting the data.

Applications WDI has application in many fields, including bioinformatics, search engines, price comparison, and forensic search data analysis, business intelligence, ecommerce, healthcare, pharmaceutical and product development. Most price comparison engines and recommendation systems use user generated data to create recommendations for their users. Similarly, healthcare systems use results of competitions conducted on websites like Kaggle to see the accuracy of data and to create user-focused products. In fact, IBM estimates that poor quality WDI is costing companies over $3 trillion in revenue each year.

References

Worked examples

Example 1 — a first encounter with Web data integration

Start with the simplest possible case. Write down what Web data integration claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Web data integration before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Web data integration ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Web data integration

In research: Web data integration appears in science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Web data integration in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Web data integration is common in secondary-school and first-year university syllabi. It links to neighbouring topics Data integration, Data management, so understanding it makes those chapters shorter.
In everyday life: Look for Web data integration outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Ask Teacher Smith questions about this articleOpens your AI tutor with a question about “Web data integration” →

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Web data integration in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Web data integration means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Web data integration out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Web data integration in simple terms?

Why does Web data integration matter?

Because it connects several science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Web data integration?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Web data integration.