science

Investigative Data Warehouse

Investigative Data Warehouse is a science topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Investigative Data Warehouse rather than just read about it. In short: Investigative Data Warehouse (IDW) is a searchable database operated by the FBI. It was created in 2004.

Key takeaways

Investigative Data Warehouse belongs to science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Investigative Data Warehouse to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Investigative Data Warehouse from memory before moving on to harder problems.

Reference excerpt

Investigative Data Warehouse (IDW) is a searchable database operated by the FBI. It was created in 2004. Much of the nature and scope of the database is classified. The database is a centralization of multiple federal and state databases, including criminal records from various law enforcement agencies, the U.S. Department of the Treasury's Financial Crimes Enforcement Network (FinCEN), and public records databases. According to Michael Morehart's testimony before the House Committee on Financial Services in 2006, the "IDW is a centralized, web-enabled, closed system repository for intelligence and investigative data. This system, maintained by the FBI, allows appropriately trained and authorized personnel throughout the country to query for information of relevance to investigative and intelligence matters."

Overview In 2004, according to a government solicitation for bids to manage the project, it was approximately 10TB in size. In 2005, according to one FBI official, the IDW contained approximately 100 million documents. In 2006 it contained more than 560 million documents and was accessible by more than 12,000 individuals. According to the FBI's website, as of August 22, 2007, the database contained 700 million records from 53 databases and was accessible by 13,000 individuals around the world. As of 2007, the FBI was the subject of a lawsuit brought by the EFF (Electronic Frontier Foundation) because of a lack of public notice describing the database and the criteria for including personal information, as required by the Privacy Act of 1974. The lawsuits were a result of two Freedom of Information Act requests filed by the EFF in 2006. It was built in part by Chiliad corporation, the FBI Office of the Chief Technology Officer, and others. Companies listed on the FOIA files include Northrop Grumman .

Purpose Investigative Data Warehouse–Secret (IDW-S) "provides data and data processing/analysis services to FBI agents and analysts as they perform counter-terrorism, counter-intelligence, and law enforcement missions". The core subsystem supports the Counter-Terrorism Division (CTD), the Special Event Unit, and via DOCLAB-S, the Joint Intelligence Committee Investigation (JICI) and IntelPlus. According to a 2005 email, "IDW will also be used for criminal and other authorized non-CT investigations as it evolves." (CT being counter terrorism)

Subsystems Within the system, there were subsystems named IDW-S Core, SPT, and DOCLAB-S The special projects team (SPT):

allows for the rapid import of new specialized data sources. These data sources are not made available to the general IDW users but instead are provided to a small group of users who have a demonstrated "need-to-know". The SPT System is similar in function to the IDW-S system, with the main difference is a different set of data sources. The SPT System allows its users to access not only the standard IDW Data Store but the specialized SPT Data Store.

Privacy According to internal emails, the FBI performed several Privacy Impact Assessments (PIAs) of the IDW system. They worked with lawyers from their National Security Law Branch (NSLB) to attempt to make sure their system was complying with various laws regarding sharing of information and secrecy (for example, rule 6e of the Federal Rules of Criminal Procedure, regarding the secrecy of Grand Jury material ). The Information Sharing Policy Group (ISPG) formed a Discretionary Access Control Team (DACT), to work on "approval of data sets" and "access control requirements" for IDW and DataMart, and responding to other Intelligence Community agencies requesting access. The EFF FOIA IDW website states "Despite the vast amount of personal information contained in the IDW, the FBI has never published a Privacy Act notice describing the system or explaining the ways in which the records might be used." There was also a 2005 email from someone on the Office of General Council (OGC) about "preliminary staff musings that maybe we should limit FBI PIA requirements to non-NS systems" (NS being National Security). There was also an email from 2006 saying that 'national security systems are exempt from E-Gov', apparently referring to the E-Government Act of 2002, which has a section that deals with privacy.

Data sources The IDW used many data sources. The FOIA documents from EFF are heavily redacted, but some of the sources are as follows:

FBI Automated Case Support system (ACS), subset of the Electronic Case File (ECF) system Joint Intelligence Committee Investigation documents (JICI), with OCR text "Open Source News" (public websites, such as the Washington Post and others) Secure Automated Messaging Network (SAMNet) Violent Gang and Terrorist Organizing File (VGTOF) DARPA TIDES program ('open source news' that has been organized and collected) IntelPlus Filerooms, with OCR text FBI National Crime Information Center (NCIC) FBI Records Management Division (RMD), Document Laboratory (DocLab), FBIHQ MiTAP (collects data from public sources, websites, etc.) SPT-Specific data sources (partial list, FOIA files have large parts redacted): Unified Name Index (UNI) extracts Financial Center (FinCen), including Bank Secrecy Act data "Various Sources", including the Transportation Security Administration FBI Counterterrorism Division (CTD) Telephone numbers / addresses from ACS Case data from ACS Terrorist Watch List (TWL) "Other NJTTF data" DoS ... Lost/Stolen Passport data No Fly List, from TSA Selectee list, from TSA ACS/ECF with some case types excluded CIA non-TS/non-SCI Technical Discussions (TDs) and Intelligence Information Reports (IIRs) from 1978 to the May 2004 There was also talk of linking the FTTTF "Data Mart" with IDW. The data in IDW is classified at the 'Secret' level or lower. Higher classifications are not allowed, and can be removed

Worked examples

Example 1 — a first encounter with Investigative Data Warehouse

Start with the simplest possible case. Write down what Investigative Data Warehouse claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Investigative Data Warehouse before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Investigative Data Warehouse ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Investigative Data Warehouse

In research: Investigative Data Warehouse appears in science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Investigative Data Warehouse in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Investigative Data Warehouse is common in secondary-school and first-year university syllabi. It links to neighbouring topics Federal Bureau of Investigation, Government databases in the United States, Information science, so understanding it makes those chapters shorter.
In everyday life: Look for Investigative Data Warehouse outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Investigative Data Warehouse in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Investigative Data Warehouse means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Investigative Data Warehouse out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Investigative Data Warehouse in simple terms?

Investigative Data Warehouse (IDW) is a searchable database operated by the FBI. It was created in 2004.

Why does Investigative Data Warehouse matter?

Because it connects several science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Investigative Data Warehouse?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Investigative Data Warehouse.