Semi-structured data is a form of structured data that is not rigidly structured. The data model associated with semi-structured data is interpreted when the data is read from a data storage medium or accessed in memory. The data model need not conform to a predefined database schema and is not optimized for succinct, low-overhead streaming or persistent storage as is normal with a relational database, graph database, data table, or graph. Instead, semi-structured data includes delimiter characters or tags to separate semantic elements or to describe relationships among records and fields within the data. The data is said to have a self-describing structure. Semi-structured data may be human-generated or machine-generated. It may be part of an unstructured document such as narrative text or may encompass unstructured data such as free-form comments. It may be intended to be read by humans, to be shared between software services or applications whose data models are not fully consistent, or to be transmitted from sensors or other devices to computers. Optical character recognition in combination with intelligent document processing can convert semi-structured data appearing on printed pages into machine-ready formats such as JSON. Electronic data interchange uses standard segments, elements, and delimiters to transfer business transaction data between computers owned by different organizations. Metrics from an array of various sensors can be combined, as heterogeneous data without predefined schemas, in JSON format.
Human-generated sources Human-generated semi-structured data includes text and information organized with tags, markers, or flexible layouts (such as forms or tables) created by people rather than by automated systems. Examples include medical charts, real property records, pay records, and structural metadata within publications.
Medical records Clinical notes containing standardized dropdown selections can be mixed with free-text physician narratives. The Fast Healthcare Interoperability Resources (FHIR) standard is used for exchanging health care information electronically. Patient intake forms featuring labeled fields (e.g., age, symptoms) and handwritten or typed comments can be digitally stored using JSON or FHIR standards. Electronic health record logs can be formatted via standards like HL7 that group data into segments.
Real estate Real estate records such as deed, mortgage, and lien documents can be processed as JSON objects. Scanned or digital property lease documents often include irregular tables, hierarchical headers, and customized clauses that can be treated as semi-structured data. Property listing descriptions represented via listing services include such information as square footage and room count along with free-form descriptive text. Human-annotated zoning and appraisal filings also combine numeric values with descriptive text.
Pay records Employee pay stubs feature fixed field labels (gross pay, deductions) along with descriptive text. Human-entered spreadsheet logs track contractor hours and commission adjustments. Corporate compensation disclosures are managed through tracking schemas like the Open Payments Search Tool associated with the Physician Payments Sunshine Act of 2010.
Publications A book or periodical has structural metadata such as its table of contents, chapters, parts, and an index. These can be represented in JSON. Bibliographical references conform to a structure, typically defined by the MLA Handbook, the APA, or the Chicago Manual of Style, that can be parsed based on sets of delimiters or using wildcards.
Machine-generated sources Machine-generated semi-structured data sources are automated systems and devices that output information organized with tags, keys, or markers rather than rigid tabular schemas. Examples include server log files, IoT sensor metrics, event streams, NoSQL database engines, email systems, mapping, and geographic information systems.
System and server logs Application outputs, error logs, and security audit trails can use text patterns or key-value pairs without a strict schema.
IoT and telemetry streams Many smart devices, industrial sensors, and interconnected hardware components generate periodic metrics packaged in JSON or XML.
Web and API activity Automated webhooks, clickstream tracking data, and web API response payloads are typically packaged in JSON.
Network & monitoring data Packet inspection summaries, router statistics, and application performance management (APM) metrics provide semi-structured data in various formats.
NoSQL database engines A NoSQL database management system, often used in big data and real-time applications, is designed to handle semi-structured data models, including key-value, document, column family, and graph formats
Email headers An email consists of headers and a body. An email header conforms to a standard key-value format that includes a field name followed by a colon that serves as a delimiter.
Mapping and geographic information systems GeoJSON is a standard for encoding geographic data structures, such as points, lines, and polygons, into a JSON-based format used to represent geographic features, such as locations, roads, boundaries, and their names. OpenStreetMap stores semi-structured data using a flexible tag-based data model built on elements such as nodes, ways, and relations.
Formats for machine-generated semi-structured data
… excerpt ends here. Continue reading the full article.
