MALICIOUS THREAT DETECTION THROUGH TIME SERIES GRAPH ANALYSIS

Malicious threat detection through time-series graph analysis, in which a data analysis device receives a data file including multiple log data entries. The log data entries include parameters associated with a computer network event in a computing network. The data analysis device produces a graphical model of the computing network based on at least one parameter included in the log data. The data analysis device also identifies a parameter associated with a node of the computer network represented by the graphical model, and performs a time-series analysis on the parameter. The data analysis device further determines, based on the time-series analysis on the parameter, at least one of an anomalous event associated with the computing network or a malicious event associated with the computing network. RECEIVE LOG DATA ENTRIES INCLUDING PARAMETERS/FIELDS ASSOCIATED WITH COMPUTER NETWORK EVENTS 302 PRODUCE GRAPHICAL MODEL TO GENERATE AT LEAST ONE TIME-SERIES FEATURE FROM THE LOG DATA ENTRIES ...

INFORMATION TECHNOLOGY TESTING AND TESTING DATA MANAGEMENT

A testing and extract, transform and load (TETL) system is operable to interface with test platforms testing hardware or software components of information technology systems. The TETL system can execute extract, transform and load operations to load test data into a data warehouse and facilitates evaluating the test data across projects, entities and domains. C:U > - )0a n .S U)/ 0_ c0 C)U)- 00 M CU U) C M00L- m C) C:CU) 03 C: 0 LL - o EC I~iL x' D 0 _ _ LI - C) i KY ...

EFFICIENT QUERY PROCESSING USING HISTOGRAMS IN A COLUMNAR DATABASE

TITLE: EFFICIENT QUERY PROCESSING USING HISTOGRAMS IN A COLUMNAR A probabilistic data structure is generated for efficient query processing using a histogram for unsorted data in a column of a columnar database. A bucket range size is determined for multiples buckets of a histogram of a column in a columnar database table. In at least some embodiments, the histogram may be a height-balanced histogram. A probabilistic data structure 0 is generated to indicate for which particular buckets in the histogram there is a data value stored in the data block. When an indication of a query directed to the column for select data is received, the probabilistic data structure for each of the data blocks storing data for the column may be examined to determine particular ones of the data blocks which do not need to be read in order to service the query for the select data.

METHOD AND APPARATUS FOR CHANGE DATA CAPTURE IN A DATABASE SYSTEM

A method of change data captured is disclosed, in which modifications made to on-line transaction processing (OLTP) tables(e.g. inserts, updates, and deletes) are maintained in a database object, referred to as a change table. The change data in the change table is then made available to analysis applications in a controlled manner, such as in accordance with a publication- subscription model.

METADATA EXCHANGE

The present invention is directed to a metadata exchange for importing metadata from one or more external sources of metadata into a metadata model. A metadata extracted from an external source is in a source format specific to the external source. The metadata model uses a model format. The metadata exchange translates the metadata in the source format into one or more model objects in the model format, and then import the model objects into the metadata model.

UPDATE PROPAGATION TO A DATA WAREHOUSES WITH SPECIFIED LATENCY

A method and system thereof for performing real time transformations of dynamically increasing databases is described. A session, identified as a real time session, is initialized. The real time session repeatedly executes a persistent (e.g., continually running) data transport pipeline of the analytic application. The data transport pipeline extracts data from a changing database, transforms the data, and writes the transformed data to storage (e.g., a data warehouse or data mart). The data transport pipeline is executed at the end of each time interval in a plurality of contiguous time intervals occurring during the real time session. The data transport pipeline remains running after it is executed, until the real time session is completed. Accordingly, new data are transformed in a timely manner, and processing resources are not consumed by having to repeatedly re-establish (re- initialize) the data transport pipeline.

DEVICE Of EXPORT OF DATA Of a DATABASE TOWARDS a TERMINALINFORMATIQUE AND PROCESS ASSOCIATES

METROLOGY SYSTEM OF AIR QUALITY

METHOD AND DEVICE FOR EXTRACTING DATA RELEVANT TO EXECUTE PROCESS

Le procédé d'extraction de données pertinentes dans l'exécution d'un processus selon l'invention comprend : - une étape d'identification (E10), à partir d'un modèle numérique du processus, d'une pluralité de variables manipulées par le processus lors de son exécution ; - une étape d'obtention (E20) d'un premier ensemble de données du processus correspondant aux valeurs prises par ladite pluralité de variables lors d'une pluralité d'instances de ce processus ; - une étape d'obtention (E30) d'un second ensemble de données exogènes collectées pendant ladite pluralité d'instances ; - une étape de partitionnement (E50) des instances en utilisant tout ou partie des données du processus du premier ensemble, cette étape conduisant à une pluralité de groupes d'instances; - une étape d'association (E60) à chaque donnée exogène du second ensemble d'un groupe ; et - une étape d'extraction (E70), pour au moins un des groupes, de données exogènes du second ensemble associées au groupe et représentatives ...

METHOD FOR SAVING AND EXPRESSING WEBPAGE IN ELECTRONIC APPARATUS

A method for saving and expressing a webpage in an electronic apparatus, according to various embodiments of the present disclosure, may comprise the following operations: extracting necessary materials from a webpage based on a parameter; saving the extracted materials; classifying items based on the stored materials; setting a function determined based on the items and the saved materials to be performed; generating a page in which the function is set; and displaying a group page including the page. An apparatus for saving and expressing a webpage according to various embodiments of the present disclosure comprises: a display unit including a display module for displaying a webpage and a group page; a memory for storing materials extracted from the webpage; and a processor for controlling a wireless communications unit for downloading the webpage. The processor may include a material extraction module which is set to perform the following operations: extracting necessary materials from ...

글로벌하게 분산된 환경에서 상호의존적인 기업 데이터 일관성을 보존하기 위한 시스템 및 방법

... 글로벌하게 분산된 환경에서 상호의존적인 기업 데이터를 포함하는 기업 데이터의 일관성을 보존하는 시스템 및 방법. 시스템은, 네트워크; 네트워크와 통신하는 사용자 인터페이스; 네트워크와 통신하고 비즈니스 엔티티들과 관련된 복수의 로컬 데이터 기록들을 포함하는 적어도 하나의 로컬 데이터 스토리지; 네트워크와 통신하고 적어도 하나의 로컬 데이터 스토리지로부터 수신된 로컬 데이터 기록들로부터 생성되는 복수의 글로벌 데이터 기록들을 포함하는 글로벌 데이터 스토리지; 네트워크와 통신하는 프로세서 및 메모리를 포함하는 컴퓨터로서, 프로세서는, 업데이트 기록 및 글로벌 데이터 기록의 최신 업데이트 포인트를 결정하고; 업데이트 기록에 대한 최신 날짜 기준이 글로벌 데이터 기록과 상이하면, 컴퓨터는 어느 기록이 우선순위를 가질지를 결정하여, 적어도 하나의 우선순위 기록 엘리먼트를 생성하는 상기 컴퓨터; 어셈블러; 및 최종 업데이트 기록을 사용자에게 배포하는 배포기를 포함한다.

DYNAMIC HIERARCHICAL TAGGING SYSTEM AND METHOD

A dynamical hierarchical tagging system connected to a user site through a remote communications network is disclosed. The system may comprise a master controller, a job management server connected to the master controller, one or more scanners in communication with the job management server, wherein the one or more scanners are configured to scan for one or more user assets located at the user site, resulting in scan results, a scan logic processor connected to the master controller, wherein the scan logic processor is configured to store the scan results in a user database, a tagging logic engine connected to the master controller, wherein the tagging logic engine is configured to tag the scan results stored in the user database, and an indexing logic processor connected to the master controller, wherein the indexing logic processor is configured to search and index the tagged scan results stored in the user database. In this system, the scan logic processor may be configured to normalize ...

ETL BUILDER

A method of building a staging database which examines the staging database metadata and constructs a schedule of operations to extract the data, transform it and load it into the staging database by using normalisation, translation blocks and filtering to create a business view of the data that can be seen using standard query languages. This is applied to products such as Navision in building an OLAP cube for use in business intelligence applications.

SYSTEMS AND METHODS FOR MANAGING DATA

Systems, methods and non-transitory computer readable media may be provided for the management of data related to risk exposure. A system may be provided including one or more utilities configured to receive data from a plurality of data sources; a rules engine for applying one or more logical rules that are triggered by one or more conditions associated with the integrity of the received data; and one or more utilities configured to apply the one or more logical rules to validate received data and automatically request updated data from a subset of the plurality of data sources where the integrity of the received data does not meet a predefined threshold. Other systems may be provided including units configured for the determination of risk exposure, and the identification of a tiered risk exposure analysis. Corresponding methods and non-transitory computer readable media may be provided.

METHOD AND APPARATUS RELATING TO DATA TRANSPORT

A method and apparatus for transporting data for a data warehouse application is described. The data from an operational data store (the source database) is organized in non-overlapping data partitions. Separate execution threads read the data from the operational data store concurrently. This is followed by concurrent transformation of the data in multiple execution threads. Finally, the data is loaded into the target data warehouse concurrently using multiple execution threads. By using multiple execution threads, the data contention is reduced. Thereby the apparatus and method of the present invention achieves increased throughput.

EXTRACTING INCREMENTAL DATA

The present disclosure introduces a method, an apparatus, and a system for extracting incremental data. Primary key information of incremental data is obtained from a backup database. The incremental data is inquired based on the primary key information from a main database that synchronizes with the backup database. The found incremental data is then inserted into a target data warehouse. The present techniques not only save a lot of time and system resources but also improve the efficiency of incremental data extraction.

DATA COMPASS

A system contains at least one load server that receives subscriber files including encrypted and unencrypted data from a vendor server connected to a subscriber network. The load server includes a load server processor that sorts the encrypted files to a decryption route and the unencrypted files to a direct loading route. The decryption route includes a decryption module that decrypts the encrypted files. The data warehouse includes at least one data warehouse processor having a job tasking module that assigns dedicated load jobs to each file. The data warehouse processor includes a staging module that loads data to a staging table. The data warehouse processor loads data from the staging table to a target table and aggregates the data into a report.

Integrated system and method of data warehousing and delivery

A system and method is capable of providing a much more effective enterprise-wide decision support system. It consists of an integrated end-to-end solution including automatic generation of data warehouses or data marts integrated with automatic delivery of information from the data warehouses or data marts to knowledge workers throughout the enterprise using a "publish and subscribe" paradigm for dissemination of the information at any time, to any places, in any formats to any number of knowledge workers. This integration allows information in the data warehouses or data marts to be delivered immediately after every refresh of the data warehouses or data marts thereby allowing maximum utilization of the valuable information in the data warehouses or data marts throughout the enterprise to gain the most optimum decision support.

Big Data Analytical Platform Tools And Methods Of Use

Methods and systems for extraction, transformation, and loading of source data into an integrated extract-transform-load (ETL) data warehouse analytical platform to map source data from at least one data source column as mapped source data to at least one Input Layout (IL) column of a selected IL table of a plurality of IL tables associated with a Data Layout (DL) table of a plurality of DL tables associated with a named standard package or to a custom target table associated with a named custom package to permit a real-time display on a graphical user interface (GUI) of one or more key performance indicators associated with each DL table.

Cloud Logistics

Implementations of complex software solutions are accompanied with an intrinsic conflict between requirements, such as an ever-growing complexity of enterprise software solutions versus a demand for more and more shortened implementation cycles. Various embodiments include at least one of systems methods and software to address such conflicts. One method embodiment includes receiving a request to assemble a virtual appliance from at least one deployment template, the request including characteristic data representative of at least one virtual appliance functional and deployment requirement. In response to the received request, such embodiments may query a template warehouse to identify and deployment templates from a template warehouse to a data storage device to form a virtual appliance. The virtual appliance may then be packaged into a transport data structure and stored. The transport data structure may then be transported by various means to a target virtual machine and deployed therein.

DATA LOADING FROM A REMOTE DATA SOURCE

A method, apparatus and article of manufacture is provided for loading data from a remote data source record by record, in a computer system network connecting a source site and a target site via a database connection communication line. The source site has at least one data source and a Distributed Relational Database Architecture (DRDA) software server having multi-database access to DBMSs. The target site requests data loading from the source site DRDA via a block of Structured Query Language (SQL) statements. Data are transported record by record via the database connection communication line according to the DRDA communication protocol, wherein the target site loads records concurrently with the unloading of records in the source site. The data loading may also be performed in a pipeline manner, loading data records in multiple partitions with a plurality of parallel streams, pointed to by a plurality of data source partition cursors.

METHOD AND APPARATUS FOR CHANGE DATA CAPTURE IN A DATABASE SYSTEM

A method of change data captured is disclosed, in which modifications made to on-line transaction processing (OLTP) tables(e.g. inserts, updates, and deletes) are maintained in a database object, referred to as a change table. The change data in the change table is then made available to analysis applications in a controlled manner, such as in accordance with a publication- subscription model.

SCALABLE ANALYSIS PLATFORM FOR SEMI-STRUCTURED DATA

A data transformation system includes a schema inference module and an ex-port module. The schema inference module is configured to dynamically create a cumulative schema for objects retrieved from a first data source. Each of the retrieved objects includes (i) data and (ii) metadata describing the data. Dynami-cally creating the cumulative schema includes, for each object of the retrieved objects, (i) inferring a schema from the object and (ii) selectively updating the cumulative schema to describe the object according to the inferred schema. The export module is configured to output the data of the retrieved objects to a data destination system according to the cumulative schema.

DEFINING AND MAPPING APPLICATION INTERFACE SEMANTICS

An example implementation includes a system having one or more processors, a definition module, a data store, and a workflow module. The definition module is executable by the one or more processors to determine a first semantic element for a first operation included in a first application interface representing a first service and a second semantic element for a second operation included in a second application interface representing a second service, such as an enterprise service bus. The definition module may further determine a mapping of the first semantic element with the second sematic element. The data store is configured to store the mapping of the first semantic element with the second sematic element by the definition module. The workflow module is executable by the one or more processors to integrate the first application interface with the second service based on the mapping.

DATA STORAGE REAL TIME

Method for processing and displaying real-time social data on map

System and method for transferring non construction information to content

METHOD FOR UPDATING A RECORD IN A DATABASE BY A DATA PROCESSING DEVICE

Procédé de mise à jour d'un enregistrement dans une base de données par un dispositif de traitement de données configuré pour la mise en œuvre d'une application de traitement d'enregistrements, l'enregistrement ayant une structure comportant une pluralité de champs prédéterminée dans la base de données, l'enregistrement étant accédé par l'application via une modélisation selon un langage objet dans lequel la structure d'un enregistrement est modélisé par une classe, chaque champs de l'enregistrement correspond à une méthode de la classe, caractérisé en ce que le procédé comporte les étapes suivantes pour chaque méthode : - Analyse 2030 de la déclaration de la méthode telle que connue au lancement de l'application pour trouver au moins une annotation prédéterminée ; - Si l'annotation prédéterminée est trouvée, alors analyse des paramètres de l'annotation pour produire 2040 un contrôle de saisie ; - Utilisation 2050 du contrôle produit pour la création d'une interface homme machine permettant ...

METHOD FOR RECONCILING INFORMATION RELATING TO A MOBILE OPERATOR AND INFORMATION RELATING TO A THIRD PARTY SERVER BASE AND ADAPTED FOR CARRYING OUT SAID METHOD

COMPUTER SYSTEM FOR AUTOMATICALLY COLLECTING DATA

La présente invention concerne un système informatique pour la collecte automatique de données de prospection et la mise à disposition des équipements informatiques distants de prospecteurs, comprenant un robot de recherche sur la toile pour la capture d'informations non structurées, ainsi qu'un moyen d'importation des données enregistrées dans les bases de données des prospecteurs et de traitement desdites données préalablement à l'enregistrement de données standardisées dans ladite base de données partagée entre une pluralité de prospecteurs et un moteur de règles commandant le robot de recherche d'enrichissement ladite base de données par une recherche automatique sur une pluralité de sources d'informations, en fonction de données relatives à au moins un descripteur numérique d'événements provenant de l'équipement informatique des prospecteurs, ou d'un traitement de données enregistrées dans ladite base de données.

낮은 지연속도 데이터 액세스를 위한 데이터 스트림의 분할

... 데이터 스트림을 분할하여 로그 데이터 프로세싱을 용이화하고 가속화하는 기술이 여기에서 개시된다. 전단 클러스터(front-end cluster)는 실시간으로 많은 양의 로그 데이터를 생성하고, 로그 데이터를 집성 클러스터(aggregating cluter)에 전송한다. 집성 클러스터는 상이한 전단 서버 및 클러스터로부터 유입되는 로그 데이터 스트림을 집성하도록 설계된다. 집성 클러스터는 또한, 로그 데이터를 복수의 데이터 스트림으로 분할하여 데이터 스트림이 병렬로 수신 애플리케이션에 전송되도록 한다. 일실시예에서, 로그 데이터가 분할된 데이터 스트림에 균일하게 분산되는 것을 보장하기 위해 로그 데이터는 랜덤하게 분할된다. 다른 실시예에서, 분할된 데이터 스트림을 수신하는 애플리케이션은 어떻게 로그 데이터를 분할할지를 결정한다.

DECLARATIVE AND UNIFIED DATA TRANSITION

Described are computer-based methods and apparatuses, including computer program products, for declarative and unified data transition. In some embodiments, a computer implemented method includes determining an unified configuration for a knowledge domain. The unified configuration can include one or more predicates for one or more system objects, and/or one or more relationships between the one or more system objects. The method can include generating one or more transformation rules based on the one or more predicates. The one or more transformations can enable transformation of input data. The input data can include information associated with the one or more system objects and/or the one or more relationships between the one or more system objects. The method can include generating one or more reconciliation rules based on the one or more predicates. The one or more reconciliation rules can be associated with a destination database and can enable reconciliation of the transformed data ...

DATA LOADING FROM A REMOTE DATA SOURCE

A method, apparatus and article of manufacture is provided for loading data from a remote data source record by record, in a computer system network connecting a source site and a target site via a database connection communication line. The source site has at least one data source and a Distributed Relational Database Architecture (DRDA) software server having multi-database access to DBMSs. The target site requests data loading from the source site DRDA via a block of Structured Query Language (SQL) statements. Data are transported record by record via the database connection communication line according to the DRDA communication protocol, wherein the target site loads records concurrently with the unloading of records in the source site. The data loading may also be performed in a pipeline manner, loading data records in multiple partitions with a plurality of parallel streams, pointed to by a plurality of data source partition cursors.

METHOD OF AND APPARATUS FOR BROADCASTING DATABASES

A method of broadcasting data using wireless communication in which the data is extracted from a source SQL database and broadcast to a receiving device programmed to reconstruct the data into a SQL database.

SYSTEM AND METHOD FOR DYNAMIC DOCUMENT MATCHING AND MERGING

A system and method for matching and merging documents from disparate data sources into a single data store for a particular entity are provided. The system and method may be particularly useful for a healthcare system to match and merge data from disparate data sources about a healthcare provider.

DATA LOAD PROCESS

A source data file can be loaded into a database by extracting data from the source data file at a computing device, validating the data from the source data file at the computing device, transforming the data from the source data file at the computing device, requesting the data to be transferred to a first database at a first message processor module, delivering the data to the first database, and responding that the data has been transferred to the first database at the first message processor module to acknowledge receipt of the data in the database at the computing device.

Aggregation and Analytics for Application-Specific Optimization Based on Multiple Data Sources

Aggregating and transforming data, and performing analytics thereupon, for application-specific optimization based on multiple data sources. The data is preferably ingressed automatically, and may originate from various public and/or private data sources. Data transformation preferably aligns the data aggregated from the various sources, to thereby allow meaningful referencing. Complex and non-aligned data can therefore be consolidated, such that it is readily digestible by simulation (or other) software. In an embodiment, risk of flooding for a supply chain is computed from the aggregated and transformed data, using data analytics based on physical computation for flood risk assessment, allowing the supply chain to be optimized with regard to threat of flooding and/or actual flooding. In another embodiment, risk of wild fire may be assessed. Other types of risk may also be assessed.

SYSTEMS AND METHODS FOR DETECTING AND PREVENTING CYBER-THREATS

A system () for detecting and preventing cyber-threats is disclosed. The system () can include an online-analytical-processing (OLAP) resource () coupled to a data mining engine (), a reporting resource () and a processor (). The processor () can run instructions stored within an extract-transform-load (ETL) module (). The ETL module () can enable the processor () to extract one or more data tuples various data sources (). The ETL module () can enable the processor to transform the extracted tuple(s). 1100100. A system () for detecting , evaluating and preventing cyber-threats , the system () comprising:{'b': '102', 'at least one online-analytical-processing (OLAP) resource ();'}{'b': 104', '102, 'at least one data mining engine () coupled to the OLAP resource ();'}{'b': 106', '102, 'at least one reporting resource () coupled to the OLAP resource (); and'}{'b': 108', '102', '110', '108', '112', '112', '108, 'claim-text': [{'b': '110', 'extract at least one data tuple from the data source ();'}, 'transform the data tuple, wherein transforming the data tuple comprises normalizing the data tuple according to a predetermined policy and removing sensitive information from the at least one data tuple; and', {'b': '102', 'load the transformed data tuple (TDT) to the OLAP resource (),'}], 'at least one processor () coupled to the OLAP resource () and coupled to at least one data source (), the processor () having at least one extract-transform-load (ETL) module () associated therewith, the ETL module () configured, upon execution by the at least one processor (), to{'b': '102', 'claim-text': store the loaded TDT,', {'b': 104', '110, 'analyze the loaded TDT with reference to data acquired by the data mining engine () and with reference to at least one previously stored TDT according to at least one predetermined algorithm wherein at least one measure of similarity between the loaded TDT and the previously stored TDT is a factor, the predetermined algorithm including a ...

METHOD AND APPARATUS FOR CHANGE DATA CAPTURE IN A DATABASE SYSTEM

METHOD AND APPARATUS RELATING TO DATA TRANSPORT

A method and apparatus for transporting data for a data warehouse application is described. The data from an operational data store (the source database) is organized in non-overlapping data partitions. Separate execution threads read the data from the operational data store concurrently. This is followed by concurrent transformation of the data in multiple execution threads. Finally, the data is loaded into the target data warehouse concurrently using multiple execution threads. By using multiple execution threads, the data contention is reduced. Thereby the apparatus and method of the present invention achieves increased throughput.

DATABASE SYSTEM AND METHOD

A transformation description language (TDL) for specifying how data is to be manipulated in a data warehousing application. The TDL is comprised of a source for storing raw data, one or more transformation objects for processing the raw data according to predefined instructions, and a target for storing the processed data. A mapping is used for directing the data flow between the I/O ports corresponding to the source, the plurality of transformation objects, and the target. The mapping specifies the connectivity between the source, transformation, and target objects as well as the order of these connections. There are a number of different transformations which can be performed to manipulate the data. Some such transformations include: an aggregator transformation, an expression transformation, a filter transformation, a lookup transformation, a query transformation, a sequence transformation, a stored procedure transformation, and an update strategy transformation.

SYSTEMS AND METHODS FOR INTEREST-DRIVEN DATA SHARING IN INTEREST-DRIVEN BUSINESS INTELLIGENCE SYSTEMS

Systems and methods for interest-driven data sharing in interest-driven business intelligence systems in accordance with embodiments of the invention are illustrated. In one embodiment, an interest-driven data sharing server system includes a processor, a memory configured to store an interest-driven data sharing application, raw data storage, metadata storage configured to store data description metadata describing the raw data, and report storage configured to store previously generated reports, wherein the interest-driven data sharing application configures the processor to receive a report specification, locate at least one previously generated report stored in the report storage, identify raw data stored in the raw data storage using the data description metadata and at least one of the reporting data requirements, calculate redundant data metadata using the located previously generated reports and the identified raw data, determine modified reporting data requirements, retrieve updated ...

用于分析制造数据的方法和装置

... 一种在集成电路制造工厂(“fab”)中获得数据挖掘信息的方法，该方法包括下列步骤：(a)从一个或多个系统、工具、和数据库中收集来自工厂的数据，所述系统、工具、和数据库在工厂中产生数据或者从工厂中收集数据；(b)将数据格式化并将格式化的数据存储在源数据库中；(c)按照用户规定的配置文件提取使用在数据挖掘中的数据部分；(d)响应于用户规定的分析配置文件，对被提取的部分数据进行数据挖掘；(e)将数据挖掘的结果存储在结果数据库中；和(f)提供对该结果的存取。 ...

PROCESS FOR A CATEGORIZATION AND A SEPARATION OF DOCUMENTS IN A STAGE

L'invention concerne un procédé, un appareil et un document papier. Le procédé prévoit la séparation et le classement en catégorie de documents, et il inclut la réception d'un lot numérisé de documents (16). Le lot inclut une pluralité de documents (16) numérisés sur lesquels des tampons séparateurs (12) de documents ont été appliqués avant la numérisation. Chaque tampon séparateur (12) de documents inclut des premier et second motifs pouvant être reconnu par une machine, appliqués sur une même page de document, les premier et second motifs étant espacés par un champ désigné afin de recevoir un code de catégorie appliqué par l'utilisateur. Le lot numérisé de documents est traité afin d'identifier les pages qui contiennent un séparateur de documents, le traitement incluant l'identification d'au moins l'un des premier et second motifs espacés, Pour chacune d'une pluralité de pages de documents pour laquelle un séparateur de documents respectif est identifié, le procédé inclut la localisation ...

Data management device for personal digital assistant, transfers selected data to information management module, so that data is stored in categories corresponding to reference categories

Un dispositif de gestion de données comprend un module de chargement (7) pour accéder à des données d'une base de données de référence, un module de sélection (8) pour permettre à un utilisateur effectuer une sélectionner parmi ces données en fonction de leur appartenance à des classes de données de référence correspondant aux classes de données d'un module de gestion d'informations (9) d'un terminal informatique principal (1) comportant un écran (12) et un module de gestion d'informations (9), et un module d'exportation (10) pour exporter les données sélectionnées vers le module de gestion d'informations (9) du terminal principal (1), de sorte qu'elles soient mémorisées avec les données des classes qui correspondent à leurs classes respectives.

SYSTEM AND METHOD FOR GENERATING/UPDATING EXTERNAL METADATA FOR DATA FLOW

PURPOSE: A system and a method for processing external metadata are provided to improve an operation of data flow in case that external data files are changed, as validation for consistency of the external data files operated with the data flow is offered. CONSTITUTION: A data flow model is stored(300). The data flow model includes use data representing use of at least one row from each external data file. The metadata including the data representing at least one row of at least one external file is stored(310). The stored metadata is used to determine whether the data flow model is agreed with the external data files(320). © KIPO 2006 ...

APPARATUS AND METHOD FOR CREATING PORTABLE ETL JOBS

A computer readable medium with executable instructions to receive a job and correlate a data store with each data source associated with the job. A first configuration profile is associated with the data store. A second configuration profile is specified for the data store. Dependent flows are identified. The dependent flow is updated to include additional configuration information derived from the second configuration profile.

APPARATUS AND METHOD

A transformation description language (TDL) for specifying how data is to be manipulated in a data warehousing application. The TDL is comprised of a source for storing raw data, one or more transformation objects for processing the raw data according to predefined instructions, and a target for storing the processed data. A mapping is used for directing the data flow between the I/O ports corresponding to the source, the plurality of transformation objects, and the target. The mapping specifies the connectivity between the source, transformation, and target objects as well as the order of these connections. There are a number of different transformations which can be performed to manipulate the data. Some such transformations include: an aggregator transformation, an expression transformation, a filter transformation, a lookup transformation, a query transformation, a sequence transformation, a stored procedure transformation, and an update strategy transformation.

DATA PROCESSING SYSTEM AND METHOD FOR APPLICATION PROGRAMS IN A DATA WAREHOUSE

The present invention relates to a data processing method comprising the steps of: providing a set of database tables in a data warehouse, each one of the database being assigned to an entity type and storing entities of its entity type, providing at least one application program for processing of classes of database tables, assigning at least one of the database tables to each one of the classes of database tables of the application program, processing of entities stored in the assigned database tables by the application program.

Real time data warehousing

A method and system for processing data into and in a database and for retrieving the processed data is disclosed. The data comprises identifiers of a plurality of entities. The method and system comprises: (a) processing data into and in a database, (b) enhancing received data prior to storage in a database, (c) determining and matching records based upon relationships between the records in the received data and existing data without any loss of data, (d) enabling alerts based upon user-defined alert rules and relationships, (e) automatically stopping additional matches and separating previously matched records when identifiers used to match records are later determined to be common across entities and not generally distinctive of an entity, (f) receiving data queries for retrieving the processed data stored in the database, (g) utilizing the same algorithm to process the queries and (h) transferring the processed data to another database that uses the same algorithm.

Sampling Events for Rule Creation with Process Selection

Embodiments are directed towards generating a representative sampling as a subset from a larger dataset that includes unstructured data. A graphical user interface enables a user to provide various data selection parameters, including specifying a data source and one or more subset types desired, including one or more of latest records, earliest records, diverse records, outlier records, and/or random records. Diverse and/or outlier subset types may be obtained by generating clusters from an initial selection of records obtained from the larger dataset. An iteration analysis is performed to determine whether a sufficient number of clusters and/or cluster types have been generated that exceed at least one threshold and when not exceeded, additional clustering is performed on additional records. From the resultant clusters, and/or other subtype results, a subset of records is obtained as the representative sampling subset.

AUTOMATIC IDENTIFICATION AND TRACKING OF LOG ENTRY SCHEMAS CHANGES

A log analysis unit compares log entries describing an event to one or more schemas associated with the event. Each of the schemas describes a different log entry structure. When a log entry is determine to have a structure that does not match any of the structures defined by any of the schemas associated with a particular event, a new schema describing the structure of the log entry is generated. In response to the generation of the new schema, one or more entities are notified. Additionally, instructions for processing log entries adhering to the new schema are generated. A cumulative schema and an intersection schema corresponding to the event are also generated.

INTEGRATING OBJECT-BASED DATA INTEGRATION TOOL WITH A VERSION CONTROL SYSTEM IN CENTRALIZED AND DECENTRALIZED ENVIRONMENTS

The present disclosure relates generally to a data integration system that integrate an object based data integration tool, such a GUI-based data integration tools, with version control systems using a relational database repository for persistence. Examples of distributed version control systems include Git, Mercurial, and Bazaar, and examples of centralized version control systems include Subversion, CVS etc. in centralized or distributed environments.

VISUALIZATION SUGGESTION APPLICATION PROGRAMMING INTERFACE

A dataset and some user selected columns of the dataset are received by a statistical analysis module for analysis. The statistical analysis module generates a score for each unselected column of the dataset based on statistical analysis of the unselected columns and all or a subset of the selected columns. A ranking of the unselected columns is presented to the user for selection of one additional column of the dataset, after which the remaining unselected columns are re-ranked according to their associated scores and once again displayed to the user. The user may continue selecting from among the ranked columns until a threshold number of columns has been selected, at which point the user may deselect a selected column in order to continue selecting additional columns. A visualization suggestion application program interface then matches the selected columns with compatible visualization configurations and presents some of these visualizations to the user.

Managing Activities Over Time in an Activity Graph

Systems and processes for managing a data warehouse using an activity graph are disclosed. One example process may include selectively tracking new versions of an entity based on a tracking type, removing entities based on data retention rules, removing entities based on data decay rules, and summarizing decayed data.

PROCESSING DATASETS OF VARYING SCHEMAS FROM TENANTS

An online database system receives a tenant dataset from a tenant and automatically generates a mapping that can be used to convert the tenant dataset from a tenant schema to a database schema used by the online database system. To generate the mapping, the online database system analyzes individual entries of the tenant dataset received from the tenant and determines a likelihood of usage. The mapping maintains entries in the tenant dataset that are likely to be used as individual entries in the converted dataset and compresses entries that are less likely to be used into fewer entries to satisfy the database schema. The online database system converts the tenant dataset from the tenant schema to the database schema using the mapping.

Efficiently Building Nutrition Intake History from Images of Receipts

This invention provides an efficient and feasible method, system and computer program for retrieving total nutrition facts from purchase transaction information including receipt images and other complementary data. The said facts are used to build up the nutrition intake history, provide nutrition intake reports and customized nutrition suggestions based on the users' personal health related information and nutrition intake data. The method initiates from receiving information on a transaction in the format of an image of a receipt, or other itemized input. If the input is a receipt image, an automatic process including image processing, machine learning and text extraction is applied to retrieve the purchased items and quantity, from which total nutrition facts are derived using the nutrition information of each purchased item found in public, or in private and undisclosed vendor and distributor databases that are reconstructed in a preferred embodiment. Other inputs, such as manual food ...

PROCEDURE AND EQUIPMENT FOR THE STORAGE OF TRANSACTION DATAS IN A DATA BASE

Apparatus and method for integrated management of data in mobile device, and mobile device

Method and system for checking data

The invention provides a method and a system for checking data. In the method and the system, a report multi-dimensional data set is generated from a data cube according to a data cube inquiry condition, a reference multi-dimensional data set is generated from source data according to a source data inquiry condition corresponding to the data cube inquiry condition, an on-line analytical processing (OLAP) multi-dimensional data set is generated by performing OLAP inverse transformation on the report multi-dimensional data set, and an extraction transformation and loading (ETL) multi-dimensional data set is generated by performing ETL inverse transformation on the OLAP multi-dimensional data set at the same time, and data inconsistency can be determined by data checking according to a matching relation among the generated multi-dimensional data sets. By the method and the system, data displayed to a user can be checked easily; and if problems exist in the data, the problems in a commercial ...

Device for managing pull flow information within companies or organizations, has database containing information managed by device, and data-processing interface for transporting requests and intelligence responses of information

Dispositif de gestion de l'information en flux tiré. L'invention concerne un dispositif permettant de collecter, stocker, classifier et communiquer efficacement de l'information, fonctionnant en flux tiré par les besoins en information issus de l'intention de créer ou mettre à jour une information. L'environnement de ce dispositif est une organisation de personnes reliées entre elles par un réseau de communication informatique. Cette invention permet de mettre à disposition exactement l'information dont les membres de l'organisation ont besoin, de réduire la quantité d'information présente dans le système à sa seule partie utile, et d'assurer la traçabilité des informations présentes. Le dispositif repose sur l'élément unitaire d'information que constitue un groupe structuré de données représentant un objet sémantique particulier. Le dispositif objet de l'invention est constitué d'une interface informatique permettant de spécifier et suivre l'ensemble des informations qui alimentent la ...

Big data extracting system and method

낮은 지연속도 데이터 액세스를 위한 데이터 스트림의 분할

... 데이터 스트림을 분할하여 로그 데이터 프로세싱을 용이화하고 가속화하는 기술이 여기에서 개시된다. 전단 클러스터(front-end cluster)는 실시간으로 많은 양의 로그 데이터를 생성하고, 로그 데이터를 집성 클러스터(aggregating cluter)에 전송한다. 집성 클러스터는 상이한 전단 서버 및 클러스터로부터 유입되는 로그 데이터 스트림을 집성하도록 설계된다. 집성 클러스터는 또한, 로그 데이터를 복수의 데이터 스트림으로 분할하여 데이터 스트림이 병렬로 수신 애플리케이션에 전송되도록 한다. 일실시예에서, 로그 데이터가 분할된 데이터 스트림에 균일하게 분산되는 것을 보장하기 위해 로그 데이터는 랜덤하게 분할된다. 다른 실시예에서, 분할된 데이터 스트림을 수신하는 애플리케이션은 어떻게 로그 데이터를 분할할지를 결정한다.

ASSOCIATED INFORMATION PROPAGATION SYSTEM

An associated information system is provided that propagates visibility of associated information. The associated information system traces a data lineage of a data warehouse. The associated information system further identifies an association between a primary data element and an associated object, where the association between the primary data element and the associated object is identified as a primary link. The associated information system further stores a first associated object mapping system record within an associated object mapping system table that represents the primary link. The associated information system further identifies an association between a secondary data element and the associated object using the data lineage, where the association between the secondary data element and the associated object is identified as a secondary link, and where the secondary data element is related to the primary data element within the data lineage. The associated information system further ...

Automatically Optimizing Resource Usage on a Target Database Management System to Increase Workload Performance

Loading a set of data into a target database management system (DBMS) from a source DBMS to increase workload performance and decrease resource usage is provided. Using a size and a change rate of tables eligible to be offloaded to the target DBMS, a total central processor unit (CPU) cost corresponding to loading the set of data into to the target DBMS and workload cost savings corresponding to executing one or more workloads on the target DBMS is determined based on a set of one or more load strategies. The total CPU cost is compared with the workload cost savings of each load strategy. An optimal load strategy is selected based on comparing the total CPU cost with the workload cost savings of each load strategy. The set of data is automatically loaded into to the target DBMS from the source DBMS in accordance with the selected optimal load strategy.

CONTENT TRANSFORMATIONS USING A TRANSFORMATION NODE CLUSTER

Content transformations can include transformation of content items in a CMS repository from a source format to a target format. Such transformations can be performed using a transformation node cluster having multiple nodes, each of which is configured for a specific content transformation type. Router nodes can receive requests for content items and route content items to transformation nodes having a proper content transformation type to either transform a requested content item to the target format or perform an intermediate transformation as part of a transformation chain. A transformation node cluster can be dynamically configurable based on estimates of expected loads for the various types of transformations. Systems, methods, and articles of manufacture are also described.

AUTOMATING CONFIGURATION OF OPERATIONAL DATA PIPELINES FOR EXTRACTION, TRANSFORMATION AND LOAD

A method, a processing device and a computer program product are provided. Based on parameters pertaining to a data source, a first processing device may determine whether a data source is a known data source. If the data source is determined to be the known data source, then the first processing device sends respective configuration information to multiple processing devices. The respective configuration information being based on configuration information from a previous integration and configuring the multiple processing devices as an operational data pipeline to extract data from the known data source, transform the extracted data to a first data model employed by a data system, and load the transformed data into the data system.

Plural architecture master data management with supplemental attributes

According to one embodiment of the present invention, a system for managing data within a plurality of data management architectures includes at least one processor. The system persists an entity managed by a first data management architecture to a second data management architecture. The first data management architecture manages entity data within data sources and the second data management architecture manages persisted entities within a common repository. Entity attributes are mapped between the first and second data management architectures. The system further provides one or more supplemental attributes for the persisted (e.g., registration mode or fully persisted mode) entity within the second data management architecture, wherein the supplemental attributes are unmapped between the first and second data management architectures. Embodiments of the present invention further include a method and computer program product for managing data within a plurality of data management architectures ...

METHODS AND SYSTEMS FOR LOADING DATA INTO A TEMPORAL DATA WAREHOUSE

System and method for mathematics ontology extraction and research

An extensive computer based online math research system (the "Research System") having as its foundation an Ontology of mathematics, and utilizing unique and intensive computer support, coordination, data structuring, data storage, computer processing, retrieval capabilities, and data-mining capabilities, and an Ontology editing system that runs on computer software with computer processors and data storage capabilities (the Ontology Editor System"). The Research System also includes a methodology to enable online reference and data manipulation of the Ontology, and an Internet based search of the concepts of mathematics and applications of mathematics to the sciences on the basis of the Ontology.

REAL TIME DATA WAREHOUSING

A method and system for processing data into and in a database (16) and for retrieving the processed data is disclosed. The data comprises identifiers of a plurality of entities (18) .The method and system comprises: (a) processing data into and in a database (16) ,(b) enhancing received data (20) prior to storage in a database (16) ,(c) determining and matching records based upon relationships between the records in the received data (20) existing data without and loss of data, (d) enabling alerts based upon user-defined alert riles and relationships, (e) automatically stopping additional matches and separating previously matched when identifiers used to match records are later determined to be common across entities and not generally distinctive of an entity, (f) receiving data queries (46) for retrieving the processed data stored in the database (16), (g) utiliziing the same algorithm to process the queries (46) and (h) transferring the processed data to another database that uses the ...

SYSTEMS AND METHODS FOR IMPROVING DATABASE PERFORMANCE

Systems and methods for mapping and propagating a source dataset to a plurality of target tables in a target database are described herein. Embodiments include a physical database design with a template-based loader, and a method to propagate changes in the source dataset to a target database. Other embodiments include a database physical design with a plurality of small fact and summary tables. Still other embodiments include a method of extracting and loading data automatically into a target database, while simultaneously maintaining current summary tables.

Data migration framework

Extensible surface for consuming information extraction services

The invention relates to an extensible surface for consuming information extraction services. Representing structured data extracted from unstructured data in fashion allowing querying using relational database concepts. A method includes receiving user input specifying one or more database views. The method further includes receiving user input specifying an information extraction technique, such as an extraction workflow. The method further includes receiving user input specifying a corpus of data. The extraction technique is applied to the corpus of data to produce the one or more database views. These views can then be queried or operated on using database tools.

The mobile device analysis engine

For centralized issue-tracking method and system

Scalable analysis platform for semi-structured data

REAL TIME DATA WAREHOUSING

SYSTEM AND METHOD FOR MULTI-SERVICE FORMAT FILE PROCESSING

A system and method for multi-service format file processing are provided, wherein the system comprises a configuration reading module, a list scanning module, one or more list analyzing modules and one or more warehousing processing modules, wherein the configuration reading module reads configuration information from pre-determined configuration files; the list scanning module scans files in the original call records directory and distributes part of the files or all the files to one or more list analyzing modules according to the first configuration rule; one or more list analyzing modules analyze the files from the list scanning modules into records in database table format according to the second configuration rule; the warehousing processing module downloads the records and saves downloaded records in database. The code maintenance cost and the code complexity can be reduced based on the technical project provided by the invention, and the stability of a program and the definition ...

SAFELY TRANSPORTING LEGACY DATA TO AN OBJECT SEMANTIC FORM DATA GRID

A computer system architecture and process transforms the organizing principles of business data management seen in traditional or common place settings across disparate data stores into an object-semantic model. The transformation is implemented in a transitory, managed risk implementation approach for data access migration. This approach will enable data agility and will move data storage closer to information storage, while supporting the rapid development of business capabilities. This approach will bridge the gaps between relational database/file based storage (less expressive) and object-semantic representation (more expressive). This process/method will connect information residing in legacy data structures such as relational database/file based storage on disk drives to instances populating an expressive object-semantic model in persistent grid based memory. It will also maintain a linkage between the original legacy data to the enhanced object-semantic data allowing for ongoing ...

DELTA-MECHANISM FOR INTEGRATION OF OLAP-BASED PLANNING AND REPORTING

A database system may include a database, a set of tools for accessing the database, a data buffer for storing data objects from the database, and a “delta buffer” for storing delta records. Simulations may be performed in the database system by the set of tools that access the database. Cumulative changes of the data may be handled by the delta buffer. The delta buffer need not contain all the data used for simulations and may include delta records. Each delta record may include cumulative changes of the data and a request identifier. The content of the delta buffer may be integrated with the unchanged, original data from the database. The delta buffer in combination with the data buffer may provide an integrated view for tools that access the database, such as planning and reporting tools.

Method and system for validating data

A method and system for validating data. Warehouse data is generated by transforming source data via an ETL transformation model. A data cube is generated by transforming the warehouse data via an OLAP transformation model. A report dataset (MDS1) is generated from the data cube. A reference dataset (S) is generated from the source data. A data validation is performed, the data validation being based on a matching relationship between MDS1 and S.

SHIPPING OF DATA THROUGH ETL STAGES

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

EVENT STREAM TRANSFORMATIONS

The formulation of transformations on one or more input event streams to generation one or more output event streams. Accordingly, the transformations may be considered to be a query on the original input event stream(s). The event query includes event stream source representations representing an input event stream available in a particular execution context. The event query also includes a transformation module identifying the transformation set to be performed on the input event streams in the execution context. Once the query is properly formed, an execution module may then cause the transformations to be executed upon the designated input event stream(s) to generate output event streams.

APPARATUS AND METHOD FOR FORMING A HOMOGENOUS TRANSACTION DATA STORE FROM HETEROGENEOUS SOURCES

A computer readable storage medium includes executable instructions to gather first committed transactional record from a first transaction log associated with a first data source. The first committed transactional record is converted to a homogeneous format to form a first homogeneous transactional record with a common transaction record header containing transactional context information from the first data source. A second committed transactional record is collected from a second transaction log associated with a second data source with a format different than the first data source. The second committed transactional record is changed to the homogeneous format to form a second homogeneous transactional record with a common transaction record header containing transactional context information from the second data source. The first homogeneous transactional record is combined with the second homogeneous transactional record to form a homogeneous transaction data store.

Processing data from multiple sources

In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Computer systems and methods for sharing asset-related information between data platforms over a network

Disclosed herein are systems, devices, and methods for sharing asset-related information between data platforms that are communicatively coupled via a network. According to an example, a first platform may receive asset-related data and determine that a portion of the received data should be pushed to another platform. Based on that determination, the first platform may prepare a portion of the received data to be transmitted to another platform and then push the portion of the data to another platform over a network connection. In addition, the first platform may be governed by a second platform (e.g., a master or seed platform). According to an example, the first platform may receive from the second platform, and then apply, criteria that governs whether the first platform is permitted to share asset-related data with one or more other platforms in the network.

METADATA EXCHANGE

The present invention is directed to a metadata exchange for importing metadata from one or more external sources of metadata into a metadata model. A metadata extracted from an external source is in a source format specific to the external source. The metadata model uses a model format. The metadata exchange translates the metadata in the source format into one or more model objects in the model format, and then import the model objects into the metadata model.

UPDATE PROPAGATION TO A DATA WAREHOUSE WITH SPECIFIED LATENCY

A method and system thereof for performing real time transformations of dynamically increasing databases is described. A session, identified as a real time session, is initialized. The real time session repeatedly executes a persistent (e.g., continually running) data transport pipeline of the analytic application. The data transport pipeline extracts data from a changing database, transforms the data, and writes the transformed data to storage (e.g., a data warehouse or data mart). The data transport pipeline is executed at the end of each time interval in a plurality of contiguous time intervals occurring during the real time session. The data transport pipeline remains running after it is executed, until the real time session is completed. Accordingly, new data are transformed in a timely manner, and processing resources are not consumed by having to repeatedly re-establish (re- initialize) the data transport pipeline.

DATA LOADING FROM A REMOTE DATA SOURCE

A method, apparatus and article of manufacture is provided for loading data from a remote data source record by record, in a computer system network connecting a source site and a target site via a database connection communication line. The source site has at least one data source and a Distributed Relational Database Architecture (DRDA) software server having multi-database access to DBMSs. The target site requests data loading from the source site DRDA via a block of Structured Query Language (SQL) statements. Data are transported record by record via the database connection communication line according to the DRDA communication protocol, wherein the target site loads records concurrently with the unloading of records in the source site. The data loading may also be performed in a pipeline manner, loading data records in multiple partitions with a plurality of parallel streams, pointed to by a plurality of data source partition cursors.

UPDATE PROCESS FOR SAVING TO A DATA BASE BY A DATA TREATMENT DEVICE

Procédé de mise à jour d'un enregistrement dans une base de données par un dispositif de traitement de données configuré pour la mise en oeuvre d'une application de traitement d'enregistrements, l'enregistrement ayant une structure comportant une pluralité de champs prédéterminée dans la base de données, l'enregistrement étant accédé par l'application via une modélisation selon un langage objet dans lequel la structure d'un enregistrement est modélisé par une classe, chaque champs de l'enregistrement correspond à une méthode de la classe, caractérisé en ce que le procédé comporte les étapes suivantes pour chaque méthode : - Analyse 2030 de la déclaration de la méthode telle que connue au lancement de l'application pour trouver au moins une annotation prédéterminée ; - Si l'annotation prédéterminée est trouvée, alors analyse des paramètres de l'annotation pour produire 2040 un contrôle de saisie ; - Utilisation 2050 du contrôle produit pour la création d'une interface homme machine permettant ...

DATABASE SYSTEM AND METHOD

A transformation description language (TDL) for specifying how data is to be manipulated in a data warehousing application. The TDL is comprised of a source for storing raw data, one or more transformation objects for processing the raw data according to predefined instructions, and a target for storing the processed data. A mapping is used for directing the data flow between the I/O ports corresponding to the source, the plurality of transformation objects, and the target. The mapping specifies the connectivity between the source, transformation, and target objects as well as the order of these connections. There are a number of different transformations which can be performed to manipulate the data. Some such transformations include: an aggregator transformation, an expression transformation, a filter transformation, a lookup transformation, a query transformation, a sequence transformation, a stored procedure transformation, and an update strategy transformation.

Multi-service form file processing system and method

The invention discloses a multi-service form file processing system and method, in the above system, comprising a configuration reading module, a list scanning module, one or more than one list analyzing module, and one or more than one input processing module, wherein the configuration reading module is used for reading the configuration information from the preset configuration file; the list scanning module is used for scanning the file of the original call-ticket list and sending part or all the file to one or more than one list based on a first configuration rule; one or more than one list analyzing module is used for analyzing part or all the file into a record of database form type based on a second configuration rule; the input processing module is used for downloading and storing the record in the database. The code maintenance cost and the code complexity can be reduced and the program stability and the flow clearness can be ensured.

Queries involving multiple databases and execution engines

Apparatus and method for structuring web page access history

Data extensibility using external database table

QUERY PIPELINE

A query pipeline is created (514) from a query request. The query pipeline includes multiple query operations including multiple query operators. A first query operator and a second query operator perform first and second query operations on a database (526) and on data outside the database (534). A result from the first query operation in the query pipeline is fed to the second query operation in the query pipeline.

FACTORY AUTOMATION TRANSACTIONS

A system that facilitates direct communication of a transaction between an automation controller and a business system comprises a request analyzer that receives a request for data relating to the automation controller and locates a transaction definition within the automation controller based upon the request. A subscribing component subscribes the business system to the automation controller based at least in part upon the located transaction definition. 1. A method , comprising:inputting, by a device including a processor, transaction definitions into an automation controller;receiving, by the device, a request for a defined transaction with a business system database;subscribing, by the device, the business system database to the defined transaction; andinitiating, by the device, the transaction directly between the business system database and the automation controller via the standard transaction engine.2. The method of claim 1 , further comprising sensing claim 1 , by the device claim 1 , at least one of an event or a condition that triggers initiating of the transaction.3. The method of claim 2 , further comprising acknowledging claim 2 , by the device claim 2 , the at least one of the event or the condition prior to initiating the transaction.4. The method of claim 2 , the at least one of the event or the condition is an alarm.5. The method of claim 2 , the at least one of the event or the condition is a passage of a defined amount of time.6. The method of claim 1 , further comprising rolling back claim 1 , by the device claim 1 , the transaction upon a communication failure between the business system database and the automation controller.7. The method of claim 1 , the transaction comprising at least one of data in a maintenance class claim 1 , a production schedule class claim 1 , or a batch record class.8. The method of claim 7 , further comprising formatting data in the production schedule class in SP95 format.9. A system claim 7 , comprising:a ...

SYSTEM AND METHODS THEREOF FOR DETECTION OF USER DEMOGRAPHIC INFORMATION

A system and method for detecting implicit information related to a user are disclosed. The system comprises a network interface for accessing one or more data sources; a plurality of agents for collecting textual content from the one or more data sources; a processing unit for extracting explicit demographic information related to the user and generating implicit demographic information related to the user respective of the explicit demographic information; and a data warehouse connected to the processing unit for storing the generated implicit demographic information. 1. A system for detecting implicit information related to a user , comprising:a network interface for accessing one or more data sources;a plurality of agents for collecting textual content from the one or more data sources;a processing unit for extracting explicit demographic information related to the user and generating implicit demographic information related to the user respective of the explicit demographic information; anda data warehouse connected to the processing unit for storing the generated implicit demographic information.2. The system of claim 1 , wherein the plurality of agents are further configured to collect data respective of the user's common activity with respect of the one or more data sources; and claim 1 , to store the data collected in the data warehouse.3. The system of claim 2 , wherein the user's common activity is at least one of: a typing speed claim 2 , language proficiency claim 2 , time of the day the user is usually active in the one or more data sources.4. The system of claim 1 , wherein the one or more data sources are at least one of: a social network claim 1 , a blog claim 1 , a news feed claim 1 , and a web page.5. The system of claim 1 , wherein the implicit demographic information is at least one of the following statistics characterizing the user: user age claim 1 , gender claim 1 , relationship status claim 1 , location claim 1 , home address claim 1 , work ...

SYSTEM AND METHOD FOR DATA AGGREGATION, INTEGRATION AND ANALYSES IN A MULTI-DIMENSIONAL DATABASE

The present invention relates to a system and method for flexibly integrating a data warehouse with one or more operational database in real-time. The system comprises of a plurality of components embedded under one or more layers. The system comprises of a staging database, an ETL (extract transform load) module, a data warehouse and a networking module. The flexible integration is enabled by invoking one or more web services from a business enterprise. The invention further provides one or more analytical modules for performing an intelligent operation to carry out comparative analyses of the data. The designing and implementation module enables a deployment of the system on any related technology platform and tools. 1. A system for flexibly integrating a data warehouse with one or more operational database in real-time , a first node communicatively accessing one or more data elements from one or more disparate databases installed at a plurality of disparate computing environment , the system comprising:an extracting module for extracting a data from one or more disparate databases;a staging database configured for correlating and accumulating the data for supporting a mining of a transactional data out of the data, the correlated data is further processed through an extraction transformation and load routine for loading it onto the data warehouse, the data warehouse configured for storing the data in a summarized form;a networking module configured to enable the flexible integration of the data warehouse with the one or more operational systems and enterprise processes to support real time sharing of data from the data warehouse to the connected operational systems and enterprise processes;one or more analytical module communicatively coupled to the data warehouse for performing an intelligent operation to provide an insight into a customer's behavior and business performance of one or more industries by enabling a comparative analyses with respect to the data; ...

ON-LINE TRANSACTION PROCESSING (OLTP) COMPRESSION AND RE-COMPRESSION OF DATABASE DATA

A computer is programmed to compress data of a database in response to database modification language (DML) statements generated by on-line transaction processing (OLTP) systems. In several embodiments, data that is initially added to a database block is left uncompressed until a predetermined condition is satisfied, which happens infrequently (relative to OLTP transactions on the block). When satisfied, the computer automatically compresses all uncompressed data in the block, which increases the amount of unused space in the block. New data is thereafter added uncompressed to the partially compressed block, until satisfaction of a predetermined condition whereby the partially compressed block is again compressed, i.e. re-compressed. Adding of new data to a partially compressed block and its compression are repeated unless another predetermined condition is met, in response to which the block is not further re-compressed, thereby to recognize a limit on the benefit from compression. 1. A method implemented in a computer for compression of data in a database , the method comprising:determining whether at least one predetermined condition is satisfied; decompressing first compressed data from a block comprising the first compressed data and first uncompressed data, to obtain second uncompressed data from the first compressed data, the block being comprised among multiple blocks in the database; and', 'compressing at least the first uncompressed data and the second uncompressed data together, to generate second compressed data; and, 'in response to the determining that the at least one predetermined condition is satisfiedstoring the block in a mass storage device.2. The method of wherein:unused space in the block after compressing is greater than unused space in the block prior to decompressing.3. The method of further comprising:writing new uncompressed data to the block if at least another predetermined condition is satisfied, else writing the new uncompressed data ...

METHOD AND APPARATUS FOR CONSTRUCTING DATA WAREHOUSE TO MANAGE MASS BUILDING ENERGY INFORMATION

A method and apparatus for constructing a data warehouse for managing mass building energy information are provided. The apparatus for constructing a data warehouse in order to manage mass building energy information includes a primitive database generator for constructing primitive databases in each building based on energy control point information, a time series data warehouse generator for generating time series data warehouses based on the constructed primitive databases, a space time series data warehouse generator for reconstructing the generated time series data warehouses based on an object construction information table including space information about objects for control points of the plurality of buildings to be controlled to generate space time series data warehouses, and a view database generator for generating view databases by a search condition in accordance with the search condition for energy usage information based on the generated space time series data warehouses. 1. An apparatus for constructing data warehouses in order to manage mass building energy information , comprising:a primitive database generator for constructing primitive databases in each building of a plurality of buildings to be controlled, based on energy control point information received from the plurality of buildings to be controlled;a time series data warehouse generator for generating time series data warehouses based on the constructed primitive databases;a space time series data warehouse generator for reconstructing the generated time series data warehouses, to generate space time series data warehouses, based on an object construction information table including space information about objects for control points of the plurality of buildings to be controlled; anda view database generator for generating view databases by a search condition in accordance with a search condition for energy usage information based on the generated space time series data warehouses.2. The ...

SYSTEM AND METHOD FOR MATHEMATICS ONTOLOGY EXTRACTION AND RESEARCH

An extensive computer based online math research system (the “Research System”) having as its foundation an Ontology of mathematics, and utilizing unique and intensive computer support, coordination, data structuring, data storage, computer processing, retrieval capabilities, and data-mining capabilities, and an Ontology editing system that runs on computer software with computer processors and data storage capabilities (the “Ontology Editor System”). The Research System also includes a methodology to enable online reference and data manipulation of the Ontology, and an Internet based search of the concepts of mathematics and applications of mathematics to the sciences on the basis of the Ontology. 1. (canceled)2. A method of creating a mathematical research system comprising:creating a plurality of concept line items (CLI)s from at least one mathematical problem, wherein a CLI is a mathematical concept expressed as a word phrase;defining interrelationships between said CLIs;generating, by a computer processor, a mapping of said interrelationships of said CLIs; andstoring said CLIs and said generated mapping in at least one database.3. The method of further comprising:creating additional CLIs from a second mathematical problem;redefining said interrelationships based on said additional CLIs;regenerating said mapping based on said redefined interrelationships; andstoring said additional CLIs and said regenerated mapping in said at least one database.4. The method of wherein said interrelationships are directional relationships claim 2 , wherein a directional relationship is defined as one of: a prerequisite relationship claim 2 , a dependency relationship claim 2 , and a lack of relationship.5. The method of wherein said generated mapping is a directional mapping positioning prerequisite CLIs before dependency CLIs claim 4 , thereby showing an order in which said CLIs are to be learned by a student.6. The method of further comprising:searching, by said computer ...

Interest-Driven Business Intelligence Systems and Methods of Data Analysis Using Interest-Driven Data Pipelines

Interest-driven Business Intelligence (BI) systems in accordance with embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata. 1. A data processing system , comprising:raw data storage containing raw data;metadata storage containing metadata that describes the raw data; andan interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data;wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.2. The data processing system of claim 1 , wherein the interest-driven data pipeline is automatically compiled based upon at least one change selected from the group consisting of a change to the reporting data requirements claim 1 , a change to the metadata claim 1 , and a change to the raw data.3. The data processing system of claim 1 , further comprising:source data storage configured to store source data; andwherein the interest-driven data pipeline is configured to create source data by performing an extract, transform, and load (ETL) process on raw data using a source model.4. The data processing system of claim 4 , further comprising:aggregate data storage configured to store aggregate data; andwherein the interest-driven data pipeline is configured to generate aggregate data by performing aggregations using the source data.5. The data processing system of claim 4 , wherein the interest-driven data pipeline is configured to generate aggregations ...

Dynamic access of data

A heterogeneous information technology system in which compatible and incompatible client systems are able to dynamically access master data stored in a master database maintained by a master data server. An integration server communicates with the client systems and the master data server, and in response to a request for data from a client system, maps the master data in the master database to mapped data based on a set of mapping rules associated with the client system, and sends the mapped data to the client system.

Interest-Driven Business Intelligence Systems and Methods of Data Analysis Using Interest-Driven Data Pipelines

Interest-driven Business Intelligence (BI) systems in accordance with embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.

SYSTEMS AND METHODS FOR CATEGORIZING AND PRESENTING PERFORMANCE ASSESSMENT DATA

The field of the invention relates to systems and methods for data mining and processing, and more particularly to systems and methods for automating content from performance assessment data. In one embodiment, an automated notes and categorization system may include a primary database comprising performance assessment data. The primary database is operatively coupled to a computer program product having a computer-usable medium having a sequence of instructions which, when executed by a processor, causes said processor to execute a process that analyzes and converts raw performance data into automated content that presents data in readable user friendly form to facilitate human understanding. 1. An automated notes and categorization system comprising:a primary database, the primary database having raw performance assessment data;wherein the primary database is operatively coupled to a computer program product having a computer-usable medium having a sequence of instructions which, when executed by a processor, causes said processor to execute an electronic process that analyzes and converts said raw performance data;a processed database for storing the processed data operatively couple to the computer program product; and data mining said performance assessment data to obtain summarized data;', 'prioritizing said summarized data based on user-defined weight values for a plurality of classification categories; and', 'converting results of the prioritization into automated plain language notes., 'said electronic process comprising2. The system of claim 1 , further comprising a secondary database having user-generated (secondary) performance assessment data.3. The system of claim 1 , wherein said process further comprises determining whether said summarized data is positive or negative to the classification category.4. The system of claim 1 , wherein the automated plain language notes include historical trend of said summarized data.5. The system of claim 4 , wherein ...

SINGLE COMMAND DATA WAREHOUSE TABLE UPDATE

Information warehouse table data is updated through a full outer join between warehouse tables and source tables in response to a single query language command statement of a relational database computer language. The single command statement comprises table mapping instructions for updating the warehouse tables that define matching conditions, delete instructions, archive instructions, and update instructions. The full outer join inserts rows in the warehouse tables for source table data rows that do not have corresponding matching rows in the warehouse table. Data rows in the warehouse tables without corresponding matching source table rows are deleted if a value of a parameter in the warehouse table system is set to delete; updated if the parameter is update; and updated with archive values and a new table row inserted with existing values if the parameter is archive. 1. A method for updating table data in an information warehouse via a single query language command , the method comprising: deleting the first data row if a value of a treat history parameter in the warehouse table system is set to delete;', 'updating the first data row with update values if the warehouse table system treat history parameter is set to update; and', 'updating the first data row with archive values and inserting a new table row into the at least one warehouse table that comprises existing values from the first data row if the warehouse table system treat history parameter is set to archive; and, 'in response to a single query language command statement of a relational database computer language comprising table mapping instructions for updating at least one information warehouse table of an information warehouse database system with data from at least one source table, performing a full outer join between the at least one warehouse table and the at least one source table based on the command matching conditions defined by the single query language command statement by the steps of, ...

System and method for correct and effective transaction execution in a data center system by implementing FIFO-grouping in a data store

A data center is a computing environment focusing on the execution of data transactions for corporations and organizations. The data center has the capability of running many execution threads in parallel. 1. A data center using FIFO-grouping enabled data store , running a number of tasks for a number of customers , where based on selection criteria , a subset of the corporate data items will be selected.2. A data center as in claim 1 , where FIFOs of data items will be established claim 1 , with a FIFO per an group-type value as defined by the selection criteria.3. A data center as in claim 2 , where the FIFOs will be derived from the FIFO-grouping enabled indices.4. A data center as in where an available execution threads will be assigned per FIFO5. A data center as in where only one execution thread can be assigned to a certain FIFO.6. A data center as in where other execution threads can be assigned to FIFOs which are not being executed upon.7. A data center as in where the selection criteria can be changed from time to time.8. A data center as in where the execution is under a transaction per FIFO data element9. A data center as in where the first data element in a FIFO is being executed This non provisional patent application claims benefit of the filing date of provisional application 61/588,191 filed on Jan. 19, 2012, as a priority date for this applicationIn high throughput OLTP systems where partial ordering of incoming data is essential to the correctness of the processing, but total ordering of the data incurs a large processing time overhead. Examples for this kind of system are financial order management systems, which have to process large amounts of orders, but in most cases require ordering of incoming orders only between orders for a specific security or stock, but not across the entire set of orders.1 A Queue based systems. In queue based system a queue (or queues) are created in advance using a queue factory. In order to process a queue the ...

SYSTEM AND METHOD FOR PRESENTING AND MANAGING SOCIAL MEDIA

A method, system, apparatus, and computer program product provide the ability to present media content on a social media network. Live streaming data is received from multiple data sources. The data is enriched with meta information using internal systems via application programming interfaces (APIs). The live streamed data is transformed and transmitted, based on the meta information, to one or more users on a social media network. 1. A computer-implemented method for presenting media content on a social media network comprising:receiving, in a computer, live streaming data from multiple data sources;enriching the live streaming data with meta information using one or more internal systems via application programming interfaces (APIs) for the one or more internal systems; andtransforming and transmitting the live streamed data based on the meta information, wherein the transformed live streamed data is transmitted to one or more users on a social media network.2. The computer-implemented method of claim 1 , further comprising:normalizing the live streaming data into a common format via extraction transformation, and loading functions.3. The computer-implemented method of claim 1 , wherein one of the one or more internal systems comprises a frame application that identifies an influence of a user or resource within a pre-defined context based on point values assigned to activities that occur within the pre-defined context.4. The computer-implemented method of claim 1 , wherein one of the one or more internal systems comprises a tag inheritance system that determines affinities for resources and users via the live streamed data.5. The computer-implemented method of claim 1 , wherein the enriching comprises:maintaining state between one or more events such that meta information from one of the one or more internal systems is applied to a second of the one or more internal systems.6. The computer-implemented method of claim 1 , further comprising ensuring that the live ...

COLUMN BASED DATA TRANSFER IN EXTRACT, TRANSFORM AND LOAD (ETL) SYSTEMS

Executing a plurality of transform stages in an extract, transform and load (ETL) job including, for each of the transform stages, receiving a plurality of input row identifiers (RIDs) corresponding to a first plurality of source database table rows in a source database table. Data is retrieved directly from a subset of the source database table columns in the first plurality of source database table rows based on the input RIDs and transform logic. Partial row data including data from the subset of the source database table columns is generated for each of the first plurality of source database table rows. Transformed data is generated based on the partial row data and to the transform logic. Output RIDs corresponding to a second plurality of rows in the source database table that include a least a subset of the transformed data are output to a downstream stage. 1. A method comprising: receiving, from an upstream stage, a plurality of input row identifiers (RIDs) corresponding to a first plurality of source database table rows in the source database table;', 'retrieving data directly from a subset of the source database table columns in the first plurality of source database table rows in the source database table, the retrieving responsive to the input RIDs and to transform logic associated with the transform stage;', 'generating partial row data for each of the first plurality of source database table rows, the partial row data comprising data from the subset of the source database table columns;', 'generating transformed data responsive to the partial row data and to the transform logic; and', 'outputting, to a downstream stage, a plurality of output RIDs corresponding to a second plurality of source database table rows that include at least a subset of the transformed data., 'executing a plurality of transform stages in an extract, transform and load (ETL) job, the ETL job including an extract stage and a load stage in addition to the plurality of transform ...

EMBEDDING AND RETRIEVING DATA IN AN APPLICATION FILE FORMAT

Third party data embedding may be provided. Data associated with a third party provider may be saved in a native application save file. If a save file opened by the application is determined to comprise third party data, the application may determine whether an appropriate data provider is available. The application may connect to the data provider and provide on-demand, asynchronous access to the third party data upon request from a user. 1opening the application file associated with an application, wherein the file comprises a third party data segment; 'in response to determining that the data provider is available, establishing a connection between the application and the data provider.', 'determining whether a data provider associated with the third party data segment is available; and'}. A method for embedding data in an application file, the method comprising: The present application is a continuation of U.S. patent application Ser. No. 12/605,126, filed Oct. 29, 2009, now U.S. Pat. No. 8,429,118 entitled Embedding and Retrieving Data in an Application File Format, which is assigned to the assignee of the present application, and hereby incorporated by reference.Embedding and retrieving data in an application file format is an extensible process for storing data in a work file without the application needing to understand the content and/or the format of the data. In some situations, users have no way to move and/or access data from an external data source. For example, data may be stored in an online database and inaccessible while working offline. If manipulating the data comprises complex, multi-dimensional queries such as Online Analytical Processing (OLAP), large amounts of data may need to be stored and/or managed separately from other working data. This conventional strategy may cause problems as the management of separate data files may cause distribution and synchronization issues, especially if the data is part of a collaboration project. Furthermore ...

Method for Operating an Automation Device

A method for operating an automation device, into the memory of which an automation solution has been loaded, wherein a technology-oriented control interpreter accesses a data warehouse of the automation solution, and is able to control external commands by virtue of such commands being analyzed and being implemented according to the analysis, where the technology-oriented control interpreter extracts at least one entity designation and at least one instruction from a respective command, the technology-oriented control interpreter searches for an object matching the entity designation in the data warehouse of the automation solution and, in the event of success, checks whether the instruction contained in the command has been defined for the found object, and where the technology-oriented control interpreter causes execution of the instruction for the found object. 1. A method for operating an automation device having a memory , comprising:loading an automation solution into the memory;accessing a data warehouse of the automation solution by a technology-oriented control interpreter;analyzing, by the technology-oriented control interpreter, external commands and implementing said external commands according to said analysis to control the external commands;extracting, by the technology-oriented control interpreter, at least one entity designation and at least one instruction from a respective external command;searching, by the technology-oriented control interpreter, for an object matching the at least one entity designation in the data warehouse of the automation solution;checking whether the instruction contained in the command has been defined for the located object in an event of successfully locating the object matching the at least one entity designation in the data warehouse of the automation solution; andexecuting, via the technology-oriented control interpreter, the instruction for the located object.2. The method as claimed in claim 1 , wherein the data ...

Defining and Mapping Application Interface Semantics

An example implementation includes a system having one or more processors, a definition module, a data store, and a workflow module. The definition module is executable by the one or more processors to determine a first semantic element for a first operation included in a first application interface representing a first service and a second semantic element for a second operation included in a second application interface representing a second service, such as an enterprise service bus. The definition module may further determine a mapping of the first semantic element with the second sematic element. The data store is configured to store the mapping of the first semantic element with the second sematic element by the definition module. The workflow module is executable by the one or more processors to integrate the first application interface with the second service based on the mapping. 1. A computer-implemented method comprising:determining, using one or more computing devices, a first semantic element for a first operation included in a first application interface representing a first service;determining, using the one or more computing devices, a second semantic element for a second operation included in a second application interface representing a second service;mapping, using the one or more computing devices, the first semantic element with the second sematic element; andintegrating, using the one or more computing devices, the first application interface with the second service based on the mapping.2. The computer-implemented method of claim 1 , wherein the second service includes one of an enterprise service bus claim 1 , an extract-transform-load framework claim 1 , and a gateway.3. The computer-implemented method of claim 1 , wherein the first semantic element includes a field associated with an input message of the first operation claim 1 , and the second semantic element includes a field associated with one of an output message and an error message of ...

Method and System for Analyzing Data Related to an Event

A system and method for analyzing data from a plurality of computer environments. A user may search for computer environments that meet a certain criterion. The computer environments are authenticated and data is copied from the computer environments to a memory location. The data may be marked so that a user may determine which computer environment provided the data. The user may add notations to the data during a review. Changes to data on the computer environments may be provided to a user using a syndication feed. 1. A method of analyzing data related to an event comprising:(a) copying a plurality of files from a plurality of storage devices to a memory location;(b) converting the plurality of files to a predetermined unified format;(c) analyzing the plurality of files; and(d) providing a report to a user based on the analysis.2. The method of wherein analyzing the data comprises sorting the data using a variable.3. The method of wherein the event is a user request.4. The method of wherein the event is a response to an incident.5. The method of wherein analyzing the data comprises sorting the data into data groups based on one or more variables.6. The method of comprising comparing the data groups and providing a report to the user showing the union of two groups.7. The method of comprising comparing the data groups and providing a report to the user showing the intersection of two groups.8. The method of comprising comparing the data groups and providing a report to the user showing the difference between two groups.9. The method of comprising comparing the data groups and providing a report to the user showing the similarities between two groups.10. The method of wherein the report comprises a list of the plurality of files.1165-. (canceled) As more businesses and governmental entities increasingly rely on computer networks to conduct their operations and store relevant data, security of these networks has become increasingly important. The need for increased ...

Method and System for Analyzing Data Related to an Event

A system and method for analyzing data from a plurality of computer environments. A user may search for computer environments that meet a certain criterion. The computer environments are authenticated and data is copied from the computer environments to a memory location. The data may be marked so that a user may determine which computer environment provided the data. The user may add notations to the data during a review. Changes to data on the computer environments may be provided to a user using a syndication feed. 110-. (canceled)11. A method of analyzing data comprising:(a) receiving a search request from a user;(b) searching for storage devices according to the search request;(c) searching for data on the storage devices according to the search request;(d) copying the data to a memory location;(e) converting the data to a user-defined format;(f) analyzing the data; and(g) providing a report to the user based on the analyzing of the data.12. The method of wherein the storage devices are coupled to a user computer through a network.13. The method of wherein the user-defined format is extensible markup language.14. The method of further comprising storing the data in a database.15. The method of wherein analyzing the data comprises searching the data for a user-defined variable.16. The method of wherein the user-defined variable is a file type.17. The method of wherein the user-defined variable is a bit string.18. The method of wherein analyzing the data comprises creating a plurality of groups of data according to a plurality of user-defined variables.19. The method of further comprising comparing the groups.20. A system for analyzing data comprising:(a) means for receiving a search request from a user;(b) means for searching for storage devices according to the search request;(c) means for searching for data on the storage devices according to the search request;(d) means for copying the data to a memory location;(e) means for converting the data to a user- ...

IN-QUERYING DATA CLEANSING WITH SEMANTIC STANDARDIZATION

The present invention relates to data cleansing, and in particular performing the semantic standardization process within a database before the transform portion of the extract-transform-load (ETL) process. Provided are a method, system and computer program product for standardizing data within a database engine, configuring the standardization function to determine at least one standardized value for at least one data value by applying the standardization table in a context of at least one data value, receiving a database query identifying the standardization function, at least one database value and the context of the data, and invoking the standardization function. 1. A system comprising: a processor; and a memory containing program code which when executed by the processor is configured to perform an operation , comprising:receiving at least one standardization table, wherein each standardization table comprises a first column containing a plurality of text representations of a standard text and a second column containing the standardized text; andupon receiving a database query identifying a standardization function and at least one data value to be converted to a standard value, converting the at least one data value to a standard value by invoking the standardization function within a database engine, wherein the standardization function uses the at least one standardization table and a context of the at least one data value to convert the at least one data value to the standard value.2. The system of wherein the standardization function is a user defined function.3. The system of claim 2 , wherein the user defined function receives the context of a least one data value as a parameter.4. The system of claim 2 , wherein the user defined function further comprises:determining a context of the data value.5. The system of claim 4 , wherein determining the context further comprises:accessing one or more metadata values describing the data value; anddetermining the ...

BIG DATA ANALYTICS SYSTEM

A big data analytics system obtains a plurality of manufacturing parameters associated with a manufacturing facility. The big data analytics system identifies first real-time data from a plurality of data sources to store in memory-resident storage based on the plurality of manufacturing parameters. The plurality of data sources are associated with the manufacturing facility. The big data analytics system obtains second real-time data from the plurality of data sources to store in distributed storage based on the plurality of manufacturing parameters. 1. A method comprising:obtaining a plurality of manufacturing parameters associated with a manufacturing facility;identifying, by a computing system comprising a processing device, first real-time data from a plurality of data sources to store in memory-resident storage based on the plurality of manufacturing parameters, wherein the plurality of data sources are associated with the manufacturing facility; andidentifying, by the computing system, second real-time data from the plurality of data sources to store in distributed storage based on the plurality of manufacturing parameters.2. The method of claim 1 , wherein the plurality of manufacturing parameters are associated with an event claim 1 , and further comprising:obtaining a subset of the first real-time data from the memory-resident storage upon the occurrence of the event;determining whether additional data is needed to analyze the event; andobtaining the additional data upon determining that the additional data is needed to analyze the event, wherein the additional data is obtained from the memory-resident storage if the additional data is stored in the memory-resident storage, and wherein the additional data is obtained from the distributed storage if the additional data is not stored in the memory-resident storage.3. The method of claim 1 , further comprising:creating a graphical representation for the first real-time data based on the plurality of ...

DETECTING REFERENCE DATA TABLES IN EXTRACT-TRANSFORM-LOAD PROCESSES

A method, system and computer program product for identifying reference data tables in an Extract-Transform-Load (ETL) process, by identifying, by operation of one or more computer processors, at least a first reference data operator in the process, wherein the first reference data operator references one or more tables and evaluating at least a first table referenced by the reference data operator to determine whether the first table is a reference data table by assigning a score to the first table, wherein the score is indicative of the likelihood that the first table is a reference data table and wherein a reference data table contains a set of values that describes other data. 1. A computer-implemented method for identifying reference data tables in an extract-transform-load data integration process comprising:identifying, by operation of one or more computer processors, at least a first reference data operator in the process, wherein the first reference data operator references one or more tables; andevaluating at least a first table referenced by the reference data operator to determine whether the first table is a reference data table by assigning a score to the first table, wherein the score is indicative of the likelihood that the first table is a reference data table, wherein a reference data table contains a set of values that describes other data.2. The computer-implemented method of claim 1 , further comprising:upon determining that the first table is a reference data table based on the score exceeding a predetermined threshold, loading the first table into a data store.3. The computer-implemented method of claim 1 , further comprising:presenting the first table to a user; andupon receiving user input specifying that the first table is a reference data table, loading the first table into a data store.4. The computer-implemented method of claim 1 , wherein identifying at least the first reference data operator comprises determining whether the first ...

AUTOMATED DATA ANALYSIS AND TRANSFORMATION

A transformation method and system is provided. The method includes generating a data hub application configured to embed extract, transform, and load (ETL) processes. The data hub application is linked to source tables and target tables. Meta data associated with the source and target tables is transferred from virtual views of the data hub application to an ETL work area of the ETL processes. An ETL job is generated and linked to the data hub application. ETL processes are executed and results are determined. 1. A method comprising:linking, by a computer processor of a data hub, source tables and target tables to a data hub application configured to embed, extract, transform, and load (ETL) processes;associating, by said computer processor, said source tables and said target tables to a local sensitive hashing (LSH) program comprising target flags;transferring, by said computer processor, metadata associated with said source tables and said target tables from virtual views of said data hub application to an ETL work area of said ETL processes, wherein said metadata comprises table definition metadata published to a DS tool comprising said source tables and said target tables;linking, by said computer processor, ETL job to said data hub application;executing, by said computer processor executing a data hub scheduler application, said ETL processes;determining, by said computer processor, results of said executing, wherein said results indicate that said executing was not successful; and analyzing, by said computer processor, a log file pattern indicating a reason that said executing was not successful;', 'decoding, by said computer processor, said log file pattern;', 'generating, by said computer processor, a change script based on a category of said log file pattern, wherein said change script comprises an exception routine associated with non-classified categories;', 'notifying, by said computer processor, users of said changes to said metadata and said change ...

AQUATIC GEOGRAPHIC INFORMATION SYSTEM

A method of processing geo-statistical data includes preparing a data log, extracting acoustic data and coordinate data from the data log, and aligning the acoustic data and the coordinate data. The method also includes cleaning and aggregating the coordinate data, validating the coordinate data geospatially, and creating an output. 1. A method of processing geo-statistical data , the method comprising:preparing a data log of geo-statistical data from a monitoring system;extracting acoustic data and coordinate data from the data log;aligning the acoustic data and the coordinate data;cleaning and aggregating the coordinate data;validating the coordinate data geospatially; andcreating an output.2. The method of claim 1 , wherein the output is a contour map.3. The method of claim 1 , wherein the coordinate data is combined with additional coordinate data prior to creating the output.4. The method of claim 3 , wherein the additional coordinate data includes a first set of additional coordinate data and a second set of additional coordinate data.5. The method of claim 1 , wherein the output is a contour map of a water body having a plurality of depth ranges claim 1 , the method further comprising:correlating a water body parameter to at least one of the depth ranges.6. The method of claim 1 , wherein the output is a contour map of a water body having a plurality of depth ranges claim 1 , the method further comprising:correlating a water body parameter to each depth range.7. The method of claim 1 , wherein creating the output comprises:creating a first contour map with a first plurality of depth ranges from the coordinate data; andcreating a second contour map with a second plurality of depth ranges from the coordinate data.8. The method of claim 7 , wherein the first plurality of depth ranges are differentiated by 0.30 meters and the second plurality of depth ranges are differentiated by 0.91 meters.9. The method of claim 1 , wherein creating the output comprises: ...

SYSTEMS AND METHODS FOR COLLECTION AND CONSOLIDATION OF HETEROGENEOUS REMOTE BUSINESS DATA USING DYNAMIC DATA HANDLING

Remote data collection systems and methods retrieve data including financial, sales, marketing, operational and the like data from a plurality of databases and database types remotely over a network in an automated, platform-agnostic manner. An Extract Transform and Load (ETL) data replication method for Chart of Account (COA) standardization includes receiving a request for remote data collection to extract data from a data source; extracting data in a non-intrusive manner from the data source, wherein the data comprises non-standard COA data; and transforming one of an entire set or a subset of the extracted data based on the request based on a template or a standardized form desired for comparisons. 1. An Extract Transform and Load (ETL) data replication method for Chart of Account (COA) standardization , comprising:receiving a request for remote data collection to extract data from a data source;extracting data in a non-intrusive manner from the data source, wherein the data comprises non-standard COA data; andtransforming one of an entire set or a subset of the extracted data based on the request based on a template or a standardized form desired for comparisons.2. The ETL data replication method of claim 1 , further comprising:extracting the data using a two phase process comprising a reconciliation phase and a collection phase.3. The ETL data replication method of claim 1 , further comprising:performing the transforming as an agent side process, a server side process, or an automated client side process.4. The ETL data replication method of claim 1 , further comprising:reforming the data as the non-standard COA data so the extracted data conforms to a new standard.5. The ETL data replication method of claim 1 , further comprising:generating read only mapped views on the extracted data using a standard COA without modifying underlying data source.6. The ETL data replication method of claim 1 , further comprising:presenting a graphical user interface of a ...

Cross-Platform Compiler for Data Transforms

Techniques for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms are provided. The techniques include performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms, automatically partitioning the transform graph to one or more target output platforms based on one or more policies, performing an optimization of the partitioned transform graph, and generating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms. 1. A method for automatically partitioning a multi-platform data transform flow graph to one or more target output platforms , wherein the method comprises:performing type inference on a transform graph, wherein the transform graph comprises one or more data transforms;automatically partitioning the transform graph to one or more target output platforms based on one or more policies;performing an optimization of the partitioned transform graph; andgenerating code, from the partitioned transform graph, for each set of the one or more data transforms based on the one or more target output platforms.2. The method of claim 1 , wherein generating code comprises generating adapter code to facilitate data flow between the one or more target output platforms.3. The method of claim 1 , wherein performing an optimization of the partitioned transform graph comprises fusing one or more of the one or more data transforms if fusing of one or more data transforms is appropriate.4. The method of claim 1 , wherein performing type inference on a transform graph comprises parsing one or more entity types from at least one of a database schema and one or more virtual stream descriptions.5. The method of claim 1 , wherein performing type inference on a transform graph comprises determining each input and output schema for all one or more data transforms.6. The method of claim 1 , ...

COMPRESSED SET REPRESENTATION FOR SETS AS MEASURES IN OLAP CUBES

A cardinality of an incoming data stream is maintained in real time; the cardinality is maintained in a data structure that is represented by an unsorted list at low cardinalities, a linear counter at medium cardinalities, and a PCSA at high cardinalities. The conversion to the linear counter makes use of the data in the unsorted list, after which that data is discarded. The conversion to the PCSA uses only the data in the linear counter. 1. A method for storing a cardinality of a plurality of unique items of data in a received data stream , the method comprising:storing, in a computer memory, unique data items in the form of an unsorted list;when the number of unique items stored in the computer memory crosses a first threshold, converting the items in the unsorted list into a linear counter executable by a processor, discarding the unsorted list from the computer memory, and causing the processor to count further unique items in the received data stream using the linear counter; andwhen the number of unique items stored in the computer memory crosses a second threshold, converting the data in the linear counter into a PCSA structure or log-log counter, discarding the linear counter from the computer memory, and causing the processor to count further unique items in the received data stream using the PCSA structure or log-log counter.2. The method of claim 1 , wherein the first threshold is 1 claim 1 ,024 elements and the second threshold is 20 claim 1 ,000 elements.3. The method of claim 1 , wherein the unsorted list claim 1 , linear counter claim 1 , and PCSA or log-log counter use the same memory footprint.4. The method of claim 3 , wherein a size of the memory footprint is less than approximately four kilobytes.5. The method of claim 1 , wherein converting the data in the linear counter into the PCSA structure comprises inserting a plurality of random numbers into the PCSA structure.6. The method of claim 5 , wherein the plurality of random numbers has a size ...

Integrating Data Transform Test with Data Transform Tool

Embodiments relate to integrating data transform test with a data transform tool. A method and system are described for creating a data transform test for a data transform job having a data transform script, the method includes determining all data transform units available in the data transform job, determining a subset of the available data transform units for a new test, and generating a subset test execution script for the subset of data transform units from the data transform script. The method further includes determining boundary test data at each boundary of the subset of data transform units, defining a data transform test, and saving the data transform test for later testing. The data transform test includes the subset of data transform units with subset test execution script and with boundary test data.

METHODS AND SYSTEMS FOR DISPLAYING AND FILTERING BUSINESS ANALYTICS DATA STORED IN THE CLOUD

In accordance with disclosed embodiments, there are provided methods, systems, and apparatuses for displaying and filtering business analytics data stored in the cloud, including, for example, means for displaying a graphical interface at a client device; communicating a business analytics query from the client device to a remote host organization via a public Internet; receiving a business analytics dataset in a complete and unfiltered form from the host organization responsive to the business analytics query; caching the business analytics dataset in its complete and unfiltered form to the memory of the client device; displaying a business analytics report at the graphical interface of the client device, the business analytics report representative of the business analytics dataset in its complete and unfiltered form; receiving filter input at the client device; applying the filter input to the business analytics dataset to yield a filtered sub-set; and updating the business analytics report displayed at the graphical interface using the filtered sub-set of the business analytics dataset. Other related embodiments are disclosed. 1. A method at a client device having at least a processor and a memory therein , wherein the method comprises:displaying a graphical interface at the client device;communicating a business analytics query from the client device to a remote host organization via a public Internet;receiving a business analytics dataset in a complete and unfiltered form from the host organization responsive to the business analytics query;caching the business analytics dataset in its complete and unfiltered form to the memory of the client device;displaying a business analytics report at the graphical interface of the client device, the business analytics report representative of the business analytics dataset in its complete and unfiltered form;receiving filter input at the client device;applying the filter input to the business analytics dataset to yield a ...

ENHANCING PERFORMANCE OF EXTRACT, TRANSFORM, AND LOAD (ETL) JOBS

A computer receives one or more files having configuration information that includes data that defines a plurality of stages of an extract, transform, and load (ETL) job, wherein the plurality of stages comprise a read stage that is preceded by a write stage, and wherein the read stage reads data from a source location, and wherein the data that is read or a modified version of the data that is read is being written by the write stage that writes data to the source location. The computer replaces the read stage with a decompressor stage. The computer replaces the write stage with a compressor stage. The computer executes the decompressor stage and compressor stage on a field-programmable gate array that is programmatically customized with data compression and data decompression functionality to enhance the performance of the ETL job. 1. A computer implemented method for accelerating a read stage and a write stage of an extract , transform , and load job comprising:receiving one or more files having configuration information that includes data that defines a plurality of stages of at least one extract, transform, and load (ETL) job, wherein the plurality of stages comprise a read stage that is preceded by a write stage, and wherein the read stage reads data from a source location, and wherein the data that is read or a modified version of the data that is read is being written by the write stage that writes data to the source location;identifying the read stage and the write stage within the one or more files, wherein the read stage and write stage each have a first parameter that represents a name of a stage, and wherein the first parameter is associated with a second parameter that represents a type of stage;replacing the read stage with a decompressor stage that reads data from a hard disk, decompresses the data from the hard disk and passes the data that is decompressed to another stage or to a computer program for further processing, wherein the decompressor ...

DISTRIBUTED, REAL-TIME ONLINE ANALYTICAL PROCESSING (OLAP)

Source data of an event stream is parsed and supplemented with additional data from reference data sources, producing an enriched event stream from the parsed event stream data. The data records of the enriched event stream are partitioned into data fields designated as a dimension partition and a metric partition, which are partitioned into sub-dimension projections mapped to a plurality of storage keys, such that each of the storage keys includes one or more placeholder wildcard values and each of the storage keys is stored into a database of the computer system by the computer processor. The stored storage keys are then aggregated onto a two-dimensional coordinate vector such that, if the computer processor identifies a permuted storage key having metric values for which a corresponding storage key already exists in the database, then the computer processor aggregates the metric values of the identified storage key with the metric values of the corresponding storage key, and if the computer processor does not identify the permuted storage key as having a corresponding storage key that already exists in the database, then the computer processor writes the metric values of the permuted storage key into the database, comprising initial values for the key combination of dimension values. 1. A method of online analytical processing (OLAP) of a computer system , the method comprising:enriching a data record read from a source database of the computer system by a computer processor with one or more enriched data fields comprising data read from a reference database by the computer processor, wherein the data record includes multiple data fields, such that the enriched data record includes one or more of the multiple data fields of the data record and one or more of the enriched data fields;partitioning the multiple data fields of the enriched data record into a dimension partition and a metric partition;permuting the dimension partition and the metric partition of the ...

METADATA MANAGER FOR ANALYTICS SYSTEM

A metadata management system receives metadata changes and automatically updates a metadata architecture which maps the data. The metadata changes may be received through a simple user interface by a user or administrator. Once received, the system may automatically update schemas and data transformation code to process data according to the new data mapping preference. The system may handle metadata updates in a multi-tenant system having one or more applications per tenant, and may update data for a single tenant and 1 or more tenant applications in a multitenancy. 1. A method for managing metadata , the method comprising:maintaining data in a data collection store, wherein the data in the data collection store corresponds to a first format associated with a tenant application at a client device;receiving a portion selection over a computer network interface from the client device, wherein the portion selection identifies at least one portion of the maintained data and is selected via a user interface of the client device;transforming the at least one portion of the maintained data from the first format into a second format according to the received portion selection, wherein the transformed data in the second format is stored in a data repository according to a schema that associates an object name to the second format data stored in the data repository; andperforming analytics on the data in the second format that is associated with the object name, wherein results of the analytics are output for display on the user device.2. The method of claim 1 , wherein the received portion selection include changes in metadata selections.3. The method of claim 2 , wherein the transformation of the at least one portion from the first format to the second format is performed in accordance with the changes in the metadata selections.4. The method of claim 1 , wherein the transformation of the at least one portion from the first format to the second format includes splitting ...

DATA MANAGEMENT METHOD, DATA MANAGEMENT DEVICE AND STORAGE MEDIUM

A data management method employing the results of an analysis of data stored in a storage unit of a computer provided with a processor and a storage unit, wherein the computer generates an analysis data set by selecting data stored in the storage unit, subjects the analysis data set to prescribed data mining, extracts a model from the analysis data set, converts the model into a relational table, and associates the relational table with a dimension table and a history table that have been stored in advance in the storage unit. 1. A data management method using results of analyzing data stored in a storage module by a computer comprising a processor and the storage module , the data management method comprising:a first step of selecting, by the computer, data stored in the storage module, and generating, a data set for analysis;a second step of performing, by the computer, prescribed data mining on the data set for analysis, and extracting, a model from the data set for analysis;a third step of converting, by the computer, the model to a relational table; anda fourth step of associating, by the computer, with a dimension table and a history table stored in advance in the storage module in association with the relational table.2. The data management method according to claim 1 , wherein claim 1 , in the second step claim 1 , either a decision tree or clustering is executed as the data mining claim 1 , and the model is extracted from the decision tree and clustering results.3. The data management method according to claim 2 ,wherein, in the clustering, specific attributes of the data set for analysis are separated into clusters on the basis of distances between data points, andwherein, in the third step, a tree structure is converted to SQL on the basis of results of separating the data points into clusters to generate the relational table.4. The data management method according to claim 2 ,wherein the decision tree extracts a model that can predict specific attributes ...

Application aware input/output fencing

Disclosed herein are methods, systems, and processes to perform application aware input/output (I/O) fencing operations. Performing such an application aware I/O fencing operation includes installing an identifier that identifies an instance of an application with a node on which the instance of the application is executing, on coordination points. A weight assigned to the instance of the application is determined, and the instance of the application is terminated based on the weight.

APPLICATION AWARE INPUT/OUTPUT FENCING

Disclosed herein are methods, systems, and processes to perform application aware input/output (I/O) fencing operations. A determination is made that a cluster has been partitioned. The cluster includes multiple nodes. As a result of the partitioning, the nodes are split between a first network partition with a first set of nodes and a second network partition with a second set of nodes. Another determination is made that instances of an application are executing on the first set of nodes and the second set of nodes. An application aware I/O fencing operation is then performed that causes termination of instances of the application executing on the first set of nodes or on the second set of nodes. 1. A method comprising: the cluster comprises a plurality of nodes, and', 'as a result of the partitioning, the plurality of nodes are split between a first network partition comprising a first set of nodes of the plurality of nodes and a second network partition comprising a second set of nodes of the plurality of nodes;, 'determining that a cluster has been partitioned, wherein'}determining that a plurality of instances of an application are executing on the first set of nodes and the second set of nodes; and 'the performing the application fencing operation causes termination of instances of the application executing on the first set of nodes or on the second set of nodes.', 'performing an application fencing operation, wherein'}2. The method of claim 1 , further comprising: [ 'the application weight matrix comprises a weight assigned to the application, and', 'accessing an application weight matrix, wherein'}, 'comparing a first total application weight of the first set of nodes and a second total application weight of the second set of nodes., 'performing the fencing race comprises, at least in part,'}, 'performing a fencing race, wherein'}3. The method of claim 2 , further comprising:bypassing the fencing race, if all instances of the application are executing on the ...

System performance logging of complex remote query processor query operations

Described are methods, systems and computer readable media for performance logging of complex query operations.

COMPUTERIZED METHODS AND PROGRAMS FOR INGESTING DATA FROM A RELATIONAL DATABASE INTO A DATA LAKE

A computerized method for ingesting data from a relational database into a data lake is provided, wherein a user-defined function (UDF) is associated with a standard operation of extract, transform, load, or ETL, of an ETL pipeline. This UDF is triggered upon performing the standard operation and thereby allow a code associated with the UDF to be executed. Upon migrating data from one or more data sources into the relational database, the standard operation is executed, which triggers the UDF and, in turn, an execution of the code. As per the execution of this code, an entity running on the data lake is notified that a set of data migrated to the relational database is to be ingested according to given ingestion modalities specified by the code. Finally, the set of data can be ingested into the data lake according to the modalities. Related computer program products are also provided. 1. A computerized method for ingesting data from a relational database into a data lake , the method comprising:providing a relational database, wherein a user-defined function, or UDF, is associated with a standard operation of extract, transform, load, or ETL, of an ETL pipeline, so as for the UDF to be triggered upon performing said standard operation and thereby allow a code associated with said UDF to be executed;migrating data from one or more data sources into the relational database, according to said ETL pipeline, whereby said standard operation is executed so as to trigger said UDF and, in turn, an execution of said code;as per the execution of said code, notifying an entity running on the data lake that a set of data migrated to said relational database is to be ingested according to given ingestion modalities specified by the code; andingesting said set of data into the data lake according to said modalities.2. The computerized method according to claim 1 , wherein whether said set of data may be ingested in parallel;', 'whether said set of data should be appended to data ...

MANAGEMENT OF EXTRACT TRANSFORM LOAD (ETL) JOBS VIA A MODEL THAT IS BASED ON A BAYESIAN NETWORK

A device may receive, from a user device, a request for a set of forecasts of an extract transform load (ETL) completion time for a group of ETL jobs associated with an organization. The device may obtain a set of performance indicators associated with the group of ETL jobs. The device may filter the set of performance indicators using one or more filtering techniques. The device may generate the set of forecasts of the ETL completion time by using a data model to process the set of performance indicators and/or a set of assumptions associated with a set of recommendations for reducing the ETL completion time. The device may provide the set of forecasts of the ETL completion time to the user device. The device may perform one or more actions that cause the ETL system to execute the group of ETL jobs within a threshold completion time. 1. A device , comprising:one or more memories; and [ 'wherein an ETL system is to execute the group of ETL jobs at particular time periods;', 'receive, from a user device, a request for a set of forecasts of an extract transform load (ETL) completion time for a group of ETL jobs associated with an organization,'}, a first subset of performance indicators relating to system performance of the ETL system supporting the group of ETL jobs,', 'a second subset of performance indicators relating to transactional performance of the group of ETL jobs, or', 'a third subset of performance indicators relating to source data generated from the group of ETL jobs;, 'wherein the set of performance indicators include at least one of, 'obtain a set of performance indicators associated with the group of ETL jobs,'}, 'generate, by processing the set of performance indicators, a set of recommendations that are capable of reducing the ETL completion time;', [ historical source data,', 'a set of historical performance indicators,', 'historical ETL completion time data, or', 'historical network modifications data, and, 'wherein the data model has been trained ...

DYNAMIC INCREMENTAL UPDATING OF DATA CUBES

Systems and methods for dynamic incremental updating of online analytical processing (OLAP) data cubes are disclosed. In embodiments, a computer-implemented method, comprises: receiving real-time transactional event data including a plurality of data fields; identifying aggregate computing functions for each of a plurality of standard query language (SQL) queries of a target data cube to be updated, wherein the target data cube is in a multidimensional form for storing OLAP data in the form of numeric values organized in fields; incrementally calculating the aggregate computing functions based on a current aggregated value obtained from an updated aggregated field database, and a value of one of the plurality of data fields; repeating the incremental calculation for each of the plurality of SQL queries to obtain updated field data for the target data cube; and updating the updated aggregated field database with the updated field data. 1. A computer-implemented method , comprising:receiving real-time transactional event data from one or more remote transaction computing devices, the real-time transactional event data including a plurality of data fields containing values;identifying aggregate computing functions for each of a plurality of standard query language (SQL) queries of a target data cube to be updated, wherein the target data cube is in a multidimensional form for storing online analytical processing (OLAP) data in the form of numeric values organized in fields;incrementally calculating the aggregate computing functions based on a current aggregated value obtained from an updated aggregated field database, and a value of one of the plurality of data fields;repeating the incremental calculation for each of the plurality of SQL queries to obtain updated field data for the target data cube; andupdating the updated aggregated field database with the updated field data,wherein the receiving the real-time transactional event data and the incrementally calculating ...

Prediction integration for data management platforms

Techniques are described for integrating prediction capabilities from data management platforms into applications. Implementations employ a data science platform (DSP) that operates in conjunction with a data management solution (e.g., a data hub). The DSP can be used to orchestrate data pipelines using various machine learning (ML) algorithms and/or data preparation functions. The data hub can also provide various orchestration and data pipelining capabilities to receive and handle data from various types of data sources, such as databases, data warehouses, other data storage solutions, internet-of-things (IoT) platforms, social networks, and/or other data sources. In some examples, users such as data engineers and/or others may use the implementations described herein to handle the orchestration of data into a data management platform.

METHOD AND SYSTEM FOR DESIGNING BUSINESS DOMAIN MODEL, DATA WAREHOUSE MODEL AND MAPPING THEREBETWEEN SYNCHRONOUSLY

Method and system for designing a business domain model, a Data Warehouse (DW) model and a mapping therebetween synchronously are provided. The method includes: designing a business domain model based on business logic; designing a DW model based on a structure of a data cube in a DW and designing a mapping between the business domain model and the DW model synchronously; analyzing the business domain model, the DW model and the mapping therebetween based on a predetermined syntax rule; and generating a transactional database corresponding to the business domain model, a DW database corresponding to the DW model, and a mapping between data in the transactional database and in the DW database automatically and synchronously. The system includes a business domain model design unit, a DW model design unit, a mapping design unit, an analysis unit and an automatic generation unit. Mappings are designed more easily, efficiently and accurately. 1. A method for designing a business domain model , a data warehouse model and a mapping therebetween synchronously , comprising:designing a business domain model based on business logic;designing a data warehouse model based on a structure of a data cube in a data warehouse and designing a mapping relation between the business domain model and the data warehouse model synchronously;analyzing the business domain model, the data warehouse model and the mapping therebetween based on a predetermined syntax rule; andgenerating a transactional database corresponding to the business domain model, a data warehouse database corresponding to the data warehouse model, and a mapping between data in the transactional database and data in the data warehouse database automatically and synchronously.2. The method according to claim 1 , wherein the business domain model comprises at least one object claim 1 , each of the at least one object comprising at least one object element; the data warehouse model comprises one fact table and at least one ...

Methodology supported business intelligence (BI) software and system

The disclosed device provides idealized and reusable data source interfaces. The process of idealizing includes reengineering of an original data model using a surrogate key based model. The technique emphasizes readability and performance of the resulting operational data store. In, addition, the disclosed device provides a unique method for handling changes which allows for all types of changes to be automatically implemented in the operational data store by table conversion. Further the disclosed device provides inline materialization which supports a continuous data flow dependency chain. A continuous dependency chain is used to provide automated documentation as well as a dynamic paralleled transformation process. 1. A method of ensuring consistency between a configured product repository and a destination operational data store when changes to one or more configurations occurs , said method comprising the steps of:creating and maintaining a static reference model further comprising a storing of object information in one or more object extended properties in said operational data store; on a table level, at least one extended property containing a data source table; on a column level, at least one extended property per column created using a primary surrogate key having a static standardized value, a foreign surrogate key having a value of a corresponding external foreign key name, and ordinary columns having a corresponding data source column name; andcomparing one or more repository configurations and definitions with one or more extended properties in said static reference model.2. The method of claim 1 , wherein said comparing of one or more repository configurations and definitions further comprises extracting definitions from said repository and producing a first intermediate internal table claim 1 , extracting definitions from said operational data store and producing a second intermediate internal table claim 1 , comparing said first and said second ...

Forecasting System and Method Using Change Data Based Database Storage for Efficient ASP and Web Application

In one embodiment, a system is provided. The system includes a first client. The system also includes an analysis server coupled to the first client. The system further includes a first customer database of information coupled to the analysis server. The first customer database is to embody forecast data and to receive essentially real-time updates to the forecast data. The first customer database supports an OLAP cube associated with the analysis server. 1. A system comprising:an analysis server adapted for coupling to at least an external first client and incorporating an analytics and aggregation unit to analyze and update the information in a first customer database; anda first database of information coupled to the analysis server, the first database embodying forecast data and receiving updates to the forecast data, the analytics and aggregation unit operable to analyze and update the information of at least a first partition of the first database.2. The system of claim 1 , further comprising the at least an external first client.3. The system of claim 1 , wherein the database comprises a customer database storing forecast data for a first customer.4. The system of claim 3 , wherein the database comprises a customer database storing forecast data for a second customer.5. The system of claim 1 , wherein the database is a database shared by a plurality of customers claim 1 , including for a first customer and a second customer.6. The system of claim 1 , wherein the change partition is aggregated by the analysis server.7. The system of claim 1 , wherein the database includes a first partition and a second partition claim 1 , the first partition including change data and the second partition including general data.8. The system of claim 1 , wherein updating the database includes both adding or changing data and calculating the effects of that added or changed data.9. The system of claim 1 , wherein the first partition containing change data may be calculated ...

LOAD PLAN GENERATION

Load Plan Generator (LPG) is a BIAPPS utility for generating ODI load plans based on desired subset of fact tables for loading BIAPPS Data Warehouse. The tool simplifies the configurations process by minimizing the manual steps and configurations and provides a guided list of configurations steps and checklists. The load plan components are basically different sets of load plans that will be stitched together by the load plan generator to create one load plan for loading chosen fact groups in the warehouse sourcing from different transaction systems. 1. A method for generating load plans used to load data from data sources into data warehouses , the method comprising:receiving, at one or more computer systems, one or more data source definitions each specifying one or more data sources from which to load data into a data warehouse;receiving, at the one or more computer systems, information indicative of one or more phases for loading data between data sources and data warehouses; and determining, with one or more processors associated with the one or more computer systems, how to configure the one or more phases with one or more of a plurality of predefined load plan components based on the one or more data sources of the data source definition satisfying one or more design dependencies, each of the plurality of predefined load plan components specifying one or more tasks indicative of how data is loaded between a data source and a data warehouse, and', 'determining, with the one or more processors associated with the one or more computer systems, how to configure the one or more of the plurality of predefined load plan components based on satisfying one or more runtime dependencies between the one or more tasks in the one or more of the plurality of predefined load plan components; and, 'for each data source definition in the one or more data source definitionsgenerating, with the one or more processors associated with the one or more computer systems, a load plan ...

DATA LINEAGE NOTIFICATION TOOLS

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph. 2. The receiver of further configured to:receive an election to receive a notification of a first change of data lineage information; andreceive an election not to receive a notification of a second change of data lineage information.3. The receiver of wherein the change of data lineage information comprises a change in the System of Record (“SOR”).4. The receiver of further configured to:receive an election to receive a notification of a change of data lineage information resulting from a first entity; andreceive an election to not receive a notification of a change of data lineage information resulting from a second entity.5. The apparatus of wherein the change of data lineage information corresponds to a change in data lineage information of one or more TDEs associated with the business element.6. The apparatus of wherein the change of data lineage information corresponds to a change in data lineage information of one or more TDEs associated with a TDE identifier.7. The apparatus of further comprising:the transmitter further ...

DATA LINEAGE TRANSFORMATION ANALYSIS

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph. 1. Apparatus for temporal data lineage analysis comprising:machine readable memory configured to store a technical data element identifier (“TDEI”); 'a query for data lineage information corresponding to a business element identifier;', 'a receiver configured to receive identify data lineage information for each TDEI corresponding to the business element identifier;', 'identify in the data lineage information a first graph of data lineage information corresponding to a first date; and', 'identify in the data lineage information for a second graph of data lineage information corresponding to a second date; and, 'a processor configured to 'toggle between the first graph of data lineage information and the second graph of data lineage information.', 'an electronic switch configured to2. The apparatus of wherein the TDEI is a Key Data Element identifier (“KDEI”).3. The apparatus of claim wherein the data lineage information comprises historical data lineage information.4. The apparatus of wherein the processor is further configured ...

MULTIMEDIA INSPECTION DATABASE SYSTEM (MIDaS) FOR DYNAMIC RUN-TIME DATA EVALUATION

An object-oriented system provides a comprehensive and integrated tool to manage, operate, analyze and inspect structures by providing 3-D visualization of the structures, a database interface, connectivity with one or more databases, a data display and analysis capabilities. The structural model is stored as an object-oriented, serialized file as a series of objects, including primitives, coordinates, object names, group names and other object identifiers. The system integrates various types of data, including tabulated textual data, annotated engineering drawings, photographic records, graphical plots, audio and videotaped records, from different tables and databases. Data are displayed to the user based on queries that the user submits to the system, which queries are evaluated at run-time. As a result, the user achieves unparalleled flexibility in data analysis and evaluation. 181-. (canceled)82. A non-transitory computer-readable storage medium containing instructions that , receiving a data retrieval request from a user;', 'retrieving live data responsive to the data retrieval request directly from a plurality of source databases, at least two of which are incompatible with one another;', 'assembling an OLAP cube using the retrieved live data; and', 'displaying a representation of the assembled OLAP cube to the user., 'when executed, cause one or more computers to perform operations comprising Priority is claimed from U.S. Provisional Patent Application 60/219,186 filed Jul. 19, 2000. The provisional application, in its entirety, is hereby incorporated by reference.This invention relates to information systems. Specifically, this invention relates to inspection information systems for evaluating structures that provide data query and update capabilities.The high cost of analyzing and inspecting structures on a regular basis to ensure structural integrity has plagued industries that rely on the soundness of these structures. Structures such as buildings, oil ...

AUTOMATIC VERIFICATION OF GRAPHIC RENDITION OF JSON DATA

Aspects provide automatic verification of graphic rendition of JavaScript Object Notation (JSON) data by using a baseline JSON file to render data values of a baseline data set into a baseline graphic presentation that meets a boundary condition requirement for displaying the data values. The baseline data set data values are rendered into a baseline graphic presentation that is stored into a local file system and used to render data values of a second set of data of a back end server data warehouse data into a new graphic presentation, which is compared to the stored baseline graphic presentation on a pixel-by-pixel basis. In response to the pixel-by-pixel comparison indicating a difference in display of boundary condition requirement values, the stored baseline JSON file is revised into a revised baseline JSON file that renders the data values of the second data set into a revised new graphic presentation. 1. A computer-implemented method for automatic verification of graphic rendition of JavaScript Object Notation (JSON) data , the method comprising executing on a processor the steps of:using a baseline JSON file to render data values of a baseline data set into a baseline graphic presentation that meets a boundary condition requirement for displaying the data values, wherein a local reverse proxy server that is disposed in a network communication structure between a front end computer device and a back end server, in response to a request from the front end computer device, retrieves the baseline JSON file from the local file system and the data values of the baseline data set from the back end server, and wherein the front end computer device renders the data values of the baseline set of data into the baseline graphic presentation;storing the baseline graphic presentation in the local file system;using the stored baseline JSON file to render in the front end device data values of a second set of data of the back end server data warehouse data into a new ...

INCREASING STORAGE SPACE FOR PROCESSES IMPACTING DATA STORAGE SYSTEMS

A processing request is received. The processing request includes information about a first location where a set of data is stored and information about a second location where the set of data is to be transferred. The size of the set of data is determined. The size of the available portion of the second location is determined. If the size of the set of data is smaller than the size of the available portion of the second location is determined. Responsive to determining the size of the set of data is larger than the size of the available portion of the second location, the size of the available portion is requested to be increased. The size of the available portion of the second location is increased to a size larger than the determined size of the set of data. 1. A method for managing data storage , the method comprising the steps of:receiving, by one or more computer processors, a processing request, wherein the processing request includes information about a first location where a set of data is stored and information about a second location where the set of data is to be transferred;determining, by one or more computer processors, a size of the set of data;determining, by one or more computer processors, a size of a available portion of the second location;determining, by one or more computer processors, if the size of the set of data is smaller than the size of the available portion of the second location; andresponsive to determining the size of the set of data is larger than the size of the available portion of the second location, requesting, by one or more computer processors, the size of the available portion of the second location be increased, wherein the size of the available portion of the second location is increased to a size larger than the determined size of the set of data.2. The method of claim 1 , further comprising:responsive to receiving an indication that the size of the available portion of the second location has increased, initiating, by ...

OPTIMIZING INCREMENTAL LOADING OF WAREHOUSE DATA

In various embodiments, a data integration system is disclosed which enables incremental loads into a data warehouse by developing a data partitioning plan and selectively disabling and enabling indexes to facilitate incremental loads into fact tables. 1. A method , comprising:receiving, by a computing system, a data dictionary that specifies a structure of data to be loaded into a data warehouse;determining, by the computing system, a plurality of partitions using the data dictionary;identifying, by the computing system, data to be incrementally loaded into the data warehouse;determining, by the computing system, which partitions of the plurality of partitions are affected by the data to be incrementally loaded into the data warehouse; andselectively disabling one or more indexes on each affected partition of the plurality of partitions in order to load the data into the data warehouse.2. The method of claim 1 , wherein determining the plurality of partitions using the data dictionary comprises determining one or more sub-partitions of a partition of the plurality of partitions.3. The method of claim 1 , wherein determining which partitions of the plurality of partitions are affected by the data comprises determining which partitions comprise new data.4. The method of claim 1 , wherein determining which partitions of the plurality of partitions are affected by the data comprises determining which partitions have changed data.5. The method of claim 1 , wherein selectively disabling the one or more indexes on each affected partition of the plurality of partitions in order to load the data into the data warehouse comprises disabling bitmap indexes on affected partitions.6. The method of claim 1 , further comprising executing a query to load a working table from a staging table prior to determining which partitions are affected by the data.7. The method of claim 6 , wherein the affected partitions are stored in a fact table claim 6 , and further comprising merging the ...

COMPUTER DATA SYSTEM DATA SOURCE REFRESHING USING AN UPDATE PROPAGATION GRAPH

Described are methods, systems and computer readable media for data source refreshing. 1. A system for updating a data object using an update propagation graph , the system comprising: determining a logical clock has transitioned to an updating state;', invoking a data source refresh method for a data source for which changes are being processed;', 'determining whether a priority queue for the data source is empty;', 'when the priority queue is not empty, retrieving a next change notification message from the priority queue and delivering the change notification to a corresponding data source and repeating determining whether the priority is queue is empty; and', 'when the priority queue is empty, setting the logical clock to an idle state., 'processing one or more changes to one or more corresponding data sources, wherein the changes are processed according to an order determined by an update propagation graph (UPG) having one or more nodes each corresponding to one of the data sources, wherein change processing for each data source includes], 'one or more hardware processors coupled to a nontransitory computer readable medium having stored thereon software instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including2. The system of claim 1 , wherein the change notifications can include one of a data add notification claim 1 , a data modify notification claim 1 , a data delete notification and a data reindex notification.3. The system of claim 1 , wherein each node of the UPG includes one of a dynamic node claim 1 , a static node and an internal node.4. The system of claim 1 , wherein the UPG is generated in a context of a remote query processor.5. The system of claim 4 , wherein a structure of the UPG is updated in response to a query being executed by the remote query processor.6. The system of claim 1 , wherein the operations further include adding an additional notification to the priority ...

METHODS AND SYSTEMS FOR MULTI-DYNAMIC DATA RETRIEVAL AND DATA DISBURSEMENT

A device includes circuitry configured to provide a configurable platform including a rules-based processing engine, access and manipulate a plurality of configurable databases, retrieve first data from one of the configurable databases, register the first data for task programs, authenticate the first data according to authenticity parameters, process the first data against processing rules, identify and configure each task program when the assessment of the first data satisfies predetermined criteria, measure the ETL data load flow against a predetermined performance threshold, route the ETL data load flow to a database processing engine, and output a data disbursement of results when the first data is authenticated and the predetermined performance threshold has been satisfied by the rules-based processing engine. In one aspect, such implementation can increase data search and retrieval times as well as outputting a more thorough and accurate count of eligible end-user services. 1. A device comprising: provide a configurable platform including a rules-based processing engine configured to perform bulk and parallel processing of an extract-transform-load (ETL) data load flow,', 'access and manipulate, in response to a request from a remote device, a plurality of configurable databases,', 'retrieve first data from one of the configurable databases,', 'register the first data for one or more task programs of the configurable platform,', 'authenticate the first data according to one or more authenticity parameters,', 'process the first data against processing rules of the rules-based processing engine to generate an assessment of the first data based on the processing rules,', 'identify and configure each task program when the assessment of the first data satisfies predetermined criteria according to the processing rules,', 'measure the ETL data load flow against a predetermined performance threshold,', 'route the ETL data load flow to a database processing engine ...

METHOD, APPARATUS, AND COMPUTER-READABLE MEDIUM FOR DATA TRANSFORMATION PIPELINE OPTIMIZATION

A system, method and computer-readable medium for data transformation pipeline optimization including storing a transformation pipeline comprising data transformation components having associated input buffers and associated output buffers and being configured to apply data transformations to data in the associated input buffers and output results of the data transformations to the associated output buffers, receiving a data set comprising data fields, identifying satellite data fields for at least one transformation component, each satellite data field comprising data that is not utilized during the data transformations of the transformation component, and processing, by the data transformation engine, the data set through each transformation component in the transformation pipeline, the processing including removing satellite data fields from the input buffers, linking the removed satellite data fields to the remaining data in the input buffers, and applying the data transformations to data in input buffers and writing results to output buffers. 1. A method executed by a data transformation engine on one or more computing devices for data transformation pipeline optimization , the method comprising:storing, by the data transformation engine, a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;receiving, by the data transformation engine, a data set comprising a plurality of data fields;identifying, by the data transformation engine, one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components, each satellite ...

Systems and Methods for Providing Performance Metadata in Interest-Driven Business Intelligence Systems

Interest-driven business intelligence server systems that provide performance metadata are described. In the disclosed embodiments, an interest-driven business intelligence server system receives a report specification. The report specification includes at least reporting data requirement. The interest-driven business intelligence server determines performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based on the report specification. The performance metadata information for the interest-driven data pipeline is transmitted to an interest-driven user visualization system by the interest-driven business intelligence server system. 1. An interest-driven business intelligence server system , comprising:a processor;a memory connected to the processor and configured to store an interest-driven business intelligence application;raw data storage configured to store raw data, where a portion of the raw data comprises unstructured data;metadata storage configured to store data description metadata describing the raw data; andreport storage configured to store previously generated reports comprising previous reporting data and previous reporting data requirements; receive a report specification, where the report specification comprises at least one reporting data requirement;', 'determine performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based on the report specification; and', 'provide the performance metadata information for the interest-driven data pipeline to an interest-driven user visualization system., 'wherein the interest-driven business intelligence application configures the processor to2. The system of claim 1 , wherein the performance metadata information for interest-driven data pipeline provided to the interest-driven user visualization system includes performance metadata for at least one of the raw data claim 1 , the reporting data claim ...

VERIFICATION OF TRANSFORMED CONTENT

A computer manages methods for determining accurate document transformation by rendering the source document into a non-rasterized format, where the non-rasterized format is a rendered source document. The computer rendering the target document into a non-rasterized format, where the non-rasterized format is a rendered target document. The computer comparing one or more aspects of the rendered source document to corresponding one or more aspects of the rendered target document. The computer determining, based, at least in part, on the compared one or more aspects, whether or not the source document was accurately transformed to the target document. 1. A method for determining accurate document transformation , the method comprising:queuing, by one or more processors, a source document for transformation to a target document according to an assigned priority for the target document, wherein the assigned priority is based on a descending order of file sizes for target documents of queued source documents;responsive to determining the source document is next in the queue, transforming, by one or more processors, content of the source document into the target document, wherein transforming the content of the source document includes transforming the content from the first format to the second format;responsive to determining an increased likelihood the source document was not accurately transformed to the target document based on a file size for the target document, rendering, by one or more processors, the source document into a non-rasterized graphical format, wherein a rendered source document represents the source document in the non-rasterized graphical format; anddetermining, by one or more processors, based at least in part on a comparison of one or more aspects of the rendered source document and a rendered target document in a non-rasterized graphical format, the source document was accurately transformed to the target document.2. The method of claim 1 , ...

DATA QUALITY MONITORING

A computer implemented method, computer program product and system for data quality monitoring includes measuring a data quality of loaded data relative to a predefined data quality metric. The measuring the data quality includes identifying delta changes in at least one of the loaded data and the data quality rules relative to a previous measurement of the data quality of the loaded data. Logical calculus defined in the data quality rules is applied to the identified delta changes. 1. A computer-implemented method for data quality monitoring , the method comprising: identifying delta changes in the loaded data and data quality rules relative to a previous measurement of the data quality of the loaded data;', 'applying a logical calculus defined in the data quality rules to the identified delta changes;', 'decomposing a selected quality rule into single terms and projecting the terms onto a tree structure with each node of the tree structure corresponding to one individual term of the selected quality rule; and', 'applying each individual data column to the tree structure resulting in an intermediary column cache for each node containing an intermediate result of a selected operation., 'measuring, by a computer, a data quality of loaded data relative to a predefined data quality metric, wherein the loaded data is loaded from a data source that includes a data set comprising a plurality of elements, the measuring comprising2. The computer-implemented method of further comprising:updating, based on the delta change of the loaded data, the intermediary column cache of a node based on determining that the delta change of the loaded data affects content of the intermediary column cache of said node.3. The computer-implemented method of further comprising:updating, based on the delta change of the data quality rule, the tree structure with the updated data quality rule and updating the intermediary column cache based on determining that the delta change of the data ...

SYSTEMS AND METHODS FOR CONVERTING DATA FROM LEGACY COMPUTER SYSTEMS INTO TARGET FORMATS FOR TARGET COMPUTER SYSTEMS

A computer system for converting data from one of a plurality of different legacy systems to one of a plurality of different target systems includes one or more processors, memory, a database stored in the memory, and a software framework stored in the memory for execution by the one or more processors. Each of the plurality of different legacy systems has a legacy data format and each of the plurality of different target systems has a target data format. The software framework includes a plurality of software components callable by an output adaptor for performing a plurality of data conversion functions. The software framework is configured to interact with each of the plurality of different legacy systems having the legacy data format and/or each of the plurality of different target systems having the target data format. Other example computer systems and methods are also disclosed. 19-. (canceled)10. A computer system for converting data from a legacy system having a legacy data format to a target system having a target data format , the computer system comprising:one or more processors;memory;a database stored in the memory;a software framework stored in the memory for execution by the one or more processors, the software framework including a plurality of software components for performing a plurality of data conversion functions;an input adaptor stored in the memory for execution by the one or more processors, the input adaptor configured to store data from the legacy system in the database of the computer system; andan output adaptor stored in the memory for execution by the one or more processors, the output adaptor configured to selectively call the plurality of software components of the software framework to convert the legacy system data stored in the database into a target format and provide the data converted into the target format to the target system,wherein the software framework stored in the memory is configured to interact with the legacy system ...

OUTPUT DRIVEN GENERATION OF A COMBINED SCHEMA FROM A PLURALITY OF INPUT DATA SCHEMAS

A computational device receives a plurality of versions of an input data schema. At least one element is selected from the plurality of versions of the input data schema based on an expected result. A combined schema is generated based on the at least one selected element. The input data is processed according to the combined schema. 1. A method , comprising:receiving, via a computational device, a plurality of versions of an input data schema;selecting at least one element from the plurality of versions of the input data schema based on an expected result; andgenerating a combined schema based on the at least one selected element; andprocessing input data according to the combined schema.2. The method of claim 1 , further comprising:generating streaming events based on the combined schema.3. The method of claim 2 , wherein the streaming events correspond to tokens generated by a parser from the input data and the combined schema corresponds to a state machine claim 2 , and wherein processing input data according to the combined schema further comprises:sending the streaming events from the parser to the state machine to validate incoming data and generate an output.4. The method of claim 1 , wherein the generating of the combined schema is performed by an Extract Transform Load (ETL) component that executes in the computational device claim 1 , and wherein the ETL component extracts information from the input data claim 1 , transforms the information in conformance with operational needs claim 1 , and generates an output.5. The method of claim 1 , wherein the at least one selected element is a first element claim 1 , and wherein at least one of the plurality of versions of the input data schema include a second element that is not a part of the expected result claim 1 , the method further comprising: omitting the second element from the combined schema.6. The method of claim 1 , wherein the generating of the combined schema based on the at least one selected ...

METHOD OF CROSS REFERENCING INFORMATION RELATING TO A MOBILE OPERATOR AND INFORMATION RELATING TO A THIRD-PARTY BASE AND SERVER ADAPTED FOR IMPLEMENTING THIS METHOD

A method is provided for cross referencing information relating to communication service operations of an operator implementing mobile terminals with data stored in a third-party base relating to the management of the mobile terminals. The method includes a replication of the third-party database, a detection of changes that have occurred in the third-party database, and a production of job-related rules pertaining to the terminals so as to decide a real-time deactivation/reactivation of the service.

RELATIONAL DATABASE ONLINE STORAGE MANAGER

A mechanism for managing the growth of databases allows the user to drop tables containing less active data without actually losing access to the data that existed within the table. The storage manager takes the form of a proxy that is configured to reside in the stream between the database client application and the database server. The storage manager acts by intercepting the DROP TABLE command within the SQL and acts to archive the contents of the table prior to passing the command on to the database server for normal execution. Subsequent database queries are monitored for references to tables that have been previously archived. When needed, the storage manager can restage the data from the archive and recreate the table that had been previously dropped in the same form as it had existed prior to that DROP TABLE command execution. 1. A system for database management comprising:a storage manager configured to receive queries of a database from a database client; anda data repository configured to send and receive tables from the storage manager, whereinwhen the storage manager receives, from the database client, a request to delete a table from the database, either through a command to drop a table from the database using common database syntax or through extension of database language by addition of data staging commands, the storage manager retrieves the table from the database and stores the table in a storage manager; andwhen the storage manager receives a query made by a database client to access data in the table dropped from the database, the storage manager restores the table to the database before forwarding the query thereto.2. The system of claim 1 , wherein the storage manager includes a query proxy engine for handling the queries claim 1 , and a file manager engine claim 1 , for interacting with the data repository.3. The system of claim 2 , wherein the query proxy engine includes a query table search routine claim 2 , a table exception handler and a ...

COMPUTER IMPLEMENTED METHOD FOR CREATING DATABASE STRUCTURES WITHOUT KNOWLEDGE ON FUNCTIONING OF RELATIONAL DATABASE SYSTEM

Computer implemented methods for storing ad hoc relations between previously unrelated database objects assigned to different database structures using an electronic computing device are presented, the methods including: causing the electronic computing device to define at least three database structures: structure A, structure B and structure C, where each of the at least three database structures each includes a number of objects, where there exists at least one relation between objects of structure A and objects of structure B, and where there exists at least one relation between objects of structure B and objects of structure C; filtering data of structure A; accessing structure B using a first selected relation between structure A and structure B; storing information about filtering of structure A and information on a first selected path between structure A and structure B; filtering results obtained from structure B. 118.- (canceled)19. A computer implemented method for storing ad hoc relations between previously unrelated database objects assigned to different data sets using an electronic computing device , the method comprising:creating, with aid of the electronic computing device, at least three data sets: data set A, data set B and data set C, wherein each of the at least three data sets includes a plurality of objects defined by a plurality of core data structures, wherein objects of data set A and objects of data set B have at least one relation, wherein objects of data set B and objects of data set C have at least one relation, and wherein the at least one relation between objects of data set A and objects of data set B and the at least one relation between objects of data set B and objects of data set C are generated based on a mind map;filtering the objects of data set A;accessing data set B using a first selected relation between data set A and data set B;storing information about filtering of data set A and information on a first selected path ...

INFORMATION EXTRACTION APPARATUS, INFORMATION EXTRACTION METHOD, AND INFORMATION EXTRACTION PROGRAM

An information extraction apparatus includes, a controller extracting, as variable elements, parts different between a plurality of structured documents and extract, as peripheral information pieces, elements within a predetermined range from the respective variable elements, and a storage unit storing the variable elements and the peripheral information pieces with respect to at least an extraction object which is at least one of the variable elements. The controller re-extracts variable elements and peripheral information pieces from the plurality of structured documents, calculates similarities of the variable elements and the peripheral information pieces between before and after the re-extraction, on the basis of (i) the re-extracted variable elements and the re-extracted peripheral information pieces and (ii) the variable elements and the peripheral information pieces stored in the storage unit, and identifies, out of the re-extracted variable elements, a variable element corresponding to the extraction object, on the basis of the calculated similarities. 1. An information extraction apparatus comprising:a controller configured to acquire a plurality of structured documents and extract, as variable elements, parts different between the acquired documents and extract, as peripheral information pieces, elements within a predetermined range from the respective variable elements; anda storage unit configured to store the variable elements and the peripheral information pieces with respect to at least an extraction object which is at least one of the variable elements, re-acquires the plurality of structured documents and re-extracts, as variable elements, parts different between the re-acquired documents and re-extracts, as peripheral information pieces, elements within the predetermined range from the respective re-extracted variable elements,', 'calculates similarities of the variable elements and the peripheral information pieces between before and after the re ...

CODE LINEAGE TOOL

A code lineage tool includes a scanner, a parser, and an integrator. The scanner identifies elements in ETL code. The parser determines that a first element is affected by a second element and adds the first element to a parse tree and the second element to the parse tree as a sub-node of the first element. The parser determines that the second element is affected by a third element and adds the third element to the parse tree as a sub-node of the second element. The parser determines that a fourth element does not affect the first element and prevents the fourth element from being added to the parse tree or removes the fourth element from the parse tree. The integrator determines that a change to the third element will change the value of the first element and generates a lineage for the first element. 1. A code lineage tool comprising:a scanner configured to identify a plurality of elements in extract, transform, load (ETL) software code by scanning the ETL software code; determine, based on a stored grammar file, that a value of a first element of the plurality of elements is affected by a second element of the plurality of elements;', 'add the first element to a parse tree;', 'add the second element to the parse tree as a sub-node of the first element;', 'determine, based on the stored grammar file, that a value of the second element is affected by a third element of the plurality of elements;', 'add the third element to the parse tree as a sub-node of the second element;', 'determine, that a fourth element of the plurality of elements does not affect the value of the first element because the second element affects the value of the first element; and', 'in response to the determination that the fourth element does not affect the value of the first element, perform at least one of preventing the fourth element from being added to the parse tree and removing the fourth element from the parse tree; and, 'a parser configured to determine, based on the parse tree, ...

TERM EXTRACTION METHOD AND APPARATUS

The present disclosure provides term extraction methods and apparatuses. One exemplary method comprises: acquiring description information of a network resource; performing an explicit-term extraction procedure on the description information to extract an explicit term from the description information; and performing a mode-term extraction procedure on the description information to extract an implicit term from the description information. Based on the technical solution of the present disclosure, both explicit terms that are easily discoverable and implicit terms that are not easily discoverable can be automatically extracted from the description information. The extraction can be more comprehensive, and the extraction quality can be improved. 1. A term extraction method , comprising:acquiring description information of a network resource;performing an explicit-term extraction procedure on the description information to extract an explicit term from the description information; andperforming a mode-term extraction procedure on the description information to extract an implicit term from the description information.2. The method according to claim 1 , wherein acquiring description information of a network resource comprises:preprocessing original description information of the network resource.3. The method according to claim 2 , wherein preprocessing the original description information comprises performing at least one of the following on the original description information: connecting-symbol retention processing claim 2 , case conversion processing claim 2 , spelling consistency check processing claim 2 , word segmentation processing claim 2 , spelling correction processing claim 2 , or noun lemmatization processing.4. The method according to claim 1 , wherein performing the explicit-term extraction procedure on the description information to extract the explicit term from the description information comprises:loading a preset explicit term rule; andextracting ...

Data processing system

A data processing system according to an embodiment of the present invention may manage one or more tables and a plurality of archive files including one or more records extracted from the table. Upon receiving a search request for the table, the data processing system may generate a query (first partial query) to search for a record from the table corresponding to a condition specified by the search request, and generate a query (second partial query) to identify the archive file including the record extracted from the table specified as a search target in the search request, and search the identified archive file for the record corresponding to the condition specified by the search request. A query for deriving a union of the output results of the first partial query and the second partial query may be generated, and processing related to the generated query may be executed in parallel.

OLAP Execution Model Using Relational Operations

In one embodiment the present invention includes an OLAP execution model using relational operations. In one embodiment, the present invention includes, a method comprising receiving a first query in an online analytic processor (OLAP) executing on one or more computers, the OLAP generating and comprising a model specifying a graph defining a plurality of nodes and a plurality of tiers, each node corresponding to a different operation on data. A second query is generated by the OLAP. The second query includes a plurality of layered subqueries each corresponding to one of the nodes in the graph for specifying the different operations on data. The second query is received in a relational engine coupled to the datastore. The relational engine executes the second query, and in accordance therewith, retrieves data. 1. A method comprising:receiving a first query in an online analytic processor (OLAP) executing on one or more computers;generating a model, the model specifying a graph defining a plurality of nodes and a plurality of tiers, each node corresponding to a different operation on data;generating a second query based on the model, the second query including a plurality of layered subqueries each corresponding to one of the nodes in the graph for specifying the different operations on the data: andreceiving the second query in a relational engine coupled to a datastore, wherein the relational engine executes the second query, and in accordance therewith, retrieves data.2. The method of wherein a first node on a highest tier in the model corresponds to a subquery comprising an exception aggregation claim 1 , the exception aggregation associating a first aggregation type with a first measure and a first dimension of a dimension table claim 1 , the first aggregation type being different than a default aggregation type used on the first measure when aggregating said first measure across other dimensions claim 1 , wherein the second query comprises at least one SQL ...

SYSTEM AND METHOD OF AUTOMATICALLY EXTRACTING DATA FROM PLURALITY OF DATA SOURCES AND LOADING THE SAME TO PLURALITY OF TARGET DATABASES

The present invention discloses system and method for automatically extracting data from plurality of data sources in various formats through source channels and loading data to plurality of target databases through connectors. The system includes a data transformation module for transforming data received from the plurality of data sources, a data processing module for automatically analyzing and organising the received data for loading into the plurality of target databases, and a metadata repository for storing metadata of the processed data for future usage. The metadata regarding data structure of the data sources is automatically extracted from the data sources and used to create predefined data structures of the target databases. The data processing module includes a data input handling module for identifying mime-type, extension and the metadata of the data sources, a data structure identification module for identifying type and subtype of the data sources and a target-data-structure creation module for creating the predefined data structures of the target databases. 1. A system for automatically extracting data from one or more data sources in various formats through one or more source channels and loading data contained therein to one or more target databases through one or more connectors , the system comprising:a data transformation module for transforming data received from the one or more data sources;a data processing module for automatically analyzing and organising the received data for loading into the one or more target databases; anda metadata repository for storing metadata of the processed data for future usage,wherein the metadata regarding data structure of the one or more data sources is automatically extracted from the data sources and used to create predefined data structures of the one or more target databases.2. The system for automatically extracting data from one or more data sources as claimed in claim 1 , wherein the data processing ...

CREATION OF CHANGE-BASED DATA INTEGRATION JOBS

A computer software implemented method for transforming a first extract transform load (ETL) job having at least some unload transform load (UTL) portions. The method includes the following steps: (i) decomposing the first ETL job into an intermediate set of one or more jobs; and (ii) for each job of the intermediate set, transforming the job into a transactionally equivalent job to yield a final set of one or more jobs. The decomposing is performed so that each job of the intermediate jobs set is a Simple UTL job. The transforming is performed so that each job of the final set includes no UTL portions. 1. A method for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion , the method comprising:decomposing the first ETL job into an intermediate set of one or more Simple UTL jobs; andfor each Simple UTL job of the intermediate set, transforming the job into a transformationally equivalent job to yield a final set of one or more jobs that are not UTL jobs;wherein: a single input action retrieves the contents of a single table, or view on a single table, in a database,', 'contains no actions that operate on the entirety of data for a job,', 'contains no actions that rely on ordering of data, and', 'at least one output stage operates on a database., 'each Simple UTL job of the intermediate set jobs meets the following criteria2. The method of wherein:the first ETL job includes at least one sort type action; andthe decomposing includes removing the at least one sort type actions from the first ETL job.3. The method of wherein:the first ETL job includes at least one funnel type action; andthe decomposing includes transforming the at least one funnel type action in the first ETL job into a continuous mode.4. The method of wherein:the first ETL job includes at least one join type action; andthe decomposing includes transforming the at least one join type action in the first ETL job into at least one lookup action.5. ...

MANAGING SPARSITY IN AN MULTIDIMENSIONAL DATA STRUCTURE

Embodiments of the present invention relate to a method, program product, and system for managing the percentage of unpopulated cells in a multidimensional data structure during the servicing of multidimensional analytical queries. In an embodiment, a multidimensional data structure is stored in a memory location of a computing device, wherein the multidimensional data structure includes a dimension that is a subset of a plurality of dimensions included in a relational database. The computing device determines a frequency of utilization of a dimension to service a multidimensional query request. The computing device combines the dimension with the multidimensional data structure responsive to the frequency of utilization. 1. A method comprising:a multidimensional data structure stored in a memory location of a computing device, wherein the multidimensional data structure includes a dimension that is a subset of a plurality of dimensions included in a relational database;determining a frequency of utilization of a dimension to service a multidimensional query request; andcombining the dimension with the multidimensional data structure responsive to the frequency of utilization.2. The method of claim 1 , wherein the multidimensional data structure includes dimensions of a predetermined cardinality.3. The method of claim 2 , wherein the predetermined cardinality is one of the following:lower than a predetermined cardinality threshold value or range;equal to a predetermined cardinality threshold value or range; andhigher than a predetermined cardinality threshold value or range.4. The method of claim 1 , wherein the multidimensional query request is drafted in a query language that is capable of specifying multivariate claim 1 , multilevel claim 1 , and/or multidimensional relationships.5. The method of claim 1 , wherein the multidimensional query request is drafted in a multidimensional expressions query language or multidimensional query language.6. The method of ...

METHOD AND SYSTEM FOR RECONCILING META-DATA IN A DATA WAREHOUSE

A data warehouse system comprising a warehouse database and a data warehouse manager communicatively coupled to the database. The warehouse manager is to store data to and retrieve data from the warehouse database and perform metadata reconciliation to update metadata of objects in the data warehouse to reflect a change in metadata of a first object in the warehouse by identifying a first set of objects impacted by the change to the metadata of the first object, identifying a second set of objects on which the first object depends, identifying at least one task that has to be completed to propagate the impact of the change in the metadata of the first object to objects in the first and second sets of objects, performing the task, and dynamically computing additional tasks based on whether or not definitions of objects in the first or second sets of objects changes based upon completion of the at least one task. 1. A method comprising:receiving, at one or more computer systems, a change in a definition of a first object describing a first portion of data stored in a data warehouse;determining, with one or more processors associated with the one or more computer systems, at least one task to propagate the change in the definition of the first object to metadata of the first object;reconciling, with one or more processors associated with one or more computer systems, lineage of the first object to identify a set of objects that are impacted by the change to the definition of the first object, each object in the second set of objects describing a second portion of the data stored in the data warehouse different from the first portion of the data;determining, with the one or more processors associated with the one or more computer systems, at least one additional task to propagate an impact of the change to metadata of one or more objects in the set of objects that are impacted by the change to the definition of the first object; andrendering, with the one or more ...

Data storage extract, transform and load operations for entity and time-based record generation

An extract, transform and load (ETL) data storage system performs extract, transform and load operations to load target tables with data extracted and transformed data from multiple data sources. Transformations include matching and linking date temporarily stored in intermediate tables by entity across data sources. Data may be organized by entity and time, and analytical records can be generated from the intermediate tables according to variable transforms. The analytical records may be used for predictive analytics.

EFFICIENT QUERY PROCESSING USING HISTOGRAMS IN A COLUMNAR DATABASE

A probabilistic data structure is generated for efficient query processing using a histogram for unsorted data in a column of a columnar database. A bucket range size is determined for multiples buckets of a histogram of a column in a columnar database table. In at least some embodiments, the histogram may be a height-balanced histogram. A probabilistic data structure is generated to indicate for which particular buckets in the histogram there is a data value stored in the data block. When an indication of a query directed to the column for select data is received, the probabilistic data structure for each of the data blocks storing data for the column may be examined to determine particular ones of the data blocks which do not need to be read in order to service the query for the select data. 123.-. (canceled)24. A system , comprising: determine a bucket range size for each of a plurality of buckets for a histogram of a column of a columnar database table, wherein each bucket of the plurality of buckets represents an existence of one or more data values of the data in the column within a range of values according to the determined bucket range size;', 'generate a probabilistic data structure for each of one or more data blocks storing data for the column of the columnar database table, wherein the probabilistic data structure indicates for which particular buckets of the plurality of buckets in the histogram there is a data value stored in the data block; and', 'examine the probabilistic data structure, responsive to a query, for each of the one or more data blocks storing data for the column to determine ones of the one or more data blocks which do not need to be read in order to service the query., 'one or more hardware processors and memory with program instructions to25. The system of claim 24 , wherein to determine the plurality of bucket range sizes for the histogram of the column of the columnar database table claim 24 , the program instructions are ...

Query capabilities of topological data analysis graphs

A method comprises receiving a data set, mapping data points from the data set to a reference space utilizing a lens function, generating a cover of the reference space using a resolution function, clustering the data points mapped to the reference space using the cover and a metric function to determine each node of a plurality of nodes of a graph, generating a graph including the plurality of nodes, the graph including an edge between every two nodes that share at least one data point as a member, and generating first and second data structures, the first data structure identifying membership of each node, the second data structure identifying each edge between each of the two nodes, the second data structure further identifying the nodes that are connected by each edge, the first and second data structure being capable of being queryable using a query language.

Data lineage system

A data lineage system is provided that traces a data lineage of a data warehouse. The data lineage system maps a target data element to one or more source data elements. The data lineage system further stores one or more source surrogate keys within one or more auxiliary columns of a target data record. The data lineage system further stores, for each source data element, a data lineage mapping system record within a data lineage mapping system table that represents the mapping of the target data element and the corresponding source data element. The data lineage system further maps a source data element to one or more target data elements. The system further stores, for each target data element, a shadow system record within a shadow system table that represents the mapping of the source data element and the corresponding target data element.

Method and Apparatus for Accelerated Format Translation of Data in a Delimited Data Format

Various methods and apparatuses are described for performing high speed format translations of incoming data, where the incoming data is arranged in a delimited data format. As an example, the data in the delimited data format can be translated to a fixed field format using pipelined operations. A reconfigurable logic device can be used in exemplary embodiments as a platform for the format translation. 1. A method comprising:a reconfigurable logic device receiving an incoming stream comprising a plurality of bytes arranged in a delimited data format, the incoming byte stream being representative of data arranged in a plurality of fields, the incoming byte stream comprising a plurality of data characters and a plurality of field delimiter characters, the field delimiter characters defining a plurality of boundaries between the fields;the reconfigurable logic device processing the received byte stream to identify the field delimiter characters that are present in the received byte stream; andthe reconfigurable logic device translating the received byte stream to an outgoing byte stream arranged in a fixed field format based on the identified field delimiter characters, the outgoing byte stream comprising a plurality of the data characters of the received byte stream arranged in a plurality of fixed-size fields.2. The method of wherein the incoming byte stream further comprises a plurality of shield characters;wherein the processing step further comprises the reconfigurable logic device identifying the shield characters that are present in the received byte stream; andwherein the translating step further comprises the reconfigurable logic device translating the received byte stream to the outgoing byte stream having the fixed field format based on the identified field delimiter characters and the identified shield characters.3. The method of wherein the translating step comprises the reconfigurable logic device removing the identified field delimiter characters from ...

Systems and Methods for Providing Performance Metadata in Interest-Driven Business Intelligence Systems

Interest-driven business intelligence server systems that provide performance metadata are described. In the disclosed embodiments, an interest-driven business intelligence server system receives a report specification. The report specification includes at least reporting data requirement. The interest-driven business intelligence server determines performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based on the report specification. The performance metadata information for the interest-driven data pipeline is transmitted to an interest-driven user visualization system by the interest-driven business intelligence server system. 1. An interest-driven business intelligence server system , comprising:a processor;a memory connected to the processor and configured to store an interest-driven business intelligence application;raw data storage configured to store raw data, where a portion of the raw data comprises unstructured data;metadata storage configured to store data description metadata describing the raw data; andreport storage configured to store previously generated reports comprising previous reporting data and previous reporting data requirements; receive a report specification, where the report specification comprises at least one reporting data requirement;', 'determine performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based on the report specification; and', 'provide the performance metadata information for the interest-driven data pipeline to an interest-driven user visualization system., 'wherein the interest-driven business intelligence application configures the processor to2. The system of claim 1 , wherein the performance metadata information for interest-driven data pipeline provided to the interest-driven user visualization system includes performance metadata for at least one of the raw data claim 1 , the reporting data claim ...

REALLOCATING JOBS FOR CHECKING DATA QUALITY

The invention provides for checking data quality of data of an application program by a data quality management system. At least one of a plurality of jobs are executed for evaluating the data for compliance with one or more quality criteria. The runtime behavior of the at least one executed job is monitored to determine a current runtime behavior of the executed job. The monitored job is reclassified by reallocating the job to a job set representing the determined current runtime behavior. 1. A computer-implemented method for checking data quality of data of an application program by a data quality management system , the method comprising:executing, by a processor of the data quality management system, at least one of a plurality of jobs that is adapted for evaluating the data for compliance with one or more quality criteria, each of the plurality of jobs is a member of one of a plurality of job sets, the plurality of job sets comprising at least a first job set and a second job set, and each of the job sets is classified into runtime behavior categories in accordance with its respective runtime behavior;monitoring the runtime behavior of at least one executed job by the data quality management system for determining a current runtime behavior of the executed job; and the data includes application data comprising one or more business objects specified by a business object schema of the application program, the application program is operatively coupled to a first database having stored the business objects, and the data quality management system is operatively coupled to a second database, the second database comprising a copy of the application data and respective copies of the business objects in accordance with a mapping of assigned attributes of the business objects of the business object schema to structural elements of the second database;', 'accessing, by the data quality management system, the business object schema and the mapping for determining ...

INFORMATION TECHNOLOGY TESTING AND TESTING DATA MANAGEMENT

A testing and extract, transform and load (TETL) system is operable to interface with test platforms testing hardware or software components of information technology systems. The TETL system can execute extract, transform and load operations to load test data into a data warehouse and facilitates evaluating the test data across projects, entities and domains. 1. A testing and extract , transform and load (TETL) system for managing test data generated by test platforms operable to test hardware or software components of information technology systems , the TETL system comprising:a communications server including a network interface to receive system under test (SUT) test data instances for each SUT tested by the test platforms;a database server including at least one storage device to store the SUT test data instances in intermediate tables, wherein the test data instances are for a plurality of projects in a plurality of domains, and the domains include a plurality of entities;an analytics server including at least one hardware processor to:determine project schemas for SUTs tested by the test platforms;determine fields from each project schema common to all the project schemas;create a common view of the test data instances from the fields determined to be common to all project schemas;determine fields from each project schema that are not common to all the project schemas; andcreate a master view with all the fields from the common view and the fields determined not to be common to all the project schemas,wherein the database server is to load data from the intermediate tables for the master view into target tables in a data warehouse, wherein the target tables include fields corresponding to the master view and comprising fields identifying projects, domains and entities, and fields for measured test data determined by the test platforms.2. The TETL system of claim 1 , wherein to load the data from the intermediate tables for the master view into the target ...

OPTIMIZING LOADING OF RELATIONAL DATA FOR APPLICATIONS

A mechanism for optimizing loading of relational data for applications is disclosed. A method includes monitoring a plurality of requests selecting entities and processes to be executed for the selected entities. The plurality of requests are generated in an application server. The method also includes associating an application context with each of the processes to be executed for the selected entities. The application context includes the process to be executed using an entity stored in a database external to the application server. The application further includes generating a fetch plan for the entity stored in the database; and modifying, the generated fetch plan in view of the selected entities and the associated application context. 1. A method , comprising:monitoring, by a processing device, a plurality of requests selecting entities and processes to be executed for the selected entities, wherein the plurality of requests are generated in an application server;associating, by the processing device, an application context with each of the processes to be executed for the selected entities, wherein the application context comprises the process to be executed using an entity stored in a database external to the application server;generating a fetch plan for the entity stored in the database; andmodifying, by the processing device, the generated fetch plan in view of the selected entities and the associated application context.2. The method of wherein the application server comprises a Java™ programming language application interface specification (JPA).3. The method of claim 1 , further comprising storing identification of the selected entities and the associated application context in a first database located inside the application server.4. The method of claim 3 , further comprising comparing the identification of the entity in the generated fetch plan with the stored identification of the selected entities.5. The method of claim 4 , further comprising ...

SYSTEMS AND METHODS FOR AN SQL-DRIVEN DISTRIBUTED OPERATING SYSTEM

Various embodiments can include systems, methods, and non-transitory computer readable media configured to receive at least one operation to be performed using (i) first data that is managed by a first computing system and (ii) second data that is managed by a second computing system, the operation being received through an interface provided by the computing system, and wherein the operation is based at least in part on a Structured Query Language (SQL). At least one optimization can be performed based at least in part on the operation. The operation can be executed using at least the first data and the second data. A result generated can be provided upon executing the operation through the interface provided by the computing system. The computing system, the first computing system, and the second computing system are each able to concurrently process, access, and create at least a portion of the generated result. 1. A computer-implemented method comprising:receiving, by a computing system, at least one operation to be performed using (i) first data that is managed by a first computing system and (ii) second data that is managed by a second computing system, the operation being received through an interface provided by the computing system, and wherein the operation is based at least in part on a Structured Query Language (SQL);performing, by the computing system, at least one optimization with respect to the operation;executing, by the computing system, the operation using at least the first data and the second data; andproviding, by the computing system, a result generated upon executing the operation through the interface provided by the computing system, wherein the computing system, the first computing system, and the second computing system are each able to concurrently process, access, and create at least a portion of the generated result.2. The computer-implemented method of claim 1 , wherein the first data is associated with a SQL database type claim 1 , ...

PROCESSING DATA IN A DATA WAREHOUSE

Processing data of a data warehouse is provided and includes receiving, by a processing device, user input to create simple filter objects. Each filter object defines an ad hoc subset of a respective dimension of a dimension table of the data warehouse. User input is received to create a filtered operation object that specifies an operation and a plurality of the simple filter objects. The ad hoc subset differs from all subsets defined in the dimension table. 1. A computer-implemented method of processing data of a data warehouse , comprising:receiving, by a processing device, user input to create simple filter objects, each simple filter object defining an ad hoc subset of a respective dimension of a dimension table of said data warehouse;receiving a user input to create a filtered operation object that specifies an operation and a plurality of said simple filter objects, wherein said ad hoc subset differs from all subsets defined in said dimension table.2. The computer-implemented method of claim 1 , wherein said user input specifies at least one variable and at least one operation of an equation with which one of said ad hoc subsets is defined.3. The computer-implemented method of claim 1 , comprising:receiving a user input to create a compound filter object that specifies at least two of said simple filter objects, wherein said filtered operation object specifies said compound filter object as said plurality of said simple filter objects, andthe dimension of each of said simple filter objects of said compound filter object differs from the dimension of all other simple filter objects of said compound filter object.4. The computer-implemented method of claim 1 , comprising:automatically grouping said simple filter objects into simple filter groups by dimension.5. The computer-implemented method of claim 1 , wherein said data warehouse comprises a fact table having rows and columns of data including columns of dimension data and columns of measures claim 1 , the ...

SYSTEMS, METHODS AND DEVICES FOR EXTRACTION, AGGREGATION, ANALYSIS AND REPORTING OF FINANCIAL DATA

Systems, methods and devices for storing and updating financial data, receiving and processing report requests and generating reports using a cloud based parallel platform with multiple sets of processor engines. The platform arranges atomic elements in a cube or data lake based on a common data model for instruments. The platform uses a set of processor engines to asynchronously update the atomic elements. The platform uses another set of processor engines to asynchronously aggregate a portion of the atomic elements to generate output data in response to on-demand reports. 1. A risk management platform comprising:an interface configured to receive input data from data sources, transforms the input data to compute atomic elements, and store the atomic elements in a distributed data storage device, the atomic elements being additive and modeled using a common data model;a first set of parallel processor engines configured to continuously monitor the data sources to detect updates to the input data, and generate corresponding updates to the atomic elements in the distributed data storage device;a second set of parallel processor engines to operate on the updated atomic data elements using ETL logic and aggregate the atomic elements using rules; anda reporting unit configured to receive on an demand request for an electronic real-time report, determine required atomic elements for generating the report, trigger the second set of parallel processor engines to aggregate the atomic elements on demand, and generate the report using the aggregated atomic elements, the report providing a plurality of visual representations of the aggregated atomic elements.2. The risk management platform of wherein the first set of parallel processor engines operates asynchronously from the second set of parallel processor engines.3. The risk management platform of wherein the input data relates to market factors claim 1 , instruments and scenarios claim 1 , and wherein the atomic elements ...

Automated Testing Materials in Electronic Document Publishing

A testing materials production system generates a testing document associated with an electronic foundation document. The testing materials production system automatically generates the testing document by populating regions of a template with testing elements extracted from the electronic document. User customizations of the testing document are received, and the testing document is published to an extended catalog database associated with the electronic document. Regions of the testing document may be delivered in association with the electronic document to designated recipients via browser applications executing on computing devices of the recipients. 1. A method for delivering a testing document associated with an electronic document , the electronic document configured for rendering by browser applications executing on client devices of a plurality of users , the method comprising:extracting, from the electronic document, a set of testing elements, wherein the electronic document comprises a plurality of content layers, wherein the plurality of content layers includes related content, advertising content, social content, and user-generated content, and wherein the extracting the set of testing elements includes extracting from the plurality of content layers;populating a template with the testing elements to generate the testing document, the template comprising a plurality of available regions and the testing document comprising a plurality of regions each populated with a testing element, wherein the regions of the testing document are configured for delivery to the users through the browser applications in association with the electronic document;associating a set of delivery conditions with the regions of the testing document, the delivery conditions comprising a specified location in the electronic document at which one or more regions of the testing document are to be delivered to the users;storing the testing document and the associated delivery ...

Techniques for data extraction

Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

System and Method for Text Extraction and Contextual Decision Support

A contextual analysis system that extracts data elements from an unstructured text input; determines whether the extracted data elements are relevant to a predetermined context; and determines, for the extracted elements deemed as relevant, whether the information contained in the relevant data elements complies with a guideline. 1. A computer implemented method for automatically generating a medical decision for a patient having an associated electronic medical record , the method comprising:retrieving, with at least one computer processor, an electronic medical record data structure defining a patient condition;receiving, with at least one computer processor, textual input associated with the patient condition and a time stamp;querying, with the at least one computer processor, a concept database for a first concept associated with the textual input;determining, with the at least one computer processor, that the first concept is associated with a time-based trigger;generating, with the at least one computer processor, a sequence comprising the first concept;querying, with the at least one computer processor, a guideline database to obtain a guideline associated with the time-based trigger;generating, with the at least one computer processor, a determination that the sequence does not comply with the guideline;generating, with the at least one computer processor, a prompt describing a course of action for treating the patient based on the determination.2. The computer implemented method of claim 1 , wherein a patient is treated according to the prompt.3. The method according to claim 1 , wherein the textual input includes at least one of the following: a word claim 1 , a keyword claim 1 , a phrase claim 1 , a numerical expression claim 1 , and a numerical value.4. The method according to claim 1 , wherein the electronic medical record data structure is updated with a description of the course of action.5. The method according to claim 1 , wherein a second ...

AIR QUALITY METROLOGY SYSTEM

Metrology system () for the management of observation data, this system being configured to collect at least one observation datum and to associate with this observation datum a quality code reflecting the usability of the observation datum with respect to a predefined quality criterion. The system includes a data acquisition module (); a centralized data management module (); a data presentation and dissemination module (); and transverse functional bricks () for data processing and production; for end-to-end data quality control; and for intermediation so as to urbanize the architecture and allow the exposure of services. 12. A metrology system () for the management of observation data , this system being configured to collect at least one observation datum and to associate with this observation datum a quality code reflecting the usability of the observation datum with respect to a predefined quality criterion , this system comprising:{'b': '21', 'a data acquisition module ();'}{'b': '22', 'a centralized data management module ();'}{'b': '23', 'a data presentation and dissemination module (); and'}{'b': '24', 'transverse functional bricks ()for data processing and production;{'b': '2', 'for end-to-end data quality control of the metrology system ();'}{'b': '2', 'for intermediation so as to urbanize the architecture of the metrology system () and allow the exposure of services.'}22. The metrology system () as claimed in claim 1 , wherein the quality criterion is chosen from the list of the following criteria: the non-redundancy of the observation datum claim 1 , the consistency of the observation datum according to at least one predefined rule claim 1 , the integrity of the observation datum.32. The metrology system () as claimed in claim 1 , configured to trigger an alert following a comparison of the quality code to a predefined alert threshold.4221212211213. The metrology system () as claimed in claim 1 , for the management of observation data and air quality ...

APPARATUS AND METHOD TO PROCESS PIECES OF COLLECTED DATA BASED ON DATA ARRIVAL STATES

An apparatus collects record data from a plurality of information sources. When a process to be performed on the collected record data is determined, with reference to definition information on the process, to be able to generate result data from a single piece of the record data, the apparatus starts the process on the collected record data without waiting for an arrival of all the record data as a collection target from the plurality of information sources. When pieces of the record data are required to be referenced in order for the process to generate the result data and the pieces of the record data includes an uncollected piece of the record data, the apparatus starts the process in response to completion of collecting all the record data as the collection target from the plurality of information sources. 1. A non-transitory , computer-readable recording medium having stored therein a program for causing a computer to execute a procedure comprising:collecting record data from a plurality of information sources;when a process to be performed on the collected record data is determined, with reference to definition information on the process, to be able to generate result data from a single piece of the record data, starting the process on the collected record data without waiting for an arrival of all the record data as a collection target from the plurality of information sources; andwhen first pieces of the record data are required to be referenced in order for the process to generate the result data and the first pieces of the record data includes an uncollected piece of the record data, starting the process in response to completion of collecting all the record data as the collection target from the plurality of information sources.2. The non-transitory claim 1 , computer-readable recording medium of claim 1 , the procedure further comprising:when the first pieces of the record data have been collected, starting the process without waiting for an arrival of ...

CALLABLE CHANGE RECORD DATA

This disclosure relates to improving the efficiency and quality of real-time extracting, transforming, and/or loading data using customer information control system (CICS) interval control element (ICE) chain processing. 1. A computer-implemented method comprising:executing, by a computer-based data retrieval system, for each child interval control element (ICE) Chain, an application service program that captures data from a plurality of databases; and wherein the consumption ready record is written to a cache table, and', 'wherein changes to the consumption ready record are populated as current data, in response to changes to the data., 'creating, by the computer-based data retrieval system, a consumption ready record,'}2. The method of claim 1 , wherein the creating the consumption ready record further comprises performing application transformation logic.3. The method of claim 1 , further comprising a spawning mechanism configured for thread management claim 1 , wherein data changes are replicated substantially in real time.4. The method of claim 1 , wherein a master ICE Chain application is associated with a legacy Cobol programed system.5. The method of claim 1 , wherein the data in the consumption ready record is available without requiring additional computations.6. The method of claim 1 , further comprising allocating claim 1 , by the processor claim 1 , among child change tables with associated child ICE Chains claim 1 , a portion of an extract task using a master ICE Chain application.7. The method of claim 1 , wherein a master change table is monitored by a master ICE Chain.8. The method of claim 1 , wherein a master change table includes a change record.9. The method of claim 1 , wherein responses to requests for calculations of the current data related to the associated account are available in substantially real-time claim 1 , without the need to perform the calculations in the responses to the requests.10. The method of claim 1 , wherein requests for ...

MODEL-DRIVEN PROFILING JOB GENERATOR FOR DATA SOURCES

Embodiments of the present invention disclose generating a data profiling jobs for source data in a data processing system, the source data being described by at least one source functional data model. A target functional data model is provided, for describing target data that can be generated from the source data. One or more source functional data models are identified that correspond to the target functional data model. At least one functional source-to-target model mapping is associated to at least one source-target pair based on the target functional data model and identified source functional data models. A physical source-to-target model mapping for at least one source-target pair based on the logical source-to-target model mapping is calculated. For all physical source attributes, the needed data profiling jobs are generated based on the target attribute for analyzing the physical source attributes. 1. A computer implemented method for generating data profiling jobs for source data in a data processing system , the data processing system comprising at least one data source system and a target system forming at least one source-target pair , the source data being described by at least one source functional data model , the method comprising:providing a target functional data model for describing target data generated from the source data, wherein each of the source and target functional models is physically implementable in the data processing system by a corresponding physical data model;identifying one or more source functional data models of the at least one source functional data models corresponding to the target functional data model;associating at least one functional source-to-target model mapping to at least one source-target pair based on the target functional data model and identified source functional data models, the functional source-to-target model mapping indicating data rules for generating the target data from the source data;calculating a ...

Method of organizing multiple hierarchical data structures in a single normalized structure of relational database for automation of data process design and for discovery analysis.

The invention is in Data Architecture (DA) area of Information Technology (IT) and presents a new method of building a common data structure for descriptive data in order to automate significant volume of design and development in IT data processing, as well as to provide a dynamic Information Analytical Base (INFABASE) for real discovery analysis vs. slice-and-dice predefined dimensions: by changing data in such structure users can change data processing and analysis, which means that there is no need for application changes while users can try their ideas, for example if user wants to change the definition of dimensions to something absolutely new and without a project and without detailed knowledge of the tables behind (long expected “sand-box” feature). 1. Method of organizing multiple hierarchical data structures in , a single normalized structure of relational database for automation of data process design and for discovery analysis comprising:at least one Element Base record in the ELEMENT BASE records entity (can be table in relational database), where each such record has a unique identifier (Element Base ID) and identifies the nature of elements by being a parent in identifying relation with Element records in Element entity;at least two Element records in a records entity ELEMENT,where each record has a uniquely identifying field or a set of fields that uniquely identify the element;at least one Relation record in RELATION entity, where each record has a uniquely identifying field or a set of fields that uniquely identify each relation, and record also has two identifiers of elements participating in the relation, where one Element ID represents a parent element of the relation and other element ID represents the ID of the Child element of this relation and both Identifiers are required fields (can not be empty or unknown), and any Relation entity record can be a parent to ELEMENT BASE entity records due to non-identifying relation between RELATION and ...

POST-MIGRATION VALIDATION OF ETL JOBS AND EXCEPTION MANAGEMENT

Handling extract-transform-load (ETL) job mismatches as “exceptions.” Exception handling may include the following steps: (i) determining a mismatch while running an extract-transform-load job with the mismatch being a mismatch of at least one of the following types: design time information mismatch, and/or operational metadata mismatch; and (ii) responsive to determining the mismatch, handling the mismatch as an exception. 1. A method comprising:determining a mismatch while running an extract-transform-load job with the mismatch being a mismatch of at least one of the following types: design time information mismatch, and operational metadata mismatch; andresponsive to determining the mismatch, handling the mismatch as an exception.2. The method of wherein the determination of the mismatch while running an extract-transform-load job includes the following:collecting, in a database, design time information and runtime operational metadata from a source instance of the extract-transform-load job; andcomparing design time information and runtime operational metadata from a target instance of the extract-transform-load job to the design time information and runtime operational metadata collected in the database from the source instance of the extract-transform-load job to determine the mismatch.3. The method of further comprising:responsive to determining the mismatch, fixing the exception.4. The method of wherein fixing the exception comprises the following:parsing a target job log to extract a failure string;fetching an appropriate predetermined fix method from a fix repository; andapplying the fetched fix method to the target instance of the extract-transform-load job.5. A computer program product comprising software stored on a software storage device claim 3 , the software comprising:first program instructions programmed to determine a mismatch while running an extract-transform-load job with the mismatch being a mismatch of at least one of the following types: ...

Storing Metadata in a Table-based Format from Metadata in a Serial Format

In one embodiment, a method receives a document including metadata in a serial format and parses the document to determine components in the metadata, a nested structure of the components, and an order of the components in the data. The metadata is stored in a table format where columns of the table define the nested structure of the components and the order of the components in the data. A query is received for the data. The method determines components from the metadata stored in the table format from the columns that satisfy the query where the components are determined based on the nested structure of the determined components and the order of the determined components from information in the columns of the table. A result for the query is output including the determined components based on the nested structure of the determined components and the order of the determined components. 1. A method comprising:receiving, by a computing device, a document including metadata in a serial format;parsing, by the computing device, the document to determine components in the metadata, a nested structure of the components, and an order of the components in the data;storing, by the computing device, the metadata in a table format, wherein columns of the table define the nested structure of the components and the order of the components in the data;receiving, by the computing device, a query for the data;determining, by the computing device, components from the metadata stored in the table format from the columns of the table that satisfy the query, wherein the components are determined based on the nested structure of the determined components and the order of the determined components from information in the columns of the table; andoutputting, by the computing device, a result for the query, the result including the determined components based on the nested structure of the determined components and the order of the determined components.2. The method of claim 1 , wherein ...

SYSTEM AND METHOD TO PROVIDE ANALYTICAL PROCESSING OF DATA IN A DISTRIBUTED DATA STORAGE SYSTEMS

The present disclosure in general relates to technologies for processing data in a distributed data storage system, and more particularly, to a method, a system, and a computer program product for analytical processing of data by using the processing power of the distributed data storage system. In one embodiment, a system for analytical processing of data in a distributed data storage system is disclosed. The system comprises: a data extraction module configured to perform analytical operations to extract data from source databases in one or more data formats; and a processing module configured to perform data refinement operations to categorize the data while the data is being extracted. The processing module comprises: a mapping module configured to perform mapping operations of the categorized data; and a transformation module configured to perform an analytical transforming operation of the mapped categorized data to obtain a transformed categorized data. 1. A system for analytical processing of data in a distributed data storage system , the system comprising a processor and a memory storing instructions , the instructions comprising:a data extraction module configured to perform one or more analytical operations to extract data from one or more source databases in one or more data formats, the one or more source databases having one or more types of constraints and structures; and a mapping module configured to perform one or more types of mapping operations of the categorized data based on one or more mapping rules; and', 'a transformation module configured to perform an analytical transforming operation, based on one or more business rules, of the mapped categorized data to obtain a transformed categorized data, wherein the transformed categorized data is stored in a target area in the distributed data storage system., 'a processing module configured to perform one or more data refinement operations to categorize the data while the data is being extracted, ...

BULK READ AND WRITE BETWEEN MULTI-DIMENSIONAL DATA STRUCTURES

A system and method for reading and writing of data values between multidimensional structures that reside in Online Analytical Processing (OLAP) databases are disclosed. Data queries may be performed and updates executed between multidimensional data structures, whether existing on the same server or separate servers. Bulk (being two or more intersections) transfers are allowed between multidimensional structures (or cubes), providing a performance gain that cannot be matched using a standard point-by-point implementation. An intersection only contains a numerical or data value if there is a value for each dimension at that intersection within the database. Multidimensional data structures naturally generate sparse intersections where no data values are found, which can greatly impact performance. Within each cube, only a small intersection of members actually contains values. The system may very quickly calculate reports which include any intersection in any very large cube. 1. A system for reading or writing data values residing in multidimensional structures , the system comprising:a. at least one server, wherein the server includes a processor and memory storing at least one multidimensional database;b. a program stored on a computer readable medium, wherein the program may access the at least one multidimensional database stored on the server;c. a communication interface, wherein the communication interface receives bulk read or write requests and transmits responses;d. wherein the processor and server perform bulk reading or writing between multidimensional structures.2. The system of claim 1 , wherein the at least one multidimensional database is an Online Analytical Processing database.3. The system of claim 1 , wherein the at least one server comprises two or more servers claim 1 , and wherein the bulk reading or writing is performed occurs between multiple databases residing on the first and second servers.4. The system of claim 1 , wherein a single ...

METHOD AND SYSTEM FOR LARGE-SCALE DATA LOADING

The present invention provides a method and system for large-scale data loading including generating a data science model with at least one million data points. The method and system includes determining at least one native data resource having native data stored thereon and determining a size of the model data generated from the native data by translating a model query format of the data science model into a native query format of the native data resource. The method and system queries the native data resources using the data science model and receiving the model data, including transporting the model data to temporary data resources. The method and system engages the model data with the data science model and trains the data science model using the model data stored in the temporary data resources. Where the iterative training process requires multiple data-loading operations made possible under the present method and system. 1. A method for large-scale data loading , the method comprising:generating a data science model using model data having at least one million data points;determining at least one native data resource having native data stored thereon;determining a size of the model data generated from the native data by translating a model query format of the data science model into a native query format of the native data resource;querying the native data resources using the data science model and receiving the model data in response thereto;partitioning the model data and transporting the model data to temporary data resources using parallel transmissions based on the partitioning;reconstituting the model data from the parallel transmissions within the temporary data resources;engaging the model data, stored in the temporary data resources, with the data science model; andtraining the data science model using the model data stored in the temporary data resources.2. The method of further comprising:generating a first score for the data science model.3. The ...

USING CLOUD PROCESSING TO INTEGRATE ETL INTO AN ANALYTIC REPORTING MECHANISM

A method and associated systems for using cloud processing to integrate ETL into an analytic reporting mechanism. A computerized reporting tool receives information from which it infers an information query. The tool determines whether existing reusable ETL code may be used in order to respond to such an inferred query. If not, the tool modifies existing instructions or generates new instructions that may be reused when responding to future queries. The system provisions a virtual computing environment on a cloud-computing platform and populates the environment with virtual components that run the reusable ETL code and then stream the resulting transformed datasets to the tool. The reporting tool then uses the returned datasets to generate or update an animated or real-time report. 1. A computerized ETL-based reporting tool comprising a processor , a memory coupled to the processor , and a computer-readable hardware storage device coupled to the processor , the storage device containing program code for a reporting tool configured to be run by the processor via the memory to implement a method for generating an ETL-based report , the method comprising:the reporting tool inferring a data query as a function of receiving information input;the reporting tool identifying one or more data sources containing retrieved data needed to respond to the inferred query;the reporting tool selecting a set of Extraction, Transform, and Load (ETL) code;the reporting tool launching the ETL code, where the ETL code extracts, transforms and loads the retrieved data from the one or more data sources, such that the retrieved data is transformed into at least one dataset;the reporting tool receiving, as a function of the ETL code, the at least one dataset; andthe reporting tool creating a report as a function of the at least one dataset.2. The reporting tool of claim 1 , further comprising the reporting tool generating the ETL code as a function of the data query.3. The reporting tool of ...

Data Model Augmentation

Techniques are described for data model augmentation. In one example, techniques include receiving a data contribution to a data model, receiving meta information about the data contribution, mapping the received data contribution and received meta information to the data model; and dynamically generating an augmentation of the data model based on the mapping. 1. A computer-implemented method for data model augmentation , the method comprising:receiving a data contribution to a data model;receiving meta information about the data contribution;mapping the received data contribution and received meta information to the data model; anddynamically generating an augmentation of the data model based on the mapping.2. The method of claim 1 , wherein the meta information includes time or location information for the data contribution.3. The method of claim 1 , wherein dynamically generating an augmentation includes:identifying data types for the meta information.4. The method of claim 1 , further comprising:sending a report based on the augmented data model to a recipient.5. The method of claim 4 , wherein the recipient is a point of sale system.6. The method of claim 1 , wherein the data contribution includes data for a supply chain management system.7. The method of claim 1 , wherein mapping the received data contribution and received meta information includes:building a database of patterns.8. A computer system for data augmentation claim 1 , the computer system comprising: program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to receive a data contribution to a data model;', 'program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to receive meta information about the data contribution;', 'program instructions, ...

Assisting a User to Identify Outliers and Anomalies in a Dataset

The present disclosure relates to system(s) and method(s) for data processing and visualization. The system is configured to enable processing and visually representing a dataset specified by a user. Initially, the system accepts a rule corresponding to at least one dimension associated with the dataset, wherein the rule is associated with a lower level of abstraction. When a rollup operation is performed along the specified dimension of the dataset for generating higher level graphical representation, the system is configured to preserve the rule associated with the dataset, regardless of any mathematical or logical cancellation of values due to the rollup operation and reflects the lower level details at a higher level of abstraction. 1. A system for assisting a user to identify a region of interest in a dataset , the system comprising:a memory; and processing raw data to generate a processed dataset, wherein the processed dataset comprises a set of abstraction levels, wherein the set of abstraction levels are hierarchal arranged, and wherein each abstraction level comprises a set of data nodes, wherein each data node at an abstraction level is associated with a subset of the set of data nodes in a preceding abstraction level;', 'accepting a rule corresponding to a target abstraction level from the set of abstraction levels;', 'identifying a set of target data nodes corresponding to the rule from the target abstraction level;', 'identifying a set of intermediate data nodes connecting a source data node at a first abstraction level and the set of target data nodes;', 'generating a multi layer graphical representation corresponding to the set of abstraction levels with a set of visual indicators, wherein each visual indicator is associated with at least one of the source data node, the set of intermediate data nodes, and the set of target data nodes; and', 'displaying the multi layer graphical representation over an output device, thereby assisting a user identify a ...