Lenguaje natural con Spark, ML Annotators y DL Ontologies. Por Atigeo.com - Seguridad Mania.com - España y América Latina

A text-mining system must go way beyond indexing and search to appear truly intelligent. First, it should understand language beyond keyword matching. (e.g. distinguishing between “Jane has the flu,” “Jane may have the flu,” “Jane is concerned about the flu," “Jane’s sister has the flu, but she doesn’t,” or “Jane had the flu when she was 9” is of critical importance.) This is a natural language processing problem. Second, it should “read between the lines” and make likely inferences even if they’re not explicitly written. (e.g. if Jane has had a fever, a headache, fatigue, and a runny nose for three days, not as part of an ongoing condition, then she likely has the flu.) This is a semisupervised ML problem. Third, it should automatically learn the right contextual inferences to make. (e.g. learning on its own that fatigue is sometimes a flu symptom—only because it appears in many diagnosed patients—without a human ever explicitly stating that rule.) This is an association-mining problem, which can be tackled via deep learning or via more guided ML techniques. David Talby leads a live demo of an end-to-end system that makes nontrivial clinical inferences from free-text patient records and provides real-time inferencing at scale. The architecture is built out of open source big data components: Kafka and Spark Streaming for real-time data ingestion and processing, Spark for modeling, and Elasticsearch for enabling low-latency access to results. The data science components include spaCy, a pipeline with custom annotators, machine-learning models for implicit inferences, and dynamic ontologies for representing and learning new relationships between concepts. David Talby is Atigeo’s CTO, working to evolve its big data analytics platform to solve real-world problems in healthcare, energy, and cybersecurity. David has extensive experience in building & operating web-scale data science and business platforms, as well as building world-class, Agile, distributed teams.