LinkedIn is the largest professional and employment-oriented service platform. The company has been leveraging AI/ML to optimise various processes such as job postings, job recommendations and business insights. LinkedIn sees more than 210 million job applications submitted every month to the 57 million companies listed on the platform. LinkedIn’s Daily Executive Dashboard (DED) contains metrics on critical growth, engagement and bookings. It monitors and provides reports on important KPIs for business profiles, indicating the health of LinkedIn’s business. In addition, the LinkedIn system visualises more than 40 metrics across the business lines to provide company leaders with business insights promptly on their dashboards. REGISTER>> The process begins with ingesting billions of records from online sources into HDFS. The Hadoop Distributed File System is designed to run on commodity hardware. The system manages data processing and storage for big data applications by providing high throughput access to application data. LinkedIn’s records are aggregated across more than 50 offline data flows, making its huge dataset applicable for Hadoop. To ensure business continuity, LinkedIn picked Teradata to meet the growing demands in batch processing. Big Data Engineering built and maintained the DWH’s data flows and datasets. LinkedIn’s data warehouse had grown to 1,400+ datasets,…
Read More
Inside LinkedIn's Big Data Pipelines
