Spark Jobs

I am building an AI-driven platform that helps telecom vendors understand exactly what has happened across their networks by turning raw data into clear, actionable insight. The focus is strictly descriptive analytics: I want to surface patterns, error trends, and usage anomalies from two key sources—real-time network performance metrics and our historical customer service logs—so engineers and account teams can quickly see where issues occurred and how widespread they were. Here’s what I need from you: • Data pipeline design and implementation that securely ingests high-volume performance counters alongside ticket and chat transcripts, cleans them, and stores them in a query-friendly format. • Feature engineering and statistical/ML models (Python, pandas, s...

Our production datalake pipeline on Hadoop is misbehaving: Spark and Flink jobs crash intermittently, completed stages fail to commit their results to the Hive tables, and overall throughput has slowed to a crawl. I need someone to dive in, trace the failures, and leave me with a clean, fully functioning data flow. You will have direct access to the existing Spark and Flink code, YARN cluster dashboards, Hive metastore, and any relevant logs. The immediate goal is to identify and correct the root causes of the job failures and the missing Hive writes, then tune the pipeline so it can keep up with daily load without time-outs or excessive retries. Acceptance criteria – once you are done: • All scheduled Spark and Flink jobs finish successfully when triggered manually and by t...