IDF Dataflow Analyst

SFI is seeking an IDF Dataflow Analyst to support a large-scale data modernization initiative transitioning legacy, on-premise systems into a centralized, high-availability cloud data platform. This role is part of a Dataflow Team that serves as the critical bridge for the movement of real-time, high-volume data across a complex enterprise environment.

Over the next 6–12 months, the primary focus will be on supporting the development and population of a centralized, real-time data repository designed to enable scalable data access, advanced analytics, and future AI/ML capabilities.

Key Responsibilities

• Support the ingestion and migration of large-scale datasets into a cloud-based data platform
• Assist in modernizing legacy dataflow pipelines into automated, cloud-native solutions
• Maintain high availability and reliability of real-time data delivery across hybrid environments
• Monitor, troubleshoot, and optimize data ingestion and routing processes

Primary Responsibilities

• Data Lifecycle Management:
  Manage the end-to-end flow of real-time data, including validation, quality control, archival, and retrieval
• Cloud Data Ingest Support:
  Execute and support ingestion processes for mission-critical datasets into the cloud environment
• Pipeline Modernization:
  Assist in transitioning legacy workflows into API-driven and automated scripting solutions (Python/Bash)
• Operational Monitoring & Support:
  Ensure continuous data flow in a 24/7 environment; troubleshoot and resolve data delivery issues
• System Integration & Performance:
  Monitor and optimize data routing applications and cloud services to ensure performance and low latency
• Scripting & Maintenance:
  Maintain and enhance scripts and legacy tools (Bash, Perl, SQL) while supporting migration to modern solutions
• CI/CD & Automation:
  Support automated deployment, testing, and monitoring of dataflow applications and pipelines
• Tier 2 Support:
  Serve as a technical escalation point for complex data ingestion and formatting issues
• Documentation:
  Develop and maintain technical documentation, processes, and configuration standards
• On-Call Support:
  Participate in a 24/7 on-call rotation to support mission-critical operations
• Stakeholder Collaboration:
  Work with internal teams and external partners to resolve data access, formatting, and integration challenges