Eosense
Visualizing Environmental Gas Data
Project Overview
Eosense, a company specializing in environmental monitoring, approached Éphémère Creative to create a user-friendly dashboard for their clients. The dashboard needed to provide real-time data visualization, role-based access, and the ability to create and manage “jobs” tied to specific devices and timeframes. Éphémère Creative designed and developed a scalable and efficient solution to meet Eosense’s requirements.
Challenges
Fetch and process data from FTP servers without duplicating old data.
Efficiently Assigning and Streamlining User Permissions Management
Visualizing and annotating multiple metrics for data analysis
Ensuring low hosting costs while allowing for future scalability
Solution
Data Retrieval and Processing
Éphémère Creative built a backend system to pull data from the FTP servers at regular intervals. Initially, data was fetched every five minutes, but the rate was later reduced to every 30 minutes or an hour as per the FTP server’s request. The system was deployed on a Kubernetes cluster using CronJobs to schedule the tasks. A Python script was developed to import the data and prevent duplication.
Role-Based Access and Job Management
The dashboard allowed the account owner or administrator to create “jobs” for specific devices and timeframes. Team members could be assigned to work with the data from different jobs based on their access permissions. Metadata related to each job, such as client information and location, could also be added.
Data Visualization and Annotation
Users with access to a specific job could plot multiple metrics from the devices, such as methane emissions, and view data in various aggregations (e.g., average, maximum, or minimum). The dashboard allowed users to add annotations to specific metrics at specific times, providing a powerful tool for in-depth data analysis.
Technologies Used
The following technologies were used to create this solution:
Kubernetes and CronJobs
for data retrieval and processing
hasura
for API and permission management
keycloak
for authentication
next.js
for the frontend, using Material-UI (MUI) for design
timescaledb
on a managed Postgres instance with Digital Ocean for hosting
graphql
for API communication
django
for migrations and providing an admin interface for Eosense
Results
The custom dashboard created by Éphémère Creative was well-received by Eosense and its clients. The end users found it easy to access and analyze data from various devices, jobs, and metrics. The system proved to be efficient, cost-effective, and scalable, allowing Eosense to manage and visualize their environmental monitoring data with ease.