Hopsworks distributed DL platform

DeepCube partner Logical Clocks is an ICT SME that has developed the most mature and complete data-intensive  AI platform with a Feature Store in the market. Specifically, Logical Clocks brings Hopsworks (https://www.hopsworks.ai/), which is an open source, interoperable big data and distributed Deep Learning platform. Hopsworks provides a unified AI platform for developing end-to-end machine learning pipelines, from data ingestion, to feature validation and engineering, to scalable deep learning with GPUs, to model deployment monitoring.

Hopsworks is an open-source project providing a unified framework called “Distribution oblivious training function” that assists developers with ML training and reduces overall ML model development time. This platform will be further enhanced and support Earth Observation value chains through specific DeepCube Use Cases. Developers will also leverage the horizontal scalable deep learning capabilities of Hopsworks to develop machine learning models at scale with GPUs. In addition, the Hopsworks Feature Store will serve as the backbone of the DeepCube architecture for developing end-to-end machine learning pipelines.

Within DeepCube, the Maggy framework is being developed and extended. Maggy is a Python library for Transparent Distributed Hyper Parameter Optimization (HPO), Ablation and Training of ML models. The term “Transparent” stands for the ability of the library to convert a single-machine training logic into a distributed one. Maggy is currently available in Hopsworks and we are also developing Maggy to work locally (for example on servers and workstations) and other platforms such as Databricks. Maggy supports TensorFlow and PyTorch. Furthermore, it implements the concept of “Oblivious training function”. Briefly, it means that developers write only one training logic and use it for Distributed HPO, Ablation and Training.

You can follow Hopsworks and Maggy developments on GitHub:
https://github.com/logicalclocks/hopsworks
https://github.com/logicalclocks/maggy

Interested in learning more? Contact us!
Jim Dowling, [email protected]