Quick overview

I write this little article to present the features of a new project started few days ago.

I’m coding a cluster management system, to handle multiple workers performing different tasks :

  • artificial intelligence algorithms
  • data-storing
  • anything else…

This system is based on a global management system containing few services concurrently running :

  • a python HTTP server for web administration
  • a python server called dispatcher performing calls to a C task-scheduler library
  • misc network utilities (proxy, load balancer, …)
  • SSH server for CLI administration

A custom shell is available both with SSH and a with small javascript web application to manage the daemon with a command line interface easily ! But it’s only a beginning !

Task scheduling and programming language

The task scheduling algorithm has been tested with a distributed computing spare-time project with François Boiteux and Camille Dollé (https://github.com/Underflow/parallel-computing) on a small cluster in a LAN (20-100 x86 machines). It is for the moment working with a round robin algorithm and tasks are indexed with an hashtable for rapid access. Task priority is not handled yet.

There is both client and server side C library, and even a almost working proof of concept compiler (https://github.com/Underflow/tiny-language) for a custom programming language to write distributed algorithm written with Guillaume Sanchez last year. This language is verry close to C, there is as less as possible side-effects and it implements an operator ‘@’ to prefix funcalls and distribute them to a node easily with low level runtime speed

TODO

  • Remote directory
  • Shell commands
  • Package management system
  • Any idea ..?