Application-server, with embedded distributed Multimodel in memory database management system

Table of Contents

1. About The Project
2. Getting Started
   • Building
   • Installation
3. Gratitudes
4. Requirements
5. Cases
6. Cluster
7. Troubles
8. Cluster
9. Contributing
10. License
11. FAQ
12. Roadmap
13. Contact

About The Project

Fluidb is an open source (BSD-3-Clause License), application-server, with embedded distributed multimodel database management system distributed uder BSD-3-Clause License based on source code of Algernon, and Rqlite which also distributed uder BSD-3-Clause License. The purpose of our project is to fix fundamental flaws in Redis, such as scaling, creating a multi-threaded server.

We do not have a docker image since docker, like any virtualization environment, forms an additional layer of abstraction that complicates both the development process itself and the program operation process.
Our goal: "To create a high-performance application based on modules architecture that is as easy to use as possible"

fluidB is often referred to as a data structures server. What this means is that fluidB provides access to mutable data structures via a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way. The storage of fluidB is implemented as follows: data can be stored according to the "key-value" model, or can be stored as a graph, which is a chain of interrelated events (which are similar to frames from an old film strip), which together represent a description of some event.

Good example is to think of fluidB as a more complex version of memcached, where the operations are not just SETs and GETs, but operations that work with complex data types like Lists, Sets, ordered data structures, and so forth.

Gratitudes

• Alexander F. Rødseth (xyproto), I would like to express our gratitude for all that you have done!!! Thank you.
• Philip O' Toole, I wish to express my appreciation for all your efforts!!!
  

Requirements

• Hardware: Intel or AMD
• Processor: 64-bit
• RAM: 1 GB or above
• Nodes: 3 (strongly recomended)
• Operating System: UNIX-like only (Linux, BSD(except OpenBSD), MacOS X) Windows isn't supported

Cases

• Logs ageregation
• Caching system
• Queue server
• Pattern "Central dispatcher" (for sharding)
• Solution that sits in front of multiple data sources and allow them to be treated as a single json database
• Web-server and database for hosting static web-sites

Getting Started

Basic modules

fluidB The project consists of two main modules:

1. deepblue-application server with lua- on board

2. flusql- relational embedded scalable database, with the follow key features

   • Trivially easy to deploy, with no need to separately install SQLite.
   • Super-simple to use, with a straightforward HTTP API. A command-line interface is also available, as are various client libraries.
   • Fully replicated production-grade SQL database, with full access to SQLite full-text search and JSON document support.
   • Multiple options for node-discovery and automatic clustering, including integration with Kubernetes, Consul, etcd and DNS, allowing clusters to be dynamically created.
   • Extensive security and encryption support, including node-to-node encryption.
   • Choice of read consistency levels, and support for choosing write performance over durability.
   • Optional read-only (non-voting) nodes, which can add read scalability to the system.
   • A form of transaction support.
   • Hot backups, as well as load directly from SQLite.

Building

1. Install Golang, build-essential in your operating system

2. Clone the repo

   git clone https://github.com/gvsafronov/deepblue.git
   

3. Build project with go-build

   cd fluidb
   go build -mod=vendor
   
   

4. Run application-server:

   ./welcome.sh
   

Installation

For Linux-systems you can install fluidB automaticly using installation script from our site:

Installation Script for Debian-based systems
Installation Script for Red-Hat-based systems
Installation Script for Arch-based systems

1. Open the link above
2. Download the script
3. Copy the tar-archive in your home directory
4. Open you terminal and then print (-terminal prompt) ``` chmod +x install.sh && ./install.sh```
5. Wait for the installation process to complete, after installation the application will start automatically
   

### Running fluidB

To run fluidB with the default configuration (without application server) just run the fluidb-server:

$ cd src
$ ./fluidB-serv

If you want run fluidB with modules you need just print in your terminal:

$ ./fluidB-serv fluidB:~> os.execute('./flusql')

Start working with fluidB

It would be better to start fluidB-cli (client fluidB) to play with fluidB. Start a fluidB-server instance, then in another terminal try the following:

---------------------------------------    
 Let's start with relational model
---------------------------------------

    $ cd src
    $ ./fluidb-serv
    lua~> os.execute ("./rqlited -node-id 1 ~/node.1 && ./rqlite")
    $ rqlite
flusql:~> CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)
0 row affected (0.000668 sec)
127.0.0.1:4001> .schema
+-----------------------------------------------------------------------------+
| sql                                                                         |
+-----------------------------------------------------------------------------+
| CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)               |
+-----------------------------------------------------------------------------+
127.0.0.1:4001> INSERT INTO foo(name) VALUES("fiona")
1 row affected (0.000080 sec)
127.0.0.1:4001> SELECT * FROM foo
+----+-------+
| id | name  |
+----+-------+
| 1  | fiona |
+----+-------+

Cluster

If you need to formed a cluster your need to start two more nodes, allowing the cluster to tolerate failure of a single node, like so:

rqlited -node-id 2 -http-addr localhost:4003 -raft-addr localhost:4004 -join http://localhost:4001 ~/node.2 rqlited -node-id 3 -http-addr localhost:4005 -raft-addr localhost:4006 -join http://localhost:4001 ~/node.3

This demonstration shows all 3 nodes running on the same host. In reality you probably wouldn't do this, and then you wouldn't need to select different -http-addr and -raft-addr ports for each rqlite node.

With just these few steps you've now got a fault-tolerant, distributed relational database. For full details on creating and managing real clusters, including running read-only nodes, check out this documentation.

Tests

To run fluidb test suite with TLS, you'll need TLS support for TCL (i.e. tcl-tls package on Debian/Ubuntu).

1. Run ./utils/gen-test-certs.sh to generate a root CA and a server certificate.

2. Run ./runtest --tls or ./runtest-cluster --tls to run fluidb and fluidb Cluster tests in TLS mode.

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the BSD 3 License. See LICENSE for more information.

FAQ

This page provides answers to frequently asked questions regarding fluidB

• What is the main idea of your project? The key idea of our product is that it solves the problems of processing, storing and analyzing a large amount of continuously incoming data, mainly geolocation data. It is intended primarily for application software developers and DBAs (Database Administrators). Our main goal is to provide the user with a simple yet flexible and functional tool for working with data.

• How is your project different from KeyDB?

When developing our product, we took KeyDB as a basis, as an improved version of the Redis sub-database, easy and understandable to learn and operate. The difference between fluidB and KeyDB lies in the implementation of a shell module that allows you to run external modules (programs) that extend the basic functionality of the application without explicitly specifying the path to the module in the configuration file.

• What is the multi-model database management system? A multi-model subdivision is such a subdivision that allows you to store data using different data storage schemes. For example, along with relational data (stored in the form of tables), data can be stored in the form of associative array, etc. Thus, incredible flexibility in storing data is achieved, depending on the type of data, the most suitable tool for storing and processing it is used.

• Is there a fluidB implementation for the Windows operating system? No, there is no implementation of FluidB for the Windows operating system. our application is client-server, with on-board caching subdivision, focused on use mainly on servers, and the vast majority of servers run under the Linux operating system. We do not plan to release a version for Windows in the future.

• Is fluidB in Linux repositories?
  
  No, FluidB is still distributed as source.

• Is there a docker image of the project? No, there is no docker image, because docker, like any virtualization environment, forms an additional layer of abstraction, which complicates both the development process itself and the program operation process. Our goal: "To create a high-performance subassembly that is as easy to use as possible"

• Why are you confident that you will be able to create a high-quality, competitive software product that people need?
  

For several reasons. First, because our product is based on the well-proven Redis DBMS. Secondly, our software product is already used as a backend in the ws-stickleback project "Biology and Soil" of the Faculty of the Department of Ichthyology and Hydrobiology, St. Petersburg State University

• Is there a paid version of your software product with advanced functionality?

There is such a version, it is calledEnterprise-version, this version includes both the functionality required by the client (discussed individually with each client) and the default functionality.

• How is authorization used?

In the traditional sense, there is no authorization. We have a funny login form that we wrote to show how you can quickly implement it using JavaScript, it is used by default in our fluwc web interface, where you can make "typical requests" in the browser. FluidB is not intended to be publicly displayed. This is an application that works on the Internet, so we do not have authorization at this stage.

• How does your product name stand for?

The name of our project is deciphered as follows: FluidB, this is a reference to the English word "fluid", which means "fluid" - a physical term meaning "a state of matter with parameters above critical", which very accurately characterizes our product. A critical state of a substance means a state in which it disappears the difference between its liquid and vapor phases. This is illustrated by the colors of the logo: At first, there is no load on the application (blue Greek letter "Phi"), then it increases sharply (three red subsequent letters), then the load begins to decrease and evenly distribute- orange symbol, differential, showing the rate of change of the value of the variable during balancing, and finally it is evenly distributed and becomes normal (seventh blue symbol). In our case, it should be interpreted as follows: thanks to our product, the line between low loads and very high loads is erased, i.e. to. it helps balance the load. A couple of words should be said separately about the symbol of the "flower" standing in front of the inscription: It personifies the mathematical graph of objects, tk. our subd is a multi-model supporting graph data storage model.

• I have decided to purchase the Enterprise version of your product, where should I contact?

You shoud write the letter in email: gvsafronov@gmail.com

Roadmap

• Raft implementation for SQL-module (fluSQL)
• Adapt application-server to the core database management system
• Allow rebalancing of connections to different threads after the connection
• Allow multiple readers access to the hashtable concurrently
• Raft implementation for core databases

Contact

Grigoriy Safronov - gvsafronov@gmail.com

Project Link: https://fluidb.icu