memcached plug-in for Kyoto Tycoon

ID: 20
creation date: 2010/12/07 13:49
modification date: 2010/12/07 13:49
owner: mikio

The latest version of Kyoto Tycoon features "pluggable server" mechanism. Moreover, a module to speak the memcached protocol is included in the core package.

image:1:1291697378-kt-plug-in-arc.png

Pluggable Server

Apache is familiar for its thorough modular design. Various pluggable modules can be inserted among each layer. MySQL is also familiar for its flexible pluggable storage mechanism. You can use your own storage engine through powerful functionality of MySQL.

As with MySQL, I want to support pluggable storage mechanism in KT. However, this article doesn't mention it. By contrast, the networking layer has become pluggable. It allows you to operate databases through arbitrary protocols.

KC manages the DBM layer and KT manages the networking layer. They are separated clearly. So, you can write your own server implementation easily with KC and without KT. Then, why KT should support the pluggable server mechanism?

Databases are shared between the core server and the pluggable server. So, if you update records through the memcached protocol, the update is seen through HTTP protocol. KT is not only a mere network interface, but also a complex data storage system featuring auto expiration, concurrent threading management, scripting extension, and asynchronous replication. Combining the pluggable server and the core server, you can enjoy interoperability for your existing application and powerful management functionality of the core server at the same time. It is similar to the HandlerSocket plug-in for MySQL in terms of advantage of multiple protocols.

The most important advantage is that you can use high availability mechanism through the memcached protocol. To implement asynchronous replication from scratch is not easy. To implement protocol handler only and use it on KT is much easier.

Using the memcached Protocol

If a shared library is load on starting the core server, a pluggable server implementation in the shared library is enabled. Because the core server and the pluggable server runs concurrently, the pluggable server must use another port except for the port of the core server. If you installed the latest version of KT, there is a shared library file "ktplugservmemc.so" under "/usr/local/libexec". Specify it when starting the core server.

$ ktserver -plsv /usr/local/libexec/ktplugservmemc.so -plex 'port=2010'

The "-plsv" option specifies the path of a shared library. The "-plex" option specifies the configuration expression for the pluggable server. The memcached module understand the format such as "host=xxxx#port=xxx". The above command assigns that the pluggable server uses the port 2010, while the default port is 11211.

Confirm that the pluggable server speaks the memcached protocol by using "telnet" command.

$ telnet localhost 2010
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set japan 0 0 5
tokyo
STORED
get japan
VALUE japan 0 5
tokyo
END
quit
Connection closed by foreign host.

The above session stores a record whose key is "japan" and whose value is "tokyo". It can be seen through HTTP of course.

$ ktremotemgr list -pv
japan   tokyo

Great. Please confirm auto expiration, replication, and so on by yourself. As for now, a portion of memcached command is supported: "set", "add", "replace", "get", "delete", "stats", "flush_all", "quit".

Writing Your Own Pluggable Server

The pluggable server mechanism is not dedicated for the memcached protocol only. I designed it originally to combine MessagePack-RPC and I considered to support not only synchronous protocols but also asynchronous protocols. Moreover, to support not only TCP but also UDP and UNIX domain socket.

The design principle is simple. The core server opens some database objects and invokes a callback function giving the references to the database objects. The callback function can receive input data from the outer world arbitrary and operate the database objects arbitrary. The core server doesn't manage concrete operations of the pluggable server. However, the life cycle of the pluggable server must be synchronized to the core server. So, the callback function should be separated into four functions: "configure", "start", "stop", and "finish". To gather them into one abstract class, the following declaration is led.

class PluggableServer {
  /**
   * Configure server settings.
   * @param dbary an array of the database objects.
   * @param dbnum the number of the database objects.
   * @param logger the logger object.
   * @param logkinds kinds of logged messages by bitwise-or: Logger::DEBUG for debugging,
   * Logger::INFO for normal information, Logger::SYSTEM for system information, and
   * Logger::ERROR for fatal error.
   * @param expr an expression given in the command line.
   */
  virtual void configure(TimedDB* dbary, size_t dbnum,
                         ThreadedServer::Logger* logger, uint32_t logkinds,
                         const char* expr) = 0;
  /**
   * Start the service.
   * @return true on success, or false on failure.
   */
  virtual bool start() = 0;
  /**
   * Stop the service.
   * @return true on success, or false on failure.
   */
  virtual bool stop() = 0;
  /**
   * Finish the service.
   * @return true on success, or false on failure.
   */
  virtual bool finish() = 0;

A shared library implementing a pluggable server must publish the following function. It must create an instance of a sub-class of the above abstract class and return it.

extern "C" {
void* ktservinit(void);
}

The function name must be "ktservinit" concisely. Although the type of the return value is "void*", it is only to escape from burden about mangling/demangling issues of C++. The actual type must be "PluggableServer*". The object is deleted implicitly by the caller.

Necessary declaration is in the header file "ktplugserv.h". To implemnet your own pluggable server, include the header in your code, and implement a sub-class of PluggableServer, and define the function "ktservinit". For detail, please read the source file "ktplugservmemc.cc" in the source package.

Conclusion

If you want other protocols except for HTTP, the "pluggable server" mechanism is useful. An implementation to support the memcached protocol is also included. So, now, I can recommend Kyoto Tycoon to all memcached users in the world.

The pluggable server mechanism is for generic purpose. You can write your own plug-in easily so that your application can operate databases of KT through arbitrary protocols and use powerful management functionality of KT at the same time.

4 comments
YT : Great work! Thank you. Are range queries for B-tree supported? (2010/12/11 02:45)
mikio : You can use the cursor mechanism for that purpose. (2010/12/11 17:15)
way : I used TT in my project, but the database file is becoming bigger and bigger, the record number is about 20 million, I will set or delete the record using memcached protocol, so would you please tell me how can I contorl the size of the database file? (2013/05/20 11:26)
way : btw, I use replication, one master and a slave, the database file is data.tch (2013/05/20 11:27)
riddle for guest comment authorization:
Where is the capital city of Japan? ...