yousry.de/blog

If I’m asked to draw an “easy to understand” diagram about the next-generation architecture for Web-Applications, one of my sketches would look like this:

SQL And NoSQL

An obvious question about this picture is:

Why are two different Database-Systems necessary?

My artless answer is as follows:

In some crucial cases a NoSQL Database is considerably less expensive than a SQL Database, but a NoSQL Database cannot completely replace a SQL Database.

For example:

• Try to store millions of Media Files or other large Documents on a distributed RDBMS. Each document has to be quickly accessible and updatable.
• Try to create a very large dynamic key/value table as a replacement for an overgrown in-memory Hashmap.

Of course you can achieve this also with a relational database, but you will be faced with following problems:

• RDBMS BLOBS are slow. You can compensate this problem with database clustering but the typical workaround is to store a link or file-path and use the file system as data storage. In this case you always have to check for inconsistencies.
• You can create a key/value store in your SQL Database. But each access creates a heavy processing overhead. (Transaction handling, logging and versioning.)

NoSQL-Databases are optimized and designed to:

• Store and access large “binary” objects very fast.
• Accomplish fast data replication and use table separation (sharding) between different storages.
• They allow very fast key-value stores.

(Easy schema modification is often stated as NoSQL advantage, but I think this is also a trivial task with DDL.

– Don’t forget to set the initial value of the new column and enable the trigger after the table alteration. )

In contrast there are tasks you should use a SQL – Database:

• Data Presentation: In most cases visual entities (for example table lines and columns), aren’t 1:1 presentations from Database objects. They are sorted, mapped, reduced (filtered) and often consist of joins between different tables. Map-Reduce and sorting can be done efficiently with NoSQL. But joins aren’t supported. Of course you can rewrite the JOIN operation in your application. For example a simplified HashMap JOIN could look like this:

  var tableBMap : HashMap[idxType, Type of tableB] = tableB.foreach( line => tableBMap += (line.joinAttributB -> line))
  tableA.map( line => (line, tableBMap.get(line.joinAttributA)))

  The Disadvantage is, that you have to materialize the complete object / table-line.

• NoSQL = No Transaction. You cannot use NoSQL for reliable database transactions (reliable like in ACID). This is a serious restriction for massive parallel database updates. (Hence, you should avoid programming the popular Bank Account example with a NoSQL Database.)

Why I prefer MongoDB as NoSQL Database System:

• Very Fast: Sequential write and random read operations are done very fast on an “average” Server. (You need a 64 Bit OS if your database is larger than 2GB)
• Scala(Java) Support: Several drivers are available scamongo, mongo-scala-driver or akka-persistence
• Easy to use: There is no setup, parameter or table type “magic”. In less than an hour you can setup a secure and robust Database Server. The client uses JS/JSON.
• A nice community: The Forum is very active. Questions should be answered in a couple of hours.

Addendum to my recent Post: FSM stands for “Flying Spaghetti Monster”

In my recent post I used the abbreviation FSM for “Finite State Machine”.

I was informed that FSM is the common abbreviation for the Flying Spaghetti Monster.

I searched for this term in the Kungle News document-storage and found some evidence for this claim:

Small Screendump

Here is the complete list of references:

Query:

Start Main

New Hist Defined with:

Primary: List(Flying Spaghetti Monster, Pastafarianism, Pastafarian)

Secondary in: List(FSM)

Secondary out: List()

Calculated Interval: { “publishingDate” : { “$gt” : “2010-01-10T23:00:00.000Z” , “$lte” : “2010-03-22T23:00:00.000Z”} , “originalLanguage” : “ENGLISH”}

…

…

2; 0; 19

Topic Connections:

(fsm,1)(monster,1)(store,1)(flying,1)(church,1)(spaghetti,1)

Emotions:

(love,14)(hope,13) (+)/(-) (hate,7)(fear,7)

Public Tendency:

10; 5; 6

Country:

gb; us; no; cn; jp; in; se; au; ru; de; fr; ie; gr; nz; ca;

0; 20; 0; 0; 0; 0; 1; 0; 0; 0; 0; 0; 0; 0; 0;

Publisher Tendency:

(Discover Magazine,3)(Huffingtonpost,2)(The Local Germany,1)(Think Progress,0)(Talking Points Memo,-1)(Crooks and Liars,-3)

Calculation Done

Since January the New “Corpus Engine” is in development and recorded about 302.000 articles. All in all 1.130.000 “news headlines” and summaries were stored since Kungle.de went online.

Now new algorithms were developed to:

• Identify the public opinion about political and economic topics.
• Follow the image status of brands, corporations or companies.
• Track public feelings and emotions about actual events.

The Challenge

The actual trend calculation, based on static dictionaries, isn’t able to identify new events like an ‘earthquake’ or a
‘political reform’. The “topic-tagging” is static and limited to 9 topics “Science, Economy, Politic, Technology, Entertainment, Sport, Boulevard, Adult and Religion”.

It would be an exhausting task to code every new subtopic or event in a FSM (Finite State Machine).

Therefore the new engine identifies topics by itself. So not only the trend is calculated dynamically also the topic classification is “calculated”.

How is this done?

A simplified breakdown:

NLP (Natural Language Processing) is based on two strategies for text analysis: Tagging via Dictionaries and word / N-gram frequency analysis.

For Example:

Get the latest Flash Player to see this player.

This is an animation of a small section from the Kungle English – Dictionary (about 300.000 words) since January. The daily word count (one hour = one frame) is represented in the column height. The Column color changes from green to red if the word occurred in more than 10 percent of all articles. The overall word count frequency decreases from left to right.

Bigrams:

Get the latest Flash Player to see this player.

This is an even smaller section from the weighted bigram Matrix (about 100.000 x 100.000 words) in the same timeframe. Also this animation is compressed you can identify some horizontal and vertical lines. These lines occur if a topic is heavily discussed.

Three issues with hundreds of similar news publications blocked the front page of Kungle.de. Each publication is interesting and informative by itself  but together they are hiding other noteworthy information.

I concluded that it was about time to build a new subsystem to reduce the amount of identical information. You can still find all articles via the new “related link”.

The new Subsystem “IssueMerger“ now merges  news with similar content. The older news entries are the more likely  they are consolidated to one issue.

For this, I defined a function to calculate the proximity of two entries. (The Result is 1 if two news entries  are identical and 0 if they completely different.)

It is necessary to  build a complete “News Topology” (A Matrix with up to 1.5 million elements) which defines the proximities of all entry combinations.

The calculation for all topics requires up to 40 hours. The Algorithm itself was coded in 80 lines of scala.

You can find a calculated result here:
http://www.kungle.de/Trend/entry/220033

Update 1: In comparison this merge was hand made:

http://www.kungle.de/Trend/entry/225189

Scala is a modern, statically typed language. Its bytecode is Java-VM compatible and the design is influenced by languages like Haskell or Erlang.

Lift is a Web Framework written in Scala. Lift applications follow the View First pattern. The View First pattern defines a complete separation of presentation and logic.

Although with long experience in business, writing my first web application in Scala/Lift wasn’t an easy task. The combination of functional and object-oriented programming paradigms enables a wide range of new possibilities to compose your source code. It may take a while to understand wiki examples or library documentation, but writing your application in Scala has crucial advantages.

• Less code:
  The elegant Type System and the modern Control Flow Structures reduce your code size dramatically.
• A manageable toolchain:
  Every new tool in your development environment means a new dependency in your application. Every update can cause trouble. To configure these tools you often implant new plugins in your ide which in turn complicates the build process. Typically you stop updating your development environment at a certain project stage.
  In contrary, you can build your Lift applications with vi. In less than 100 lines of code you can create a complete web application!

The result is a fast and flexible development cycle.

You can visit my site at kungle.de. It is an application to identify relevant news. The news rating is statistically calculated by an adaptive network and updated every 30 minutes.

References:
Scala homepage: http://www.scala-lang.org/
Lift homepage: http://liftweb.net/
Introduction “View First Pattern”: http://wiki.liftweb.net/index.php/Lift_View_First

You can now play my 3D memory-game on Ubuntu/Linux.

Make sure you have installed java6 from sun.

You can find it here: http://www.yousry.de/Memory-Game.html

You can find a small bugfix release of the memory-game. Reason was a newly discovered VM coredump under Vista (ogl library conflict).

First results show comparable processing speeds to commercial or proprietary products. A backport to Java5 / Mac was successful. The application may be used on all current desktop operation systems.

In subsequent versions, major functional enhancements are planned. The complex user interface will be simplified.

Tasks until alpha release:

• The user interface must be completed.
• Software testing.

Tasks until beta release:

• Add new encryption algorithms.
• Add simplified user interface.
• Add server functionality.
• Batch processing.
• Create API documentation.

My first simple JavaFX application.

“Verschlüsselung”  – an application for fast encryption and compression.  A usefull tool if you want to transport sensitive data with your flashdrive.

The Memory-Game source code is now available at: http://code.google.com/p/memory-game/. It is published under the GPLv3 license.