yousry.de/blog

If I’m asked to draw an “easy to understand” diagram about the next-generation architecture for Web-Applications, one of my sketches would look like this:

SQL And NoSQL

An obvious question about this picture is:

Why are two different Database-Systems necessary?

My artless answer is as follows:

In some crucial cases a NoSQL Database is considerably less expensive than a SQL Database, but a NoSQL Database cannot completely replace a SQL Database.

For example:

• Try to store millions of Media Files or other large Documents on a distributed RDBMS. Each document has to be quickly accessible and updatable.
• Try to create a very large dynamic key/value table as a replacement for an overgrown in-memory Hashmap.

Of course you can achieve this also with a relational database, but you will be faced with following problems:

• RDBMS BLOBS are slow. You can compensate this problem with database clustering but the typical workaround is to store a link or file-path and use the file system as data storage. In this case you always have to check for inconsistencies.
• You can create a key/value store in your SQL Database. But each access creates a heavy processing overhead. (Transaction handling, logging and versioning.)

NoSQL-Databases are optimized and designed to:

• Store and access large “binary” objects very fast.
• Accomplish fast data replication and use table separation (sharding) between different storages.
• They allow very fast key-value stores.

(Easy schema modification is often stated as NoSQL advantage, but I think this is also a trivial task with DDL.

– Don’t forget to set the initial value of the new column and enable the trigger after the table alteration. )

In contrast there are tasks you should use a SQL – Database:

• Data Presentation: In most cases visual entities (for example table lines and columns), aren’t 1:1 presentations from Database objects. They are sorted, mapped, reduced (filtered) and often consist of joins between different tables. Map-Reduce and sorting can be done efficiently with NoSQL. But joins aren’t supported. Of course you can rewrite the JOIN operation in your application. For example a simplified HashMap JOIN could look like this:

  var tableBMap : HashMap[idxType, Type of tableB] = tableB.foreach( line => tableBMap += (line.joinAttributB -> line))
  tableA.map( line => (line, tableBMap.get(line.joinAttributA)))

  The Disadvantage is, that you have to materialize the complete object / table-line.

• NoSQL = No Transaction. You cannot use NoSQL for reliable database transactions (reliable like in ACID). This is a serious restriction for massive parallel database updates. (Hence, you should avoid programming the popular Bank Account example with a NoSQL Database.)

Why I prefer MongoDB as NoSQL Database System:

• Very Fast: Sequential write and random read operations are done very fast on an “average” Server. (You need a 64 Bit OS if your database is larger than 2GB)
• Scala(Java) Support: Several drivers are available scamongo, mongo-scala-driver or akka-persistence
• Easy to use: There is no setup, parameter or table type “magic”. In less than an hour you can setup a secure and robust Database Server. The client uses JS/JSON.
• A nice community: The Forum is very active. Questions should be answered in a couple of hours.

Addendum to my recent Post: FSM stands for “Flying Spaghetti Monster”

In my recent post I used the abbreviation FSM for “Finite State Machine”.

I was informed that FSM is the common abbreviation for the Flying Spaghetti Monster.

I searched for this term in the Kungle News document-storage and found some evidence for this claim:

Small Screendump

Here is the complete list of references:

Query:

Start Main

New Hist Defined with:

Primary: List(Flying Spaghetti Monster, Pastafarianism, Pastafarian)

Secondary in: List(FSM)

Secondary out: List()

Calculated Interval: { “publishingDate” : { “$gt” : “2010-01-10T23:00:00.000Z” , “$lte” : “2010-03-22T23:00:00.000Z”} , “originalLanguage” : “ENGLISH”}

…

…

2; 0; 19

Topic Connections:

(fsm,1)(monster,1)(store,1)(flying,1)(church,1)(spaghetti,1)

Emotions:

(love,14)(hope,13) (+)/(-) (hate,7)(fear,7)

Public Tendency:

10; 5; 6

Country:

gb; us; no; cn; jp; in; se; au; ru; de; fr; ie; gr; nz; ca;

0; 20; 0; 0; 0; 0; 1; 0; 0; 0; 0; 0; 0; 0; 0;

Publisher Tendency:

(Discover Magazine,3)(Huffingtonpost,2)(The Local Germany,1)(Think Progress,0)(Talking Points Memo,-1)(Crooks and Liars,-3)

Calculation Done

Since January the New “Corpus Engine” is in development and recorded about 302.000 articles. All in all 1.130.000 “news headlines” and summaries were stored since Kungle.de went online.

Now new algorithms were developed to:

• Identify the public opinion about political and economic topics.
• Follow the image status of brands, corporations or companies.
• Track public feelings and emotions about actual events.

The Challenge

The actual trend calculation, based on static dictionaries, isn’t able to identify new events like an ‘earthquake’ or a
‘political reform’. The “topic-tagging” is static and limited to 9 topics “Science, Economy, Politic, Technology, Entertainment, Sport, Boulevard, Adult and Religion”.

It would be an exhausting task to code every new subtopic or event in a FSM (Finite State Machine).

Therefore the new engine identifies topics by itself. So not only the trend is calculated dynamically also the topic classification is “calculated”.

How is this done?

A simplified breakdown:

NLP (Natural Language Processing) is based on two strategies for text analysis: Tagging via Dictionaries and word / N-gram frequency analysis.

For Example:

Get the latest Flash Player to see this player.

This is an animation of a small section from the Kungle English – Dictionary (about 300.000 words) since January. The daily word count (one hour = one frame) is represented in the column height. The Column color changes from green to red if the word occurred in more than 10 percent of all articles. The overall word count frequency decreases from left to right.

Bigrams:

Get the latest Flash Player to see this player.

This is an even smaller section from the weighted bigram Matrix (about 100.000 x 100.000 words) in the same timeframe. Also this animation is compressed you can identify some horizontal and vertical lines. These lines occur if a topic is heavily discussed.

I have updated the kungle.de sitemap without announcement.
I apologize for the trouble caused. Please update your bookmarks.

Null Pointer Exceptions (NPEs) are most common runtime errors in Java (more than 500.000 results in Google). Most Java libraries are interspersed with methods returning null if the computation could not be finished. Invoking method calls on null raises Null Pointer Exceptions.

The best practice in Java to avoid such errors is a test against null.

if(aObject == null) {...

or pass the null test to another function.

For example, Instead of writing:

if(aResultString.equals(“literal”)) {
...

you write:

if(“literal”.equals(aResultString)) {
...

More than eleven years ago (beginning with the java era) a data type existed, avoiding the NPE problem:

data  Maybe a =  Nothing | Just a deriving (Eq, Ord, Read, Show)

A Haskell one-liner ported to Java could have prevented thousands of errors.

Why?

You cannot forget to test against None (unlike the null check) because this would raise a compile time error. Based on Haskells Maybe-Type, Scala defines the Some and None Type with:

final case class Some[+A](x: A) extends Option[A]
...

and

case object None extends Option[Nothing]
...

To test an Option Type against None you write:

val result =  a getOrElse b..

If a is Some(_) than result is _. If result is None, than result is b. The value of result is therefore always defined!

Runtime Errors on my News Page Kungle.de so far

To Keep it short:
None

It is very easy to use Apache Derby with the embedded JDBC driver. The disadvantage of this solution is that you can only open one connection at a time. If you are developing an application with data persistence, it is often helpful to monitor the database from a different application (e.g. the Data Source Explorer from Eclipse). If you are not interested in reading the administration guide (http://db.apache.org/derby/docs/dev/adminguide/) you can follow these simple steps to configure Derby for Network/Client Mode:

Step  1: Download Apache Derby

Download the bin distribution from: http://db.apache.org/derby/derby_downloads.html

Step 2: Install

Unpack the archive in an appropriate application directory.

Step 3: Create a database directory

For example: mkdir db_storage

Step 4: Set environment variables:

Set DERBY_HOME and add its bin directory to your system path. Also amend your CLATHPATH with the Derby jars.

For example on your mac:

export DERBY_HOME=/Users/.../Programme/db-derby-10.5.1.1-bin

export PATH=$PATH:$DERBY_HOME/bin

export CLASSPATH="$DERBY_HOME/lib/derbynet.jar:$DERBY_HOME/lib/derbytools.jar:\ $DERBY_HOME/lib/derbyclient.jar:$DERBY_HOME/lib/derby.jar:$CLASSPATH"

On Windows systems:

SET CLASSPATH="%DERBY_HOME%/lib/derbynet.jar;%DERBY_HOME%/lib/derbytools.jar;%DERBY_HOME%/lib/derbyclient.jar;%DERBY_HOME%/lib/derby.jar;%DERBY_HOME%;%CLASSPATH%"

(You probably want to put it in a script or batch.)

Step 5: Create the database

Open a terminal/shell/command prompt, change into the database directory (Step3) and start ij.

Type:

connect ‘jdbc:derby:simple;create=true’;
exit;

Step 6: Database Configuration

In your database directory create the file derby.properties.

Add the following lines:

derby.connection.requireAuthentication=TRUE
derby.authentication.provider=BUILTIN
derby.user.alice=mypass

Step 7: Start Derby

Type:

startNetworkServer

on your mac or

startNetworkServer.bat

on your Windows system.

Step 8: Create a user schema and add a table

Open a new terminal/shell/command prompt, change into the database directory  and start ij.

Type:

CONNECT ‘jdbc:derby://localhost:1527/simple’ user ‘alice’ password ‘mypass’;
Create schema alice;

create table dummy (id int primary key, text varchar(20));

insert into dummy values (1, ‘eins’);

insert into dummy values (10, ‘zehn’);

insert into dummy values (100, ‘einhundert’);

exit;

Step 9: Connect from your IDE:

you can now use the derby jdbc network client driver to connect to your database.

You can use my small java application “wecker” on Ubuntu.

You can find it here: http://yousry.de/AppWecker.html.

You can now play my 3D memory-game on Ubuntu/Linux.

Make sure you have installed java6 from sun.

You can find it here: http://www.yousry.de/Memory-Game.html

You can find a small bugfix release of the memory-game. Reason was a newly discovered VM coredump under Vista (ogl library conflict).

First results show comparable processing speeds to commercial or proprietary products. A backport to Java5 / Mac was successful. The application may be used on all current desktop operation systems.

In subsequent versions, major functional enhancements are planned. The complex user interface will be simplified.

Tasks until alpha release:

• The user interface must be completed.
• Software testing.

Tasks until beta release:

• Add new encryption algorithms.
• Add simplified user interface.
• Add server functionality.
• Batch processing.
• Create API documentation.