yousry.de/blog

Java applets were originally introduced to provide interactive features for web applications (Wikipedia). With javascript based libraries (jquery, yui), proprietary plugins (flash, silverlight or javaFX) and the oncoming HTML5 this intended purpose disappeared, but java applets remain useful for following reasons:

• Scala/Java is type safe: You can find errors early.
• You can use the same language like you used for your web application.
• The Implementation is browser independent.

Example: Distributed Computation of Perfect Numbers

Definition: Perfect Number (Wikipedia)

Testing numbers for perfection is a simple but process-time consuming task. It isn’t complicated to distribute the calculation on several computers.  You can easily think of other domains where similar types of computations could be useful.

Application Overview

Instead of a javascript a java applet is used to factorize numbers. With the typical features of the lift framework, the application looks as follows:

PerfectNumbers Overview

Details: Server -> Client Communication

The snippet: You can deploy your java/scala applet with your web application.

You can invoke your applet method like a javascript function:

// OnLoad(JsRaw("findFactors('"+ c.toJson +"');"))
   OnLoad(JsRaw(c.toJsCall))

Update your case class:

case class JobPackage(ticket: Int,
                   start: Long,
                   end: Long,
                   master: Long) {
  import net.liftweb.json._
  import net.liftweb.json.JsonDSL._
  def toJson = {
    val json = ("ticket" -> ticket) ~
      ("start" -> start) ~
      ("end"-> end) ~
      ("master" -> master)

    compact(JsonAST.render(json))
  }

  def toJsCall = "$('#some-div').html($('#clientEngine').get(0).doCalc(" + ticket
  + ", " + start + ", "+ end +", " + master +"));"

Client -> Server Communication

Add your ajax call to your applet:

JSONObject serverPackage = new JSONObject();
...
private void sendResult() {
		try {
			String sps = serverPackage.toString();
			getAppletContext().showDocument(
					new URL("javascript:replyResult('replyResult', '" + sps + "')"));
		} catch (MalformedURLException e) {
			errorText = "Error: " + e.getMessage();
		}
	}

and dispatch the result:

  def PMCall(s: String) = { 

    logger.info("PMCall "  + s)

    val cResJs = net.liftweb.json.JsonParser.parse(s)
    val ticket  = (cResJs \\ "ticket").extract[Int]

    // With 2.8: facs.extract[List[Long]]
    val facs = for{
      JInt(fac) <- cResJs \\ "factors"
    } yield fac.extract[Long]

    PerfectNumberMaster ! PerfectNumberMaster.CalcDone(ticket , facs)
    Empty
  }

Download and run the example

The Sourcecode is available here.
Type: mvn jetty:run and open the url http://localhost:8080

The Following directed graph shows the “top 5″ countries in the news (grouped by countries).

The query was done for all stored articles since January 2010 until now.

The edges are labelled with the percentage share of all articles for the origin country:

Q1 Country Interest

You can find the original picture here.

If I’m asked to draw an “easy to understand” diagram about the next-generation architecture for Web-Applications, one of my sketches would look like this:

SQL And NoSQL

An obvious question about this picture is:

Why are two different Database-Systems necessary?

My artless answer is as follows:

In some crucial cases a NoSQL Database is considerably less expensive than a SQL Database, but a NoSQL Database cannot completely replace a SQL Database.

For example:

• Try to store millions of Media Files or other large Documents on a distributed RDBMS. Each document has to be quickly accessible and updatable.
• Try to create a very large dynamic key/value table as a replacement for an overgrown in-memory Hashmap.

Of course you can achieve this also with a relational database, but you will be faced with following problems:

• RDBMS BLOBS are slow. You can compensate this problem with database clustering but the typical workaround is to store a link or file-path and use the file system as data storage. In this case you always have to check for inconsistencies.
• You can create a key/value store in your SQL Database. But each access creates a heavy processing overhead. (Transaction handling, logging and versioning.)

NoSQL-Databases are optimized and designed to:

• Store and access large “binary” objects very fast.
• Accomplish fast data replication and use table separation (sharding) between different storages.
• They allow very fast key-value stores.

(Easy schema modification is often stated as NoSQL advantage, but I think this is also a trivial task with DDL.

– Don’t forget to set the initial value of the new column and enable the trigger after the table alteration. )

In contrast there are tasks you should use a SQL – Database:

• Data Presentation: In most cases visual entities (for example table lines and columns), aren’t 1:1 presentations from Database objects. They are sorted, mapped, reduced (filtered) and often consist of joins between different tables. Map-Reduce and sorting can be done efficiently with NoSQL. But joins aren’t supported. Of course you can rewrite the JOIN operation in your application. For example a simplified HashMap JOIN could look like this:

  var tableBMap : HashMap[idxType, Type of tableB] = tableB.foreach( line => tableBMap += (line.joinAttributB -> line))
  tableA.map( line => (line, tableBMap.get(line.joinAttributA)))

  The Disadvantage is, that you have to materialize the complete object / table-line.

• NoSQL = No Transaction. You cannot use NoSQL for reliable database transactions (reliable like in ACID). This is a serious restriction for massive parallel database updates. (Hence, you should avoid programming the popular Bank Account example with a NoSQL Database.)

Why I prefer MongoDB as NoSQL Database System:

• Very Fast: Sequential write and random read operations are done very fast on an “average” Server. (You need a 64 Bit OS if your database is larger than 2GB)
• Scala(Java) Support: Several drivers are available scamongo, mongo-scala-driver or akka-persistence
• Easy to use: There is no setup, parameter or table type “magic”. In less than an hour you can setup a secure and robust Database Server. The client uses JS/JSON.
• A nice community: The Forum is very active. Questions should be answered in a couple of hours.

Addendum to my recent Post: FSM stands for “Flying Spaghetti Monster”

In my recent post I used the abbreviation FSM for “Finite State Machine”.

I was informed that FSM is the common abbreviation for the Flying Spaghetti Monster.

I searched for this term in the Kungle News document-storage and found some evidence for this claim:

Small Screendump

Here is the complete list of references:

Query:

Start Main

New Hist Defined with:

Primary: List(Flying Spaghetti Monster, Pastafarianism, Pastafarian)

Secondary in: List(FSM)

Secondary out: List()

Calculated Interval: { “publishingDate” : { “$gt” : “2010-01-10T23:00:00.000Z” , “$lte” : “2010-03-22T23:00:00.000Z”} , “originalLanguage” : “ENGLISH”}

…

…

2; 0; 19

Topic Connections:

(fsm,1)(monster,1)(store,1)(flying,1)(church,1)(spaghetti,1)

Emotions:

(love,14)(hope,13) (+)/(-) (hate,7)(fear,7)

Public Tendency:

10; 5; 6

Country:

gb; us; no; cn; jp; in; se; au; ru; de; fr; ie; gr; nz; ca;

0; 20; 0; 0; 0; 0; 1; 0; 0; 0; 0; 0; 0; 0; 0;

Publisher Tendency:

(Discover Magazine,3)(Huffingtonpost,2)(The Local Germany,1)(Think Progress,0)(Talking Points Memo,-1)(Crooks and Liars,-3)

Calculation Done

Since January the New “Corpus Engine” is in development and recorded about 302.000 articles. All in all 1.130.000 “news headlines” and summaries were stored since Kungle.de went online.

Now new algorithms were developed to:

• Identify the public opinion about political and economic topics.
• Follow the image status of brands, corporations or companies.
• Track public feelings and emotions about actual events.

The Challenge

The actual trend calculation, based on static dictionaries, isn’t able to identify new events like an ‘earthquake’ or a
‘political reform’. The “topic-tagging” is static and limited to 9 topics “Science, Economy, Politic, Technology, Entertainment, Sport, Boulevard, Adult and Religion”.

It would be an exhausting task to code every new subtopic or event in a FSM (Finite State Machine).

Therefore the new engine identifies topics by itself. So not only the trend is calculated dynamically also the topic classification is “calculated”.

How is this done?

A simplified breakdown:

NLP (Natural Language Processing) is based on two strategies for text analysis: Tagging via Dictionaries and word / N-gram frequency analysis.

For Example:

Get the latest Flash Player to see this player.

This is an animation of a small section from the Kungle English – Dictionary (about 300.000 words) since January. The daily word count (one hour = one frame) is represented in the column height. The Column color changes from green to red if the word occurred in more than 10 percent of all articles. The overall word count frequency decreases from left to right.

Bigrams:

Get the latest Flash Player to see this player.

This is an even smaller section from the weighted bigram Matrix (about 100.000 x 100.000 words) in the same timeframe. Also this animation is compressed you can identify some horizontal and vertical lines. These lines occur if a topic is heavily discussed.

A short summary about the new vote feature.

Last Week Results

Total Votes: 623

Votes by Topic:

Pie Chart:

Why should you vote?

Because with all collected votes it will be possible to calculate better results for following matters:

• Offer new subtopics
  For example visitors are especially interested in Science/Health, Entertainment/Bollywood or Politic/War. Subtopics will be implemented in future versions of  the application.
• Forecast the period of time a news entry is relevant
  First indications are, that Boulevard-News have the longest lasting relevance.
• Estimate publisher/editor reputations
  More votes mean a higher reputation for a publisher.

Vote Guidance

You cast a vote by pressing the check-mark on the right-hand side of the news-entry.

You have ten votes in total.

Your Privace

Kungle.de will not store any personal information about your visit.

In Detail

Kungles web server like any other web server on the internet stores your IP in a log file. Independent from this logfile  Kungles application server stores your selection and votes temporarily in memory.  It is very difficult to connect these two information sources to generate an individualized user profile.

Finally the database counts the votes passed by the application server and logs theses events independent from the two other logs.

I’m not interested in individual user behavior. 

Recent events showed that you should be very careful with your personal data:

http://www.kungle.de/Trend/entry/275800

http://www.kungle.de/Trend/entry/275766

http://www.kungle.de/Trend/entry/275526

Therefore I see the necessity to inform you about Kungles information processing.

Three issues with hundreds of similar news publications blocked the front page of Kungle.de. Each publication is interesting and informative by itself  but together they are hiding other noteworthy information.

I concluded that it was about time to build a new subsystem to reduce the amount of identical information. You can still find all articles via the new “related link”.

The new Subsystem “IssueMerger“ now merges  news with similar content. The older news entries are the more likely  they are consolidated to one issue.

For this, I defined a function to calculate the proximity of two entries. (The Result is 1 if two news entries  are identical and 0 if they completely different.)

It is necessary to  build a complete “News Topology” (A Matrix with up to 1.5 million elements) which defines the proximities of all entry combinations.

The calculation for all topics requires up to 40 hours. The Algorithm itself was coded in 80 lines of scala.

You can find a calculated result here:
http://www.kungle.de/Trend/entry/220033

Update 1: In comparison this merge was hand made:

http://www.kungle.de/Trend/entry/225189

With the 73th version of the Kungle trend engine it is now possible to track a meaningful trend again.

It was difficult to rebuild a functional trend prediction after the decision to remove the search engine results from my calculations.

Cause of the decision:

The search terms got more and more insignificant. For Example on Sep. 29 (The Tsunami Day)  the highest ranked search phrase was “pink toed tarantula”.  Other examples were “who is mysterion” or “out of memory at line 130″ !

The Trend Analysis is now extremely fast. The response time is about  30 minutes after an occurrence. In contrary, searchengine results are delayed up to four hours!

I’m now using microblogs, blogs, and newsfeeds as datasource.

The Trend Score is calculated by following factors:

• Time of publication
• Frequency of occurrence
• Subject significance (by Dictionary)
• Subject insignificance (by Dictionary)
• Registration Date
• Publisher significance

Votes:

I have added a voting mechanism to compare the trend results with my readers opinions. To offer an incentive I have added a Trend bonus for every vote.

Next Steps:

Dependent on the success of Kungle.de and the related earnings:

1. Switch to bigger hardware.
2. Improve the topic identification.
3. Deep text analysis.
4. Global Impact calculation

Improved user interaction:

Kungle.de is a Scala/Lift Project based on HTML and Ajax.  Next Steps would require to build Restful Web Service and a Rich Internet Application. for a better user experience.

I have just uploaded the first Kungle release with an automatic translation of foreign languages and an improved utf encoder.

Click on the [Translated] sign to read the translation.

Hacker and Botnets had been very alive this week. Several night-shifts were necessary to keep kungle alive. However some drop outs were unavoidable.

Features:

The development of  the translation feature has begun. The results are adequate so far.

Some fresh statistics from my database:

New: I’m now following political, cultural and news blogs.

If you want to participate create your own free blog (for example here wordpress.com or here www.blogger.com) and leave me a note or comment with your address of your website.

If you like, use your native language. I will add a new feature  “automatic translation” in the next few days.