The Demo is ready

I am thrilled to announce the first tech demo of the new WebFusion (codenamed odin) is completed.

The address of the demo is: http://www.shisoft.net/

This demo exhibits the function of recommend feeds (from about 1,000 twitter accounts I followed in this demo) by user selected categories. This function is an imitation of mobile app Zite, which have been acquired by Flipboard[1]. You can simply type your thoughts in the text box, the system will give you hints automatically.

Snip20160108_1

You can then use your mouse or keyboard to select the item. If you prefer to let the system suggest categories combined feeds, feel free to select more. Then, click the search button to submit your query.

Snip20151222_30

Your recommendation of feeds should be displayed in no time. In the image above, you can see four feeds from twitter. You can also scroll the page down to get more until there is no more feeds in the database that related to your selected categories. In your attempts, the results should vary because the system is receiving and processing new feeds published by other persons in real time, you can always see the newest feeds once it arrived in the system.

This system also supports recommend feeds by user inputs that categories does not exists in my database (updated at 8 January 2016). Every internal categories was prepended with symbol "♦", indicates that this category has comprehensive features in my database that the system can provide accurate recommendations. A category that does not prepended with the symbol indicates that the system will use features according to the words user inputed to provide recommendations which it may not very accurate compare to internal categories.

You can switch categories by input new names in the text box, or you can simply click on the links below the feeds that indicated as "Related Categories".

This is NOT a search engine. The system considers both keywords and the meaning of the contents to categories, including probability distribution on topics.

This function is still experimental. After filtering, there are over 100,000 categories that the user can select for recommendations. The data training processes are all unsupervised, it is impossible for me to check the accuracy for all of them. As far as I know, there are some flaws right now which makes it imperfect. There are a lot of overlapping categories that have almost the same features. Their results of recommendations are identical. It can distinguish Apple computer and apple on trees, but it cannot tell the difference from "Google" and "Microsoft". Because their topics are the same and their keywords are almost identical. Due to all of the categories and its data comes from Wikipedia dumps, the quality cannot be controlled, some categories are also overfitted by more popular subcategories.

This function is a part of the new WebFusion. In the demo version, "Share" and "Reply" button are not functional right now, they only use as placeholder.

The dedicated server for this demo is in Shanghai right now because this system is huge and I cannot afford virtual servers in some cloud provider like DigitalOcean. For the concern of the national great firewall, U.S users may suffer time out exceptions or service temporally unavailable. I am trying to overcome these problems. If you have any difficulties on reviewing this demo, please don't be hesitated to contact me.

* Right now I have made my best efforts on improving the experience in your visiting this demo by redirecting traffic through 3 more servers. The outbound server is in San Francisco with this blog site, the origin server is still in Shanghai.

* I have already use shadowsocks to replace ssh for internal proxy service to twitter and other banned feed source in China. But the route to my shadowsocks server is also not stable. I tried to solve this issue by using HaProxy to load-balance and failover 9 different routes to the shadowsocks server in U.S, it works.

* It have been a long time after I first start to collect twitter feeds and compute their features. I adjusted the parameters which slightly more strict so that the recommendations for categories should be more accurate.

* You can get more technical details from here

The basic concepts of designing the new WebFusion

After one year of pause on developing WebFusion. I can look back and pick up some defects on the original design. When the defects in the design accumulated to an unacceptable rate, I decide to start it over with new one.

New WebFusion project codenamed odin, will use Clojure as the main programming language on server side. In the nearly one year work day with Clojure, I found its pretty and elegant syntax, functional programming can boost the development with clear expressions. Another reason I choose Clojure rather than other Lisp or FP languages is because I can make use of the code bases from old WebFusion when I want to. Another benefits I can take from JVM based programming language is that I can use tones of library that proven stable, high-performance and well documented.

Odin is fully distributed and event driven. That means the each components of odin can run on different servers in any number of processes and working threads. They use Remote Function Invocation from a project that I was created in my employed work time named cluster-connector to communicate with each other. The inner-interactive is event driven, which means odin will not open new thread pool for new users, it will dispatch tasks in one thread to make full use of it and with concurrency benefits. It works like a reactor to dispatch messages by using a project that based on Project Reactor named meltdown for asynchronous.

Odin also benefits from Clojure by using some language features like macro and multi-methods. Macro can make the compiler generate codes from your code, Which means you direct the compiler to do some heavy job to generate repeatedly code or your own DSL to replace complex syntax that should write in native form. And you can even generate macros from macros make works easier for humans. Multi-method is a feature that reflects object-oriented programming, but in a flexible way. It use a dispatch function to determinate the pathway of reaching the actual function rather than fixed class hierarchy. It dispatch to methods according to its parameter, allows me to configure the behavior for each analogous modules in files.

All of the design was tested separately, I am trying to put them together to make sure the system works as expected. The results are not sure, I will report in another article in this blog when it is done.

Things happened in recent 1 month

At last, I did quit my job.

The company I used to work for did make a lot of interests by selling mobile phone game virtual currency (About 2,600,000 USD cash flow per month), as we called gems. But they still have not figure out how to divide the incomes with their publisher. So I still have no profit sharing.

As they told me the amount of profits I will get can only cover my tuition fee for one year, that does not include the cost of food, lodging and equipments, I have low passion on dealing with my works. Instead, I start to thinking about make some design that noteworthy in my resume.

Reviewed the source code of old WebFusion project that was written in Java. I found that I can make a better architecture by using the knowledge I have learned from my employed works. Improving the performance and robustness of the whole system. So, I announced the cancellation of the old project and start new one from scratch codenamed odin. The new design will be revealed in other article in this blog.

I have meet my study buddy and best friend, they informed me that I may off schedule on the plan to prepare for language tests. That is not a good news to me but I have confidence to catch up the schedule. I know my weaknesses, what I need to do is keep practicing.

 

Hello world!

I created this blog, to take notes from my software development, daily studying and language practice. Considering the dishonoured network condition in my country, which is so-called the only communism region in this world, I setup this site in the US, avoiding unpleasants paperwork.

As you can see I am not a native English speaker. But I intend to improve my skills for better understanding of the material I read and represent my thought. You can consider it as the struggle from a young and angry man which have early midlife crisis.

A little more about myself. I am a software engineer now working for a game company which made the potential most popular League of Legends theme mobile game in China. Responsible for architecture of server clusters. I built a middleware that make the cluster scalable to bear the increasing players only with deploying a single machine. This design helped my company reduced the costs on hardware and maintenance manpower. I also built the backstage tools for operation team alone. Those works consists of business intelligence,logging system and player/content management systems. Operation team use it as reliable sources for analyzing data and decision making. When our chief server programmer was too busy, I helped him to finish some work like VIP system (which is the system for stimulating players to pay more in our game by providing various discount) and messaging system. I also skilled in iOS programming, when the client team have some trouble with native layer, I also provide supports like In App Purchase.

Before employed, I was busy on a project name WebFusion, integrating information from various web services. Providing instant messaging, micro-blogging, e-mail and content recommending systems. If you are interested, take a look at https://www.shisoft.net/. This project is paused due to lacking of time to maintain when employed.

I am not pleased with my job right now. It was interesting in the beginning because I learned new stuff and did improved my skills. But that feeling faded out when I found most of my work was something like CRUD, I found it boring. I really want to continue working on WebFusion project in my spare time but when employed, the job squeezed out my time. Building such complex system requires knowledge that I need a lot of time to learn. Then I realized that I  will never have a chance to create things I really care about. So I decide to quit my job and get a chance for overseas studying. My boss said I am crazy, but really, I DON'T CARE. I had enough with those incessant unpaid overtime works and there is no anything like profit share contract for me, so beat it!