Sunday, June 26, 2011

JSON Compression algorithms

About

JSON (Java Script Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It can be used as a data interchange format, just like XML. When comparing JSON to XML, it has several advantages over the last one. JSON is really simple, it has a self-documenting format, it is much shorter because there is no data configuration overhead. That is why JSON is considered a fat-free alternative to XML.

However, the purpose of this post is not to discuss the pros and cons of JSON over XML. Though it is one of the most used data interchanged format, there is still room for improvement. For instance, JSON uses excessively quotes and key names are very often repeated. This problem can be solved by JSON compression algorithms. There are more than one available. Here you'll find an analysis of two JSON compressors algorithms and a conclusion whether JSON compression is useful and when it should be used.

Compressing JSON with CJSON algorithm

CSJON compress the JSON with automatic type extraction. It tackles the most pressing problem: the need to constantly repeat key names over and over. Using this compression algorithm, the following JSON:

[
  { // This is a point
    "x": 100, 
    "y": 100
  }, { // This is a rectangle
    "x": 100, 
    "y": 100,
    "width": 200,
    "height": 150
  },
  {}, // an empty object
]
Can be compressed as:

{
  "templates": [ 
    [0, "x", "y"], [1, "width", "height"] 
  ],
  "values": [ 
    { "values": [ 1,  100, 100 ] }, 
    { "values": [2, 100, 100, 200, 150 ] }, 
    {} 
  ]
}
The more detailed description of the compression algorithm, along with the source code can be found here:

Compressing JSON with HPack algorithm

JSON.hpack is a lossless, cross language, performances focused, data set compressor. It is able to reduce up to 70% number of characters used to represent a generic homogeneous collection. This algorithms provides several level of compression (from 0 to 4). The level 0 compression performs the most basic compression by removing keys (property names) from the structure creating a header on index 0 with each property name. Next levels make it possible to reduce even more the size of the JSON by assuming that there are duplicated entries.

For the following JSON:

[{
  name : "Andrea",
  age : 31,
  gender : "Male",
  skilled : true
}, {
  name : "Eva",
  age : 27,
  gender : "Female",
  skilled : true
}, {
  name : "Daniele",
  age : 26,
  gender : "Male",
  skilled : false
}]
the hpack algorithm produces a compressed version which looks like this:

[["name","age","gender","skilled"],["Andrea",31,"Male",true],["Eva",27,"Female",true],["Daniele",26,"Male",false]]
More details about hpack algorithm can be found at project home page.

Analysis

The purpose of this analysis is to compare each of the described JSON compressor algorithms. For this purpose we will use 5 files with JSON content having different dimensions, varying from 50K to 1MB. Each JSON file will be served to a browser using a servlet container (tomcat) with the following transformations:

  • Unmodified JSON - no change on the server side
  • Minimized JSON - remove whitespaces and new lines (most basic js optimization)
  • Compressed JSON using CJSON algorithm
  • Compressed JSON using HPack algorithm
  • Gzipped JSON - no change on the server side
  • Gzipped and minimized JSON
  • Gzipped and compressed using CJSON algorithm
  • Gzipped and compressed using HPack algorithm

Results

This table contains the results of the benchmark. Each row of the table contains one of the earlier mentioned transformation. The table has 5 columns, one for each JSON file we process.
\
json1 json2 json3 json4 json5
Original JSON size (bytes) 52966 104370 233012 493589 1014099
Minimized 33322 80657 180319 382396 776135
Compress CJSON 24899 48605 108983 231760 471230
Compress HPack 5727 10781 23162 49099 99575
Gzipped 2929 5374 11224 23167 43550
Gzipped and Minimized 2775 5035 10411 21319 42083
Gzipped and compressed with CJSON 2568 4605 9397 19055 37597
Gzipped and compressed with HPack 1982 3493 6981 13998 27358

Relative size of transformations(%)

The relative size of transformation graphic is useful to see if the size of the json to compress affects the efficiency of compression or minimization. You can notice the following:
  • the minimization is much more efficient for smaller files. (~60%)
  • for large and very large json files, the minimization has constant efficiency (~75%)
  • compressors algorithms has the same efficency for any size of json file
  • CJson compressing algorithm is less efficient (~45%) than hpack algorithm (~8%)
  • CJson compressing algorithm is slower than hpack algorihtm
  • Gzipped content has almost the same size as the compressed content
  • Combining compression with gzip or minimization with gzip, doesn't improve significantly efficiency (only about 1-2%)

Conclusion

Both JSON compression algorithms are supported by wro4j since version 1.3.8 by the following processors: CJsonProcessor & JsonHPackProcessor. Both of them provide the following methods: pack & unpack. The underlying implementation uses Rhino engine to run the javascript code on the serverside.

JSON Compression algorithms considerably reduce json file size. There a several compression algorithms. We have covered two of them: CJson and HPack. HPack seems to be much more efficient than CJson and also significantly faster. When two entities exchange JSON and the source compress it before it reach the target, the client (target) have to apply the inverse operation of compression (unpacking), otherwise the JSON cannot be used. This introduce a small overhead which must be taken into account when deciding if JSON compression should be used or not.

When gziping of content is allowed, it has a better efficiency than any other compression algorithm. In conclusion, it doesn't worth to compress a JSON on the server if the client accept the gzipped content. The compression on the server-side does make sense when the client doesn't know how to work with gzipped content and it is important to keep the traffic volue as low as possible (due to cost and time).

Another use-case for JSON compression algorithm is sending a large JSON content from client to server (which is sent ungzipped). In this case, it is important to unpack the JSON content on the server before consuming it.

119 comments:

  1. This site use full to all around the would, use it develop both.........
    this site use of all the would and system news and helps known every one...........
    System Optimization

    ReplyDelete
    Replies
    1. I really appreciate you for all the valuable information about algorithms you are providing us through your blog.

      Thanks
      SEO Company New York

      Delete
    2. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Front end developer learn from Javascript Training in Chennai . or learn thru JavaScript Online Training in India. Nowadays JavaScript has tons of job opportunities on various vertical industry. JavaScript Training in Chennai

      Delete
  2. Very cool. I just heard of MesssagePack today, and decided to look around and see what else was available out there. I hadn't ever thought about optimizing my JSON because I assumed since it was so much more efficient than XML, it was good enough. Thanks again.

    ReplyDelete
  3. I have been looking around for this kind of information. Will you post some more in future? I’ll be grateful if you will.
    SEO Services New York

    ReplyDelete
  4. Appreicate your thoughts, Im not always in agreement, but you do cause a peron to think keep blogging!carpet cleaning adelaide

    ReplyDelete
  5. Appreicate your thoughts, Im not always in agreement, but you do cause a peron to think keep blogging!carpet cleaning adelaide

    ReplyDelete
  6. If truth be told informative and valuable detail is here.
    air compressor best price

    ReplyDelete
  7. An unbelievable blog. This blog will indisputably be definitely recommended to my friends as well.
    Joycelyn Corria

    ReplyDelete
  8. Would you please consider removing the spam comments? They're a distraction, and they encourage spammers to keep spamming the rest of our blogs.

    Otherwise, thanks for the JSON compression comparisons.

    ReplyDelete
  9. Optimising JSON is also of interest to me (see https://sourceforge.net/projects/ojson/ for an opensource Java API I've developed) I'm experimenting with eliminating repeated keys using embedded tags and templating, but also using tagged string references so that examples such as [ "a repeated string", "a repeated string" ] is mapped to [ "@1:a repeated string", "@1" ] which can be easily mapped back to the original text.

    It appears my approach is mostly aligned with the hpack algorithm for optimising keys. I'd be interested to obtain a copy of the 5 json texts used in the results above so that I can see what figures I can produce.

    ReplyDelete
    Replies
    1. A quick follow up with at least one test result, using the json.hpack test data I discovered by following the above link https://github.com/WebReflection/json.hpack/tree/master/test (the file 5000.txt) I'm getting the following test result using my 'ojson' API:

      json.hpack-5000: Reduction: 76.0% from [776133] to optimised size [186598]

      I'm also mapping the 'ojson' output back to plain json and using this java package:

      org.skyscreamer.jsonassert.JSONAssert

      to verify the original hpack json is indeed being restored (from json -> ojson -> newJson):

      JSONAssert.assertEquals(json, newJson, JSONCompareMode.STRICT);

      Hope this is of interest.

      Delete
  10. Web site optimization produces highly competitive web sites that out-perform on every measure; traffic, speed, conversion rates, sell-throughs, and, most importantly, return on your investment.Thanks for your useful information.
    Web Developers Adelaide

    ReplyDelete
  11. Thank you for this amazing Json tutorial. Very helpful. Check Our Service Below. Hope it will you too. Thanks
    SEO Services
    Local SEO Services
    Link Building Services
    Web Design & Development Services
    WordPress Development Services

    ReplyDelete
  12. Thank you for sharing this informative and interesting article. Keep up the good work!


    Melbourne seo services

    ReplyDelete
  13. I read this article. I think You put a lot of effort to create this article. I appreciate your work.
    thesis Writing Service

    ReplyDelete
  14. Nice post. By reading your blog, i get inspired and this provides some useful information. Thank you for posting this exclusive post for our vision. 
    Microsoft azure training in Bangalore
    Power bi training in Chennai

    ReplyDelete
  15. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

    DevOps is currently a popular model currently organizations all over the world moving towards to it. Your post gave a clear idea about knowing the DevOps model and its importance.

    Good to learn about DevOps at this time.


    devops training in chennai | devops training in chennai with placement | devops training in chennai omr | devops training in velachery | devops training in chennai tambaram | devops institutes in chennai | devops certification in chennai | trending technologies list 2018

    ReplyDelete
  16. Such a wonderful blog on Machine learning . Your blog have almost full information about Machine learning .Your content covered full topics of Machine learning that it cover from basic to higher level content of Machine learning . Requesting you to please keep updating the data about Machine learning in upcoming time if there is some addition.
    Thanks and Regards,
    Machine learning tuition in chennai
    Machine learning workshops in chennai
    Machine learning training with certification in chennai

    ReplyDelete
  17. you exlained very well abouut the resource.thanks for sharing this useful infromation.good work.keep going.

    Apple service center in Chennai
    Apple service center

    ReplyDelete
  18. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
    Thanks & Regards,
    VRIT Professionals,
    No.1 Leading Web Designing Training Institute In Chennai.

    And also those who are looking for
    Web Designing Training Institute in Chennai
    SEO Training Institute in Chennai
    Photoshop Training Institute in Chennai
    PHP & Mysql Training Institute in Chennai
    Android Training Institute in Chennai

    ReplyDelete
  19. Nice post!Everything about the future(học toán cho trẻ mẫu giáo) is uncertain, but one thing is certain: God has set tomorrow for all of us(toán mẫu giáo 5 tuổi). We must now trust him and in this regard, you must be(cách dạy bé học số) very patient.

    ReplyDelete
  20. very good post!!! Thanks for sharing with us... It is more useful for us...
    Selenium Training in Chennai | SeleniumTraining Institute in Chennai

    ReplyDelete
  21. Time is free but it's priceless(khóa học toán tư duy) . You cannot own it, but you can use it(cách dạy bé học số) . You can use it, but you can't keep it(toán tư duy logic là gì). Once you lose it, you will not be able to get it back.

    ReplyDelete
  22. A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  23. A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  24. A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  25. Awesome information.
    such an useful article.
    thanks for posting.keep sharing.
    Best Java training in Bengaluru

    ReplyDelete
  26. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  27. It’s always so sweet and also full of a lot of fun for me personally and my office colleagues to search you blog a minimum of thrice in a week to see the new guidance you have got.
    fire and safety course in chennai
    safety course in chennai

    ReplyDelete
  28. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  29. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  30. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  31. Do you mind if I quote a couple of your posts as long as I provide credit and sources back to your blog?
    safety course in chennai
    nebosh course in chennai

    ReplyDelete
  32. Really useful information. Thank you so much for sharing.It will help everyone.Keep Post.

    Cloud Computing Interview Questions and Answers


    ReplyDelete
  33. Website Planning Best institute for digital marketing course in delhi. Initialisation of Digital Marketing.. Website Creation. Content Writing. Search Engine Optimization. Local Seo. Google Webmaster. Bing Webmaster.
    Digital Marketing training in Laxmi Nagar

    ReplyDelete
  34. மகிழ்ச்சியான மற்றும் மகிழ்ச்சியான நாள். கட்டுரையைப் பகிர்ந்தமைக்கு மிக்க நன்றி

    máy phun tinh dầu

    máy khuếch tán tinh dầu tphcm

    máy khuếch tán tinh dầu hà nội

    máy xông phòng ngủ

    ReplyDelete
  35. The article is so informative. This is more helpful for our
    software testing training courses
    selenium classes Thanks for sharing

    ReplyDelete
  36. It is a very popular application which provides you to download videos, songs, mashups, remixes, trailers in your mobile , and off course in any quality. It contains many videos and of many types .
    like funny videos, motivational videos, songs lyrics, trailer and many more.
    Want to watch movie in slow connection,
    Vidmate app,
    Vidmate app to watch videos,
    vidmate,
    Vidmate Features,
    Vidmate Download,
    Multi sSource Downloader,
    cm security antivirus,
    leaked Movie Online,
    Why do people prefer Vidmate App.

    ReplyDelete
  37. This is a nice Site to watch out for and we provided information on
    vidmate make sure you can check it out and keep on visiting our Site.

    ReplyDelete
  38. I feel this article have given such a lot of vital info for me. And I am satisfied studying your article. However wanna commentary on few general things, The website style is ideal, the articles are truly nice.
    Interior Designers in Chennai | Interior Decorators in Chennai | Best Interior Designers in Chennai | Home Interior designers in Chennai | Modular Kitchen in Chennai

    ReplyDelete
  39. Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows.

    ReplyDelete
  40. This is really a big and great source of information. We can all contribute and benefit from reading as well as gaining knowledge from this content. Just amazing
    experience. Thanks for sharing such nice information.
    Event Management in Pondicherry | Wedding Decorators in Trichy | Wedding Photographers in Trichy | Wedding Planner in Pondicherry | Wedding Decorators in Pondicherry | Candid Photography Pondicherry | Wedding Photographers in Pondicherry

    ReplyDelete
  41. I think you did a super job in giving an explanation of JSON. This info will help many others. discover.com/activate

    ReplyDelete

  42. Nice! you are sharing such helpful and easy to understandable blog in decoration. i have no words for say i just say thanks because it is helpful for me.

    robotic process automation companies in us
    Robotic Process Automation in us
    machine maintanance in us
    erp in chennai
    mobility software companies in chennai
    erp providers in us

    ReplyDelete
  43. <a href="https://vidmate.vin/

    ReplyDelete
  44. We are leading supplier and manufacturer of filler masterbatches all over India
    We are providing you best quality filler masterbatch

    ReplyDelete
  45. Thanks a lot for writting such a great article. It's really has lots of insights and valueable informtion.
    If you wish to get connected with AI world, we hope the below information will be helpful to you.
    Python Training Institute in Pune
    Python Interview Questions And Answers For Freshers
    Data -Science
    ML(Machine Learning) related more information then meet on EmergenTeck Training Institute .
    Machine Learning Interview Questions And Answers for Freshers
    Thank you.!

    ReplyDelete
  46. Home Salon's, Certified Beauticians are highly experienced in her core domain and keep an update of most recent trend styling
    and hair shading, which can give a surprising and new look to ladies. In the Digital Era, the Beauty Parlour Services at home
    is now quickly accessible at your fingertips. Why go to traditional salon services and wait in queues when you can call a salon
    at home.
    Home Salon

    ReplyDelete
  47. thanks for your details It's very amazingweb design company in velacheryQbigpro branding solution is the best web design company in velachery web design company in velachery.we will create the web site and managing the site.we will help for all business.website is very important for all business.

    ReplyDelete
  48. If you are looking for Best Gyms in Ghaziabad then click on the given link.

    ReplyDelete
  49. Thanks for sharing such a wonderful blog on Mean Stack .This blog contains so much data about Mean Stack ,like if anyone who is searching for the Mean Stack data,They will easily grab the knowledge of from this.Requested you to please keep sharing these type of useful content so that other can get benefit from your shared content.
    Thanks and Regards,
    Mean Stack training in Chennai
    Best mean stack training in Chennai
    Top Mean stack raining in Chennai
    Course fees for Mean stack in Chennai
    Mean stack training fees in Velachery, Chennai

    ReplyDelete
  50. thanks for your information really good and very nice web design company in velachery

    ReplyDelete
  51. While composing the last paper of your last B.Com test, a plenty of considerations experience your psyche. A great deal of these contemplations incorporate plans about resting for a whole week or celebrating for a whole week, contingent upon your individual inclinations. Be that as it may, trust us, none of that is really going to happen in light of the fact that when you complete your tests your folks, relatives, neighbours, and even your Facebook companions will begin getting some information about your feasible arrangements. What's more, don't mistake them for your gathering or dozing plans since they are alluding to your vocation Career after B com plans. In the present focused world, you are offered with many profession improving courses. On the off chance that you are not happy with the profession or course you decided for yourself at that point there are some present moment yet high worth – low speculation courses accessible in the market.

    ReplyDelete