Sunday, June 26, 2011

JSON Compression algorithms

About

JSON (Java Script Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It can be used as a data interchange format, just like XML. When comparing JSON to XML, it has several advantages over the last one. JSON is really simple, it has a self-documenting format, it is much shorter because there is no data configuration overhead. That is why JSON is considered a fat-free alternative to XML.

However, the purpose of this post is not to discuss the pros and cons of JSON over XML. Though it is one of the most used data interchanged format, there is still room for improvement. For instance, JSON uses excessively quotes and key names are very often repeated. This problem can be solved by JSON compression algorithms. There are more than one available. Here you'll find an analysis of two JSON compressors algorithms and a conclusion whether JSON compression is useful and when it should be used.

Compressing JSON with CJSON algorithm

CSJON compress the JSON with automatic type extraction. It tackles the most pressing problem: the need to constantly repeat key names over and over. Using this compression algorithm, the following JSON:

[
  { // This is a point
    "x": 100, 
    "y": 100
  }, { // This is a rectangle
    "x": 100, 
    "y": 100,
    "width": 200,
    "height": 150
  },
  {}, // an empty object
]
Can be compressed as:

{
  "templates": [ 
    [0, "x", "y"], [1, "width", "height"] 
  ],
  "values": [ 
    { "values": [ 1,  100, 100 ] }, 
    { "values": [2, 100, 100, 200, 150 ] }, 
    {} 
  ]
}
The more detailed description of the compression algorithm, along with the source code can be found here:

Compressing JSON with HPack algorithm

JSON.hpack is a lossless, cross language, performances focused, data set compressor. It is able to reduce up to 70% number of characters used to represent a generic homogeneous collection. This algorithms provides several level of compression (from 0 to 4). The level 0 compression performs the most basic compression by removing keys (property names) from the structure creating a header on index 0 with each property name. Next levels make it possible to reduce even more the size of the JSON by assuming that there are duplicated entries.

For the following JSON:

[{
  name : "Andrea",
  age : 31,
  gender : "Male",
  skilled : true
}, {
  name : "Eva",
  age : 27,
  gender : "Female",
  skilled : true
}, {
  name : "Daniele",
  age : 26,
  gender : "Male",
  skilled : false
}]
the hpack algorithm produces a compressed version which looks like this:

[["name","age","gender","skilled"],["Andrea",31,"Male",true],["Eva",27,"Female",true],["Daniele",26,"Male",false]]
More details about hpack algorithm can be found at project home page.

Analysis

The purpose of this analysis is to compare each of the described JSON compressor algorithms. For this purpose we will use 5 files with JSON content having different dimensions, varying from 50K to 1MB. Each JSON file will be served to a browser using a servlet container (tomcat) with the following transformations:

  • Unmodified JSON - no change on the server side
  • Minimized JSON - remove whitespaces and new lines (most basic js optimization)
  • Compressed JSON using CJSON algorithm
  • Compressed JSON using HPack algorithm
  • Gzipped JSON - no change on the server side
  • Gzipped and minimized JSON
  • Gzipped and compressed using CJSON algorithm
  • Gzipped and compressed using HPack algorithm

Results

This table contains the results of the benchmark. Each row of the table contains one of the earlier mentioned transformation. The table has 5 columns, one for each JSON file we process.
\
json1 json2 json3 json4 json5
Original JSON size (bytes) 52966 104370 233012 493589 1014099
Minimized 33322 80657 180319 382396 776135
Compress CJSON 24899 48605 108983 231760 471230
Compress HPack 5727 10781 23162 49099 99575
Gzipped 2929 5374 11224 23167 43550
Gzipped and Minimized 2775 5035 10411 21319 42083
Gzipped and compressed with CJSON 2568 4605 9397 19055 37597
Gzipped and compressed with HPack 1982 3493 6981 13998 27358

Relative size of transformations(%)

The relative size of transformation graphic is useful to see if the size of the json to compress affects the efficiency of compression or minimization. You can notice the following:
  • the minimization is much more efficient for smaller files. (~60%)
  • for large and very large json files, the minimization has constant efficiency (~75%)
  • compressors algorithms has the same efficency for any size of json file
  • CJson compressing algorithm is less efficient (~45%) than hpack algorithm (~8%)
  • CJson compressing algorithm is slower than hpack algorihtm
  • Gzipped content has almost the same size as the compressed content
  • Combining compression with gzip or minimization with gzip, doesn't improve significantly efficiency (only about 1-2%)

Conclusion

Both JSON compression algorithms are supported by wro4j since version 1.3.8 by the following processors: CJsonProcessor & JsonHPackProcessor. Both of them provide the following methods: pack & unpack. The underlying implementation uses Rhino engine to run the javascript code on the serverside.

JSON Compression algorithms considerably reduce json file size. There a several compression algorithms. We have covered two of them: CJson and HPack. HPack seems to be much more efficient than CJson and also significantly faster. When two entities exchange JSON and the source compress it before it reach the target, the client (target) have to apply the inverse operation of compression (unpacking), otherwise the JSON cannot be used. This introduce a small overhead which must be taken into account when deciding if JSON compression should be used or not.

When gziping of content is allowed, it has a better efficiency than any other compression algorithm. In conclusion, it doesn't worth to compress a JSON on the server if the client accept the gzipped content. The compression on the server-side does make sense when the client doesn't know how to work with gzipped content and it is important to keep the traffic volue as low as possible (due to cost and time).

Another use-case for JSON compression algorithm is sending a large JSON content from client to server (which is sent ungzipped). In this case, it is important to unpack the JSON content on the server before consuming it.

519 comments:

  1. This site use full to all around the would, use it develop both.........
    this site use of all the would and system news and helps known every one...........
    System Optimization

    ReplyDelete
    Replies
    1. I really appreciate you for all the valuable information about algorithms you are providing us through your blog.

      Thanks
      SEO Company New York

      Delete
    2. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Front end developer learn from Javascript Training in Chennai . or learn thru JavaScript Online Training in India. Nowadays JavaScript has tons of job opportunities on various vertical industry. JavaScript Training in Chennai

      Delete
  2. Very cool. I just heard of MesssagePack today, and decided to look around and see what else was available out there. I hadn't ever thought about optimizing my JSON because I assumed since it was so much more efficient than XML, it was good enough. Thanks again.

    ReplyDelete
  3. Appreicate your thoughts, Im not always in agreement, but you do cause a peron to think keep blogging!carpet cleaning adelaide

    ReplyDelete
  4. An unbelievable blog. This blog will indisputably be definitely recommended to my friends as well.
    Joycelyn Corria

    ReplyDelete
  5. Would you please consider removing the spam comments? They're a distraction, and they encourage spammers to keep spamming the rest of our blogs.

    Otherwise, thanks for the JSON compression comparisons.

    ReplyDelete
  6. Optimising JSON is also of interest to me (see https://sourceforge.net/projects/ojson/ for an opensource Java API I've developed) I'm experimenting with eliminating repeated keys using embedded tags and templating, but also using tagged string references so that examples such as [ "a repeated string", "a repeated string" ] is mapped to [ "@1:a repeated string", "@1" ] which can be easily mapped back to the original text.

    It appears my approach is mostly aligned with the hpack algorithm for optimising keys. I'd be interested to obtain a copy of the 5 json texts used in the results above so that I can see what figures I can produce.

    ReplyDelete
    Replies
    1. A quick follow up with at least one test result, using the json.hpack test data I discovered by following the above link https://github.com/WebReflection/json.hpack/tree/master/test (the file 5000.txt) I'm getting the following test result using my 'ojson' API:

      json.hpack-5000: Reduction: 76.0% from [776133] to optimised size [186598]

      I'm also mapping the 'ojson' output back to plain json and using this java package:

      org.skyscreamer.jsonassert.JSONAssert

      to verify the original hpack json is indeed being restored (from json -> ojson -> newJson):

      JSONAssert.assertEquals(json, newJson, JSONCompareMode.STRICT);

      Hope this is of interest.

      Delete
  7. Web site optimization produces highly competitive web sites that out-perform on every measure; traffic, speed, conversion rates, sell-throughs, and, most importantly, return on your investment.Thanks for your useful information.
    Web Developers Adelaide

    ReplyDelete
  8. Thank you for sharing this informative and interesting article. Keep up the good work!


    Melbourne seo services

    ReplyDelete
  9. Nice post. By reading your blog, i get inspired and this provides some useful information. Thank you for posting this exclusive post for our vision. 
    Microsoft azure training in Bangalore
    Power bi training in Chennai

    ReplyDelete
  10. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

    DevOps is currently a popular model currently organizations all over the world moving towards to it. Your post gave a clear idea about knowing the DevOps model and its importance.

    Good to learn about DevOps at this time.


    devops training in chennai | devops training in chennai with placement | devops training in chennai omr | devops training in velachery | devops training in chennai tambaram | devops institutes in chennai | devops certification in chennai | trending technologies list 2018

    ReplyDelete
  11. Such a wonderful blog on Machine learning . Your blog have almost full information about Machine learning .Your content covered full topics of Machine learning that it cover from basic to higher level content of Machine learning . Requesting you to please keep updating the data about Machine learning in upcoming time if there is some addition.
    Thanks and Regards,
    Machine learning tuition in chennai
    Machine learning workshops in chennai
    Machine learning training with certification in chennai

    ReplyDelete
  12. you exlained very well abouut the resource.thanks for sharing this useful infromation.good work.keep going.

    Apple service center in Chennai
    Apple service center

    ReplyDelete
  13. Nice post!Everything about the future(học toán cho trẻ mẫu giáo) is uncertain, but one thing is certain: God has set tomorrow for all of us(toán mẫu giáo 5 tuổi). We must now trust him and in this regard, you must be(cách dạy bé học số) very patient.

    ReplyDelete
  14. very good post!!! Thanks for sharing with us... It is more useful for us...
    Selenium Training in Chennai | SeleniumTraining Institute in Chennai

    ReplyDelete
  15. Time is free but it's priceless(khóa học toán tư duy) . You cannot own it, but you can use it(cách dạy bé học số) . You can use it, but you can't keep it(toán tư duy logic là gì). Once you lose it, you will not be able to get it back.

    ReplyDelete
  16. A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  17. Awesome information.
    such an useful article.
    thanks for posting.keep sharing.
    Best Java training in Bengaluru

    ReplyDelete
  18. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  19. It’s always so sweet and also full of a lot of fun for me personally and my office colleagues to search you blog a minimum of thrice in a week to see the new guidance you have got.
    fire and safety course in chennai
    safety course in chennai

    ReplyDelete
  20. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  21. A Computer Science portal for geeks. It contains well written, well thought and well
    explained computer science and programming articles, quizzes and practice/competitive
    programming/company interview Questions.
    website: geeksforgeeks.org

    ReplyDelete
  22. Do you mind if I quote a couple of your posts as long as I provide credit and sources back to your blog?
    safety course in chennai
    nebosh course in chennai

    ReplyDelete
  23. Really useful information. Thank you so much for sharing.It will help everyone.Keep Post.

    Cloud Computing Interview Questions and Answers


    ReplyDelete
  24. Website Planning Best institute for digital marketing course in delhi. Initialisation of Digital Marketing.. Website Creation. Content Writing. Search Engine Optimization. Local Seo. Google Webmaster. Bing Webmaster.
    Digital Marketing training in Laxmi Nagar

    ReplyDelete
  25. மகிழ்ச்சியான மற்றும் மகிழ்ச்சியான நாள். கட்டுரையைப் பகிர்ந்தமைக்கு மிக்க நன்றி

    máy phun tinh dầu

    máy khuếch tán tinh dầu tphcm

    máy khuếch tán tinh dầu hà nội

    máy xông phòng ngủ

    ReplyDelete
  26. The article is so informative. This is more helpful for our
    software testing training courses
    selenium classes Thanks for sharing

    ReplyDelete
  27. This is a nice Site to watch out for and we provided information on
    vidmate make sure you can check it out and keep on visiting our Site.

    ReplyDelete
  28. Download and install Vidmate App which is the best HD video downloader software available for Android. Get free latest HD movies, songs, and your favorite TV shows.

    ReplyDelete
  29. I think you did a super job in giving an explanation of JSON. This info will help many others. discover.com/activate

    ReplyDelete

  30. Nice! you are sharing such helpful and easy to understandable blog in decoration. i have no words for say i just say thanks because it is helpful for me.

    robotic process automation companies in us
    Robotic Process Automation in us
    machine maintanance in us
    erp in chennai
    mobility software companies in chennai
    erp providers in us

    ReplyDelete
  31. <a href="https://vidmate.vin/

    ReplyDelete
  32. We are leading supplier and manufacturer of filler masterbatches all over India
    We are providing you best quality filler masterbatch

    ReplyDelete
  33. Home Salon's, Certified Beauticians are highly experienced in her core domain and keep an update of most recent trend styling
    and hair shading, which can give a surprising and new look to ladies. In the Digital Era, the Beauty Parlour Services at home
    is now quickly accessible at your fingertips. Why go to traditional salon services and wait in queues when you can call a salon
    at home.
    Home Salon

    ReplyDelete
  34. thanks for your details It's very amazingweb design company in velacheryQbigpro branding solution is the best web design company in velachery web design company in velachery.we will create the web site and managing the site.we will help for all business.website is very important for all business.

    ReplyDelete
  35. If you are looking for Best Gyms in Ghaziabad then click on the given link.

    ReplyDelete
  36. thanks for your information really good and very nice web design company in velachery

    ReplyDelete
  37. While composing the last paper of your last B.Com test, a plenty of considerations experience your psyche. A great deal of these contemplations incorporate plans about resting for a whole week or celebrating for a whole week, contingent upon your individual inclinations. Be that as it may, trust us, none of that is really going to happen in light of the fact that when you complete your tests your folks, relatives, neighbours, and even your Facebook companions will begin getting some information about your feasible arrangements. What's more, don't mistake them for your gathering or dozing plans since they are alluding to your vocation Career after B com plans. In the present focused world, you are offered with many profession improving courses. On the off chance that you are not happy with the profession or course you decided for yourself at that point there are some present moment yet high worth – low speculation courses accessible in the market.

    ReplyDelete
  38. Thanks for sharing such a wonderful blog on Mean Stack .This blog contains so much data about Mean Stack ,like if anyone who is searching for the Mean Stack data,They will easily grab the knowledge of from this.Requested you to please keep sharing these type of useful content so that other can get benefit from your shared content.
    Thanks and Regards,
    Mean Stack training in Chennai
    Best mean stack training in Chennai
    Top Mean stack raining in Chennai
    Course fees for Mean stack in Chennai
    Mean stack training fees in Velachery, Chennai

    ReplyDelete
  39. Great! list of top JSON Compression algorithms blogs. will be very helpful for the people, who are looking for it. You made it so simple for them to get it without any hassle.
    Keep it up.
    All the best,
    Digital Marketing Course

    ReplyDelete
  40. your article on data science is very good keep it up thank you for sharing.thanks for your information really good and very nice web design company in velachery

    ReplyDelete
  41. For Data Science training in Bangalore, Visit:
    Data Science training in Bangalore

    ReplyDelete
  42. very interesting .good job and thanks for sharing such good blog .you blog is so conniving that i never to myself to say something about it .you are dining great job keep it up
    Tuition Service Lucknow | Home Tuition Service

    ReplyDelete
  43. Visit for Data Science training in Bangalore:
    Data Science training in Bangalore

    ReplyDelete
  44. Visit for Python training in Bangalore:
    Python training in Bangalore

    ReplyDelete
  45. nice information.its really helpful.thanks for sharing it. i apreciate your work.
    see more
    website
    click here
    more details

    ReplyDelete
  46. your post is really very interesting to read. I got Very valuable information from your blog.Thanks for sharing it.

    Python Training
    Python Classes
    Python Placement
    Python Institute Pune
    Python courses

    ReplyDelete
  47. Thanks for sharing the valuable information here. So i think i got some useful information with this content. Thank you and please keep update like this informative details.


    wedding catering services in chennai
    birthday catering services in chennai
    corporate catering services in chennai
    taste catering services in chennai
    veg Catering services in chennai

    ReplyDelete
  48. www.nalaico.com great thank you to you for sharing this

    ReplyDelete
  49. This is something new to me... thanku so much for sharing..
    Also Visit: Sourcekode Training Institute.

    ReplyDelete
  50. This comment has been removed by the author.

    ReplyDelete
  51. Well explained your.
    I appreciate Your Article.
    I Am a Developer at Skydevelopers Software's & I search For this.
    here You are The well Explained
    For More Info About My Services:
    skydevelopers
    Give me a chance to work once For Your Any Issue in Your Digital Business.
    Thank You.

    ReplyDelete
  52. Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.
    Are You looking for best online courses and free webniars you just click following links
    Online Certificate Courses
    Machine Learning Online Courses
    Best Online Certificate Courses In India
    Online Courses On Digital Marketing
    Online It Courses In India

    ReplyDelete
  53. This comment has been removed by the author.

    ReplyDelete
  54. thanks for sharing this awosome information about article
    also visit my website for songs
    Thodi Jagah From Marjaavaan Song

    ReplyDelete
  55. This is Very very nice article. Everyone should read. Thanks for sharing. Don't miss WORLD'S BEST BikeRacingGames

    ReplyDelete
  56. WEBSITE 24X7 EXCELLENCE IT | Digital Marketing | Web Designing | SEO | SMO | Logo Designing Services
    Great post! I really enjoyed reading it. Keep sharing such articles. Looking forward to learn more from you.
    Best SEO Company Chennai
    Digital Marketing Chennai
    App Development Company Chennai
    Web Design Company Chennai
    Graphic Designing Company Chennai
    CRM Services Chennai
    Web Hosting company Chennai

    ReplyDelete
  57. Appreciation for really being thoughtful and also for deciding on certain marvelous guides most people really want to be aware of....
    Data science training chennai | data science course chennai

    ReplyDelete
  58. I gathered a lot of information through this article.Every example is easy to undestandable and explaining the logic easily.selenium training in bangalore

    ReplyDelete
  59. Nice blog, it's so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.
    Home Tutor In Lucknow | Home Tutor in Lucknow

    ReplyDelete
  60. v i found this article more informative, thanks for sharing this article!
    showbox
    showbox for pc

    ReplyDelete
  61. Thank you for excellent article.You made an article that is interesting.
    Best AWS certification training courses. Build your AWS cloud skills with expert instructor- led classes. Live projects, Hands-on training,24/7 support.
    https://onlineidealab.com/aws-certification/

    ReplyDelete
  62. It’s really great information Thanks for sharing. Best Manual Testing Training in Bangalore, BTM layout. My Class Training Academy training center for certified course, learning on Manual Testing Course by expert faculties, also provides job placement for fresher, experience job seekers.

    ReplyDelete
  63. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. dot net training in bangalore

    ReplyDelete
  64. It’s really great information for becoming a better Blogger. Keep sharing, Thanks...

    Bangalore Training Academy located in BTM - Bangalore, Best Informatica Training in Bangalore with expert real-time trainers who are working Professionals with min 8 + years of experience in Informatica Industry, we also provide 100% Placement Assistance with Live Projects on Informatica.

    ReplyDelete
  65. Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article. We are top CRM Software | CRM Software Mumbai | CRM Software Provider | CRM Software Pune | Field Management Software | CRM Software India

    ReplyDelete
  66. Thanks, You for such a great post. I have tried and found it really helpful.

    Best Hadoop Training in Bangalore, offered by BangaloreTrainingAcademy. Bangalore's No.1 Hadoop Training Institute. Classroom, Online and Corporate training.

    Thanks and Regards,
    BangaloreTrainingAcademy

    ReplyDelete
  67. Really very happy to say, your post is very interesting to read. I never stop myself to say something about it. You’re doing a great job. Keep it up...

    Upgrade your career Learn AWS Training from industry experts get Complete hands-on Training, Interview preparation, and Job Assistance at Softgen Infotech Located in BTM Layout.

    ReplyDelete
  68. Thank you so much for the great and very beneficial stuff that you have shared with the world.

    Become an Expert In Python Training! The most trusted and trending Programming Language. Learn from experienced Trainers and get the knowledge to crack a coding interview, @Softgen Infotech Located in BTM Layout.

    ReplyDelete
  69. Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck…

    Start your journey with DevOps Course and get hands-on Experience with 100% Placement assistance from experts Trainers @Softgen Infotech Located in BTM Layout Bangalore.

    ReplyDelete
  70. Your topic is very nice and helpful to us … Thank you for the information you wrote.

    Are you looking for Best Training Institute for Data Warehousing Training center in BTM? Bangalore Training Academy Training provides Data Warehousing course, live project with job placement by experienced faculties.

    ReplyDelete
  71. I am really happy to say it’s an interesting post to read. I learn new information from your article, you are doing a great job. Keep it up…

    Best SAP Hybris Training in Bangalore , Marathahalli. Real Time Experts training center for certified course, learning on SAP Hybris Course by expert faculties, also provides job placement for fresher, experience job seekers.

    ReplyDelete
  72. I read this post your post so nice and very informative post thanks for sharing this post...

    Best SAP ABAP Training in Bangalore for SAP, We provides the sap training project with trainers having more than 5 Years of sap training experience, We also provide 100% placement support.

    ReplyDelete
  73. Thanks for sharing this fantastic blog, really very informative. Your writing skill is very good, you must keep writing this type of blogs

    Home Salon Dubai
    wedding car hire gurgaon
    wedding car hire banglore wedding car hire delhi
    wedding car hire dehradun
    wedding car hire noida

    ReplyDelete
  74. Here is the details of best plastic manufacturing company in GCC. Taldeen.com.sa they are manufacturing different kinds of plastic products. Here is some products details under Handling Solutions.
    Handling Solutions - Plastic Pallets

    ReplyDelete
  75. Taldeen is one of the best plastic manufacturing company in Saudi Arabia. They are manufacturing different type of plastic products like plastic pipes, water tanks etc.. They are classified their products under four different category. They are,
    Pipes Solutions
    Agriculture Solutions
    Handling Solutions
    Water Tank Solutions
    Under Handling Solutions, Taldeen manufacturing two products. They are Plastic Pallets and Plastic Crates.

    ReplyDelete
  76. Wonderful thanks for sharing an amazing idea. keep it...

    Learn Best PEGA Training in Bangalore from Experts. Softgen Infotech offers the Best PegaTraining in Bangalore.100% Placement Assistance, Live Classroom Sessions, Only Technical Profiles, 24x7 Lab Infrastructure Support.

    ReplyDelete
  77. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts.

    sap hybris training in bangalore

    sap scm training in bangalore

    sap pm training in bangalore

    sap crm training in bangalore

    sap ewm training in bangalore

    ReplyDelete