OSSRF indicators (OSS2018 version)

Open Source Software Resilience Framework (OSSRF)


Following the City Resilience Framework paradigm, Open Source Software Resiliency Framework is also being structured to four (4) dimensions that are then analyzed in twelve (12) goals and on a third level on a set of indicators.

DIMENSIONS

Source Code

Business & Legal

Integration & Reuse

 Social (Community)

GOALS

Source Code

Business & Legal

Integration & Reuse

 Social (Community)
Continuous Growth Economic Sustainability Low Dependability Well defined Project Standards
Holistic Documentation Flexible Licensing Low Complexity Well Defined Governance
Systematic Testing & Violation Minimization External Organization Support Ease of Integration Developer Base Activity

INDICATORS


Continuous Growth

  1. Development Stage: based on the classification provided on and the latest tag / release of the project in github we define the following development stages: (1) Alpha (if the tag contains ‘a’ or ‘alpha’, (2) Beta (if the tag contains ‘b’ or ‘beta’), (3) Stable (if the tag is equal or greater than v1.0).

  2. Size: based on the NLOC metric (also known as SLOC from ) to measure the source / effective lines of code (not comments or blanks) as it measured from the PHPQA tool .

  3. Operating Systems: number of different operating systems supported (Windows family, Linux distributions, Mac OS X, Android, iOS, Web Application). We based our classification on and added the mobile operating systems Android and iOS and the Web Application option.

Holistic Documentation

  1. Source Code Documentation: based on the percentage of comments to the lines of code. This can be measured by dividing Comment Lines of Code (CLOC) with Lines of Code (LOC) but the percentage itself is provided by the PHPQA tool.

  2. User guide: 0 (no) or 1 (yes) based on whether there is an end-user guide available. Examples of user guides can be readme files, pdf files or Wiki pages containing but not limited to installation, usage instructions.

  3. Technical guide: 0 (no) or 1 (yes) depending on whether the “Contributing” item to the github’s community profile checklist is checked.

Systematic Testing & Violation Minimization

  1. Unit testing: 0 (no) or 1 (yes) based on whether the project is utilizing unit testing or not.

  2. Critical: the percentage of violations referring to critical. Measured by the PHPQA tool.

  3. Errors: the percentage of violations referring to errors. Measured by the PHPQA tool.

  4. Warnings: the percentage of violations referring to warnings. Measured by the PHPQA tool.

  5. Information: the percentage of violations referring to information. Measured by the PHPQA tool.

Economic Sustainability

  1. Donations: 0 (no) or 1 (yes) based on whether the OSS project accepts donations. 0 is considered a non resilient value.

  2. Commercial features: Commercial features: 0 (no) or 1 (yes) based on whether the OSS project offers commercial features or a pro (paid) version. The indicator was based to the work of [24]. 0 is considered a non resilient value.

  3. Paid support: Paid support: 0 (no) or 1 (yes) based on whether the OSS project offers a paid plan for support [24]. 0 is considered a non resilient value.

Flexible Licensing

  1. Level or permissiveness: 0 (all restrictive - i.e. commercial), 1 (persistent i.e. GPL), 2 (all permissive - i.e. BSD). We base the indicator to the of [25]}. The indicator is considered non resilient when it is less than 1.

  2. Level of persistence: 0 (no) or 1 (yes) based on whether there are parts of the project's code or dependencies published under persistent licenses (i.e. GPL). We base the indicator to the of \cite{valimaki2002evaluation}. 1 is considered a non resilient value.

External Organization Support

0 (no) or 1 (yes) based on whether the OSS project is supported by an external organization (non profit, governmental or corporate). 0 is considered a non resilient value.

Low Dependability

  1. Infrastructure: based on the type of application. 0 (mobile), 1 (desktop), 2 (web).

  2. Integration: based on the way the project under review integrated with other systems. 0 (stand-alone), 1 (file exchange), 2 (database syncing), 3 (application programming interface (API)).

  3. Dependencies: based on the number of 3rd party projects that are dependencies to the project under review. This information can be found to “Dependency graph” under github’s insights tab.

Low Complexity

  1. Number of classes: based on the NOC metric proposed by . The lower the metric, the simpler the system. The information about this indicators can be found directly from the PHPQA tool.

  2. Average Lack of Cohesion Metric: based on the LCOM metric proposed by . The lower the metric, the simpler the system. The information about this indicators can be found directly from the PHPQA tool.

  3. Average Cyclomatic Complexity by class: based on the WMC metric proposed by . The lower the metric, the simpler the system. The information about this indicators can be found directly from the PHPQA tool.

Ease of Integration

  1. Projects Instability: based on the Robert C. Martin’s metric . Indicates the class’s resilience to change. We are interested in the trend of instability (increase or decrease) therefore we are taking under consideration the average instability of the project using the analysis from the PHPQA tool.

  2. Dependents: based on the number of projects dependent to the project under review. This information can be found to “Dependency graph” under github’s insights tab.

Well defined Project Standards

  1. Coding style: 0 (no) or 1 (yes) based on whether the project follows a specific coding style (i.e. Google, Automattic, Zend).

  2. Features roadmap: 0 (no) or 1 (yes) based on whether the project keeps an official requirement analysis document or feature roadmap (i.e. Jira, Trello Board).

  3. Documentation automation: 0 (no) or 1 (yes) based on whether the project uses a documentation automation system (i.e. javadoc, phpdoc).

Well Defined Governance

  1. Participation: based on how open is the project to external contributions (0 - unknown / not defined, 1 - closed / invited members only, 2 - open with pull request, 3 - open to everyone). We base this indicator to the work of .

  2. Decision making: 0 (unknown / not defined), 1 (benevolent dictatorship), 2 (meritocracy) based on who is making the decisions for the future of the project. We are following the analysis of with the addition of unknown if there is no decision making process in place.

  3. Tracking system: 0 (no) or 1 (yes) based on whether the project uses an issue tracking system (i.e. bugzilla). We base this indicator to the work of .

Increasing Developer Base Activity

  1. Active developers: based on the number of active developers that contributed to the specific tag or release of the project. The information comes from the analysis of git via the Gitstats tool .

  2. SLOC contributed: based on the difference between the SLOC added and SLOC deleted during release or tag of the project. The information comes from the analysis of git via the Gitstats tool.

  3. Number of commits: based on the number of commits contributed during release or tag of the project. The information comes from the analysis of git via the Gitstats tool.

References

[1] Jeff McAffer. Microsoft joins the Open Source Initiative. https://open.microsoft.com/2017/09/26/microsoft-joins-open-source-initiative/, 2017. [Online]. [ bib ]
[2] Steve Weber. The success of open source. Harvard University Press, 2004. [ bib ]
[3] Dharmesh Thakker Max Schireson. The Money In Open-Source Software. https://techcrunch.com/2016/02/09/the-money-in-open-source-software/, 2016. [Online]. [ bib ]
[4] Eric Raymond. The cathedral and the bazaar. Philosophy & Technology, 12(3):23, 1999. [ bib ]
[5] José P Miguel, David Mauricio, and Glen Rodríguez. A review of software quality models for the evaluation of software products. arXiv preprint arXiv:1412.2977, 2014. [ bib ]
[6] Vishal Midha and Prashant Palvia. Factors affecting the success of open source software. Journal of Systems and Software, 85(4):895--905, 2012. [ bib ]
[7] Vision Mobile. Open governance index-measuring the true openness of open source projects from android to webkit. 2011. [ bib ]
[8] City Resilience Index. City resilience framework. The Rockefeller Foundation and ARUP, 2014. [ bib ]
[9] Andreas Wieland and Carl Marcus Wallenburg. The influence of relational competencies on supply chain resilience: a relational view. International Journal of Physical Distribution & Logistics Management, 43(4):300--320, 2013. [ bib ]
[10] C Warren Axelrod. Investing in software resiliency. 2009. [ bib ]
[11] J Da Silva and B Morera. City resilience framework. Arup & Rockefeller Foundation. Online: http://publications. arup. com/Publications/C/City_Resilience_Framework. aspx [12/15/2015], 2014. [ bib ]
[12] Organización Internacional de Normalización. ISO-IEC 25010: 2011 Systems and Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)-System and Software Quality Models. ISO, 2011. [ bib ]
[13] Anthony Wasserman, Murugan Pal, and Christopher Chan. The business readiness rating model: an evaluation framework for open source. In Proceedings of the EFOSS Workshop, Como, Italy, 2006. [ bib ]
[14] Anthony I Wasserman, Xianzheng Guo, Blake McMillian, Kai Qian, Ming-Yu Wei, and Qian Xu. Osspal: Finding and evaluating open source software. In IFIP International Conference on Open Source Systems, pages 193--203. Springer, 2017. [ bib ]
[15] Sandro Andrade and Filipe Saraiva. Principled evaluation of strengths and weaknesses in floss communities: A systematic mixed methods maturity model approach. In IFIP International Conference on Open Source Systems, pages 34--46. Springer, 2017. [ bib ]
[16] Jose Teixeira, Gregorio Robles, and Jesús M González-Barahona. Lessons learned from applying social network analysis on an industrial free/libre/open source software ecosystem. Journal of Internet Services and Applications, 6(1):14, 2015. [ bib ]
[17] 100 Resilient Cities. http://www.100resilientcities.org/, 2013. [Online]. [ bib ]
[18] James Piggot and Chintan Amrit. How healthy is my project? open source project attributes as indicators of success. In IFIP International Conference on Open Source Systems, pages 30--44. Springer, 2013. [ bib ]
[19] PHPQA Tool, Official website. https://edgedesigncz.github.io/phpqa/. [Online]. [ bib ]
[20] David A Wheeler. More than a gigabuck: Estimating gnu/linux’s size, 2001. [ bib ]
[21] Robert Martin. Oo design quality metrics. An analysis of dependencies, 12:151--170, 1994. [ bib ]
[22] Javier Luis Cánovas Izquierdo and Jordi Cabot. Enabling the definition and enforcement of governance rules in open source systems. In Proceedings of the 37th International Conference on Software Engineering-Volume 2, pages 505--514. IEEE Press, 2015. [ bib ]
[23] Ross Gardler and Gabriel Hanganu. Governance models. Open Source Software Watch, last modified February, 14, 2012. [ bib ]
[24] Neeshal Munga, Thomas Fogwill, and Quentin Williams. The adoption of open source software in business models: a red hat and ibm case study. In Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, pages 112--121. ACM, 2009. [ bib ]
[25] M Välimäki and Ville Oksanen. Evaluation of open source licensing models for a company developing mass market software. Law and Technology, 2002. [ bib ]
[26] Gitstats Tool - Official Website. http://gitstats.sourceforge.net/. [Online]. [ bib ]
[27] Shyam R Chidamber and Chris F Kemerer. A metrics suite for object oriented design. IEEE Transactions on software engineering, 20(6):476--493, 1994. [ bib ]
[28] OKapi Github Repository. https://github.com/liip/Okapi. [Online]. [ bib ]
[29] WooCommerce Github Repository. https://github.com/liip/Okapi. [Online]. [ bib ]
[30] Jonas Gamalielsson and Björn Lundell. Sustainability of open source software communities beyond a fork: How and why has the libreoffice project evolved? Journal of Systems and Software, 89:128 -- 145, 2014. [ bib | DOI | http ]

Credits