Systematic Reviews Studies

Definition

English

Are a form of study aimed at clasifying, characterizing or understanding a phenomenom or a field of interest. They are based on a formal review of the literature, and support their execution on Research Procols in order to make their process repeatable (hence the use of the word 'Systematic').

English

We currently clasify Systematic Reviews Studyes acroding to their aim and converage of the PICO Structure [1]:

• Structured (Targeted) Review

This definition is proposed to characterize studies in which researchers do not intend to exhaustively characterize a field of interest neither to gather all published evidence available to understand a phenomenon. It allows to systematically map publications from a research field in a reduced but recognized relevant subset of sources (eventually from a specific period), such as relevant journals and conferences regarding a specific research field. Such reviews typically adopt resources for systematically finding publications but not to extract their data and evaluating them. A targeted review can be used as a first step before performing more comprehensive studies. Examples can be found at:

1. Magne Jorgensen and Martin Shepperd. "A systematic review of software development cost estimation studies." Software Engineering, IEEE Transactions on 33.1 (2007): 33-53.

2. Michael Franklin Bosu and Stephen G. MacDonell. "Data quality in empirical software engineering: a targeted review." Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering.ACM, 2013.

3. Robert Feldt and Ana Magazinius."Validity Threats in Empirical Software Engineering Research-An Initial Survey." SEKE. 2010.

4. Rafael Maiani de Mello and Guilherme Horta Travassos. "Characterizing Sampling Frames in Software Engineering Surveys." ESELAW 2015.

• Systematic Mapping Studies

For those studies where the researcher aims to build a classification scheme and structure a software engineering field of interest. The analysis of results focuses on frequencies of publications for categories within the scheme. Thereby, the coverage of the research field can be determined [3]. However, systematic mapping studies are not focused on evaluate the quality of the studies and its data extraction can be supported through thematic analysis. Different facets of the scheme can also be combined to answer more specific research questions. Examples are:

1. Barcelos, R. F. ; TRAVASSOS, G. H. . Evaluation Approaches for Software Architectural Documents: A Systematic Review. In: Ideas 2006 - 9o Workshop Iberoamericano de Ingenieria de Requisitos y Ambientes de Software, 2006, La Plata. Actas del Ideas 2006 - 9o Workshop Iberoamericano de Ingenieria de Requisitos y Ambientes de Software. La Plata: Clei, 2006. v. 1. p. 433-446.

2. Da Silva, Fabio QB, et al. "Replication of empirical studies in software engineering research: a systematic mapping study." Empirical Software Engineering 19.3 (2014): 501-557.

3. Munir, Hussan, Krzysztof Wnuk, and Per Runeson. "Open innovation in software engineering: a systematic mapping study." Empirical Software Engineering (2015): 1-40.

• Quasi Systematic Literature Review

All characteristics and rigor of a Systematic Literature Review can be observed but the Comparison perspective. It usually happens for those secondary studies that deal with no previously established baseline although different primary studies can be observed in the field. Therefore, baselines represent an expected result in this sort of study. Examples of quasi Systematic Literature Reviews: 1. França, B. B. N. ; Travassos, G. H. . Are We Prepared for Simulation Based Studies in Software Engineering Yet?. CLEI Electronic Journal, v. 16, p. 1 - Paper 8-25, 2013.

2. Abrantes, J.F. ; Travassos, G. H. . Towards Pertinent Characteristics of Agility and Agile Practices for Software Processes. CLEI Electronic Journal, v. 16, p. Paper 5-31, 2013.

3. Dias Neto, A. C. ; Travassos, Guilherme Horta . A Picture from the Model-Based Testing Area: Concepts, Techniques and Challenges. Advances in Computers, v. 80, p. 45-120, 2010.

4. de Mello, Rafael Maiani, et al. "Verification of Software Product Line Artifacts: A Checklist to Support Feature Model Inspections." Journal of Universal Computer Science 20.5 (2014): 720-745.

• Systematic Literature Reviews

For those studies where the researcher aims to understand a phenomenon and compare it against a pre-established baseline. A systematic literature review (often referred to as a systematic review) is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest [2]. Individual studies contributing to a systematic review are called primary studies; a systematic review is a form of secondary study [4]. Systematic reviews protocols must be developed to specify all the steps and criteria that shall be used to undertake the review, including a formulation of its research questions [1]. Well formulated questions usually follow the PICO (Population, Intervention, Comparison, Outcome) perspectives [4]. Studies' quality assessments are expected, as well as, the Comparison perspective makes possible the evolution of knowledge and meta analysis (in case of quantitative results). Examples of Systematic Literature Reviews:

1. Kitchenham, B. ; Mendes, E. ; TRAVASSOS, G. H. . Cross versus within-company cost estimation studies: A systematic review. IEEE Transactions on Software Engineering, v. 33, p. 316-329, 2007.

2. Kalinowski, M. ; Card, D. N. ; TRAVASSOS, G. H. . Evidence-Based Guidelines to Defect Causal Analysis. IEEE Software, v. 29, p. 16-18, 2012.

3. MacDonell, Stephen G., and Martin J. Shepperd. "Comparing Local and Global Software Effort Estimation Models--Reflections on a Systematic Review." Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on. IEEE, 2007.

Systematic Mapping Study vs (quasi)Systematic Literature Review

Although both Systematic Mapping Studies and (quasi) Systematic Literature reviews share the aim of identifying research gaps, (quasi) Systematic Literature reviews allows seeing where particular evidence is missing or is insufficiently reported in existing studies. In fact, systematic maps and reviews are different in terms of goals, breadth, validity issues and implications [1]. For this end, the (quasi) Systematic Literature Review process includes a quality evaluation of the publications and in depth data extraction activities focused on quantitative analysis.

PICO Strucnture for defining search strings

English

The Acronym PICO stands for Population, Intervention, Comparison and Outcome [4]. During the construction of a search string for conduction the studies described in this pages, the recommendation is to structure the keywords according the PICO structure. Keywords that characterize the same elements are combined with the logical "OR" operator. The resulting search string is constructed by combining the different elements with the logical "AND" operator.

• Population refers to classification of the subjects of interest for the study.
• Intervention refers to the treatment that is going to be evaluated.
• Comparison refers to the baseline results of the treatment.
• Outcome refers to the expected result of the treatment.

Reference

[1] Guilherme H. Travassos, Paulo Sérgio M. dos Santos, Paula Mian, Arilo Claudio Dias Neto, José Biolchini. An Environment to Support Large Scale Experimentation in Software Engineering. In: IEEE International Conference on Engineering of Complex Computer Systems, 2008, 2008, Belfast. Proceedings of ICECCS 2008. p. 193-202.

[2] José Biolchini, Paula. G. Mian, Ana Candida Natali and Guilherme H. Travassos (2005). “Systematic Review in Software Engineering: Relevance and Utility.” Technical Report. PESC - COPPE/UFRJ. Brazil. Available at: http://cronos.cos.ufrj.br/publicacoes/reltec/es67905.pdf

[3] Kai Petersen. Robert Feldt, Shahid Mujtaba, Michael Mattsson. "Systematic Mapping Studies in Software Engineering". 12th International Conference on Evaluation and Assessment in Software Engineering. 2008.

[4] Pai, M., McCulloch, M., Gorman, J.D., Pai, N., Enanoria, W., Kennedy, G.,Tharyan, P. Colford Jr, J.M. (2004). “Systematic Reviews and meta-analysis: An illustrated, step-by-step guide”. The National Medical Journal of India. Vol 17, No 2