Computer Network Simulation with ns-3: A Systematic Literature Review

: Complexity of current computer networks, including e.g., local networks, large structured networks, wireless sensor networks, datacenter backbones, requires a thorough study to perform analysis and support design. Simulation is a tool of paramount importance to encompass all the different aspects that contribute to design quality and network performance (including as well energy issues, security management overheads, dependability), due to the fact that such complexity produces several interactions at all network layers that is not easily modellable with analytic approaches. In this systematic literature review we aim to analyze, basing our investigation on available literature, the adoption of a popular network simulator, namely ns-3, and its use in the scientiﬁc community

fosters the collective effort to produce or improve new components, to augment the usefulness of the simulation tool and to strengthen the effectiveness of its use into simulation processes, or, together with open-sourceness, the implementation of third-party tools to support or automate simulation campaigns, simulation generation, model-based simulation development or other useful tools that ease or empower the management of simulation and its application in structured development and assessment processes and methodologies.
Many alternatives exist for computer network simulation: in this Systematic Literature Review (SLR) we focus on ns-3 (www.nsnam.org/), a modular, programmable, extensible, open, open-source, community-supported simulation framework for computer networks. Our choice is due to the fact that the technical characteristics of ns-3 match the needs of our research activities, and to the increasing popularity that this tool is gaining among the computer networks research community and, slowly, among professionals.
ns-3 is an open computer networks simulation environment that is based on discrete-event simulation. In such a simulator, each event is associated with its execution time, and the simulation proceeds by executing events in the temporal order of simulation time. When an event is processed, it may generate zero, one or more events. As a simulation executes, events are consumed, but more events may (or may not) be generated. The simulation will stop automatically when no further events are in the event queue, or when a special "Stop" event is found [1].
It is designed for research use, thinking of the needs of the research community, and fosters a community-based collaboration model both to support the development of new modules and to perform validation or peer review activities across different groups. ns-3 is distributed under an open-source model and a free software paradigm. The simulation environment is designed to allow its use as a real-time emulator as well, integrating simulated portions and physical portions in the same network. This level of integration is provided by a real-time scheduler, besides the simulation support, to enable simulation-in-the-loop. The simulator is intended to be used by means of its API inside programmatic descriptions of users' scenarios and configurations, which may be developed following usual workflows and processes. The API is used to produce outputs in terms of traces that are coherent with the traces that may be obtained by observing traffic in real-world networks with tools like Wireshark (https://www.wireshark.org/) or TCPdump (https://www.tcpdump.org/), so that they may be analyzed by the same tools that are already in use for experimental setups or diagnostics. The authors of ns-3 explicitly state that they aim at allowing the reuse of real-world implementation of protocols in ns-3, so that the effort for the design and production of very detailed simulation components is reduced as much as possible. The goal is to provide a very realistic simulation support for a large number of different type of network infrastructures, communication and routing protocols, technologies, layers, allowing network-enabled software to run on top of the simulation with minimal effort (e.g., virtual machines).
ns-3 is the result of a long experience that evolved through two previous simulators (namely ns-1 and ns-2): however, the development team decided to completely restart from scratch the design and development process with ns-3 on different premises, with no backward compatibility, actually giving birth to a completely new artifact but keeping and leveraging the experience from previous versions. ns-3 simulations may be written in C++ and Python, and their basic setup and execution is guided by a process suggested by the development team. ns-3 has been designed to be scalable.
We believe that ns-3 has a significant potential and we expect its community to grow. We compared it against alternatives, and our internal evaluation led us to choose ns-3 over them. However, after a test period on several scenarios that are in the scope of our research activities, we decided to corroborate our evaluation with a SLR before using it to support the activities of a research project.
Our investigation aims to understand what are the actual diffusion and the actual use of ns-3 in the research community, also to leverage literature to individuate possible weaknesses and limits of the tool that may affect the progress and the result of our research activities, and to classify the application areas in which adoption is wider and potential community support is stronger, more reactive and more intense, as well as the practically available modules and extensions that have been published and integrated. We also decided not to contact or involve the ns-3 development team while conducting this SLR to avoid any bias, to actually rely on the scientific papers we were able to find, identify and obtain by ourselves by using the main research papers search engines, listed in Table 2: by the way, the development team does not provide a list of papers mentioning ns-3 to be used for a SLR. We excluded non-indexed documents to be sure that our sources have been peer reviewed or at least referred or considered to be reliable by the scientific community. We considered all scientific papers indexed by the indexes between 2009 and mid October 2019. This paper is organized as follows: in Section 2 a set of popular networking oriented simulators is presented and examined; in Section 3 the review process is described and the methodology is presented that has been used for this review; in Section 4 the research questions that guide this study are presented and discussed; in Section 5 the methods for conducting the study are detailed; in Section 6 the results of the study are provided and commented; finally, Section 7 draws the conclusions resulting from the study.

Related Work
Simulation is a widely accepted tool in the field of computer networks. Literature offers both free and commercial simulation tools that aim at supporting, at different levels and with different purposes, design and analysis of computer networks. With no claim to completeness, this Section presents some relevant cases to provide a first reference to the readers.
Within the category of open-source simulators, besides ns-3 literature reports ns-2, OMNET++ and SWANS as noticeable examples.
ns-2 (https://www.isi.edu/nsnam/ns/), actually the previous version of ns-3, deserves to be mentioned as a different tool, because it is still widely used notwithstanding it is not maintained anymore and its last version has been released in 2011, since, as previously reported, ns-3 has been completely redesigned on a different basis. ns-2 implements discrete-event simulation. ns-2 is characterized by a separation between the approach to the definition of the simulated components and the approach to the definition of simulation management and setup: the first is based on a compiled software base, programmed in C++, including the components defined by the user or third parties, if any, while the second is managed by interpreted code written in OTcl, an object-oriented extension of the popular Tcl language, to ease the process. ns-2 has a significant base of available simulation components, due both to its popularity and lifespan, including protocols, reusable simulation objects that represent real-world equipment or routing algorithms. Support exist for wired and wireless networks, mobile networks, generation of specific patterns of traffic and energy-oriented evaluation. Its wide adoption fostered the development of third-party tools that partially compensate its limited support for visual management and analysis. OMNET++ (https://omnetpp.org/) is an event-based simulator that aims to provide a generic support to all kind of networked systems, from mobile networks to on-chip networks. The core of the simulator is a C++-based kernel that provides several services on which different application areas are made available by additional third-party frameworks, including very different cases such as photonic networks or sensor networks, or more abstract system performance modeling tools such as queuing networks: the official website reports a very comprehensive list of domains that have been covered by users and made publicly available under open-source licenses. OMNET++ provides a proprietary scripting language to describe topologies in model setups and can be used both from the command line and a rich graphical user interface. Models and components may be specified by an event-based paradigm and a process-oriented paradigm. The simulation kernel, designed for portability, allows distributed and parallel execution to support large simulation scenarios and speed up the runs. The simulation definition process is integrated in Eclipse to ease setup and management, and interactive simulation is provided. OMNET++ supports analysis of performance metrics by visual tools. SWANS (http://jist.ece.cornell.edu/) is a simulator for wireless networks and sensor networks that has been designed with special attention to allow high simulation performance and larger simulation scenarios with respect to ns-2 and GloMoSim, the open-source version of the QualNet simulator described later in this section. SWANS uses JiST, a Java-based high-performance discrete-event simulation engine, as simulation support. The main aim of SWANS is scalability. Its architecture is component-based, and it can run Java network applications over simulated networks. The last version has been released in 2005.
Within the category of commercial simulators, the market offers tools such as NetSim and QualNet. NetSim (https://www.tetcos.com/) is a tool that can be licensed for professional, research or teaching use. It is reported by developers to be or have been used in different professional environments such as military and space organizations, utilities distribution companies, railways industry, network equipment manufacturers and services providers for the design, development and evaluation of a rich variety of computer and communication networks. NetSim provides user friendly tools to support the various activities and allows both simulation and emulation. It provides packet animation and a native analysis support. Simulations can be based on existing modules or user defined modules written in C.
QualNet (https://www.scalable-networks.com/qualnet-network-simulation) is designed to allow users to test, be trained on or plan large realistic networks by means of simulation (the same company provides an emulation tool as well). QualNet is originally based on GloMoSim and supports operationally accurate contexts. It provides a suite of tools for the configuration, the tracing and the analysis of traffic events.
In Table 1 the main characteristics of cited network simulators are shown. It is worth noting that literature also reports the implementation of network simulations based on general purpose simulation frameworks that have not been considered in the scope of this analysis.

SLR Process
A SLR is a methodology designed to evaluate and obtain information about a specific argument and about one or more punctual research question(s). A SLR should provide an accurate and rigorous analysis of the topic of interest by means of a well-documented and standard process that evolves into three main phases (see Figure 1): Reporting the results

SLR Target
This work wants to study in a systematic manner the research literature about ns-3 in the period between 2009 and 2019. The aim is to find the main research areas in which it has been used and how it could adapt, in an effective way, to new fields of application, by the development of new modules or extensions.

SLR Protocol
The review protocol used in this work is a slightly modified version of the protocol used in [2], because it was already validated, and it worked well for our purpose: it is depicted in Figure 2.
The review protocol enactment started from the definition of the search query that has then been executed to retrieve data by a scientific literature search engine (namely Scopus) to automatically find the most relevant papers. This first list of papers was augmented with manually selected papers considered of special relevance and already known to the authors; all abstracts and conclusions sections of the papers in the list have been read to select which of those paper were relevant for the SLR. Finally, these selected papers have been carefully examined and contents have been evaluated by using of the data extraction form to collect the most relevant data.

Research Questions
To manage the goals of our review, we defined two research questions to guide the process: • RQ1: which are the impacted application domains, according to the literature? • RQ2: how easily extensible did ns-3 prove to be with respect to its adaptation to new domains or uses?
As we aim at providing a way to introduce to ns-3 to potential users (like our research group) that do research activities on which it could have a positive impact, the purpose of the two questions is to document that the scientific community actually uses it in several areas and that those areas have been covered by native or third-party modules or extensions, so it is likely that other areas that may be of interest for potential users may be easily covered in the future or by investing in the development of new modules or extensions in first person. RQ1 has been used to search and categorize the main research areas in which ns-3 has been successfully adopted: from this question three other refining questions have been derived that help to categorize with more details the use cases: RQ2 tries and measures qualitatively how simple and effective is to implement custom modules and extensions to use ns-3 in new application areas or in a way it was not designed for. This being a qualitative measure, the related answer derives from an evaluation, done by this authoring team, that is partially founded onto explicit elements provided in the papers by their authors, partially founded on the technical descriptions provided in the papers and partially founded onto the number and frequency of presented extensions in the literature. Two other refining questions have been derived: • RQ2.1 Is it a new extension or module? • RQ2.2 Does it use an external module or extension?

Data Source and Search Strategy
We searched through the most important scientific literature search engines to retrieve a wide perspective on the topic, keeping as main reference Scopus and integrating results with search results from the other databases.
The reasons behind this choice is the will to consider all major journals, conferences and workshops that are considered by the scientific community relevant to the topic or are declared as relevant by publishers or editors, but filtered and included by well-known indexes. The set of examined papers is the union of the set of papers obtained by the queries from each database. We decided to exclude all informal literature (power point slides, conference reviews, informal reports, work in progress and technical notes) because this kind of literature is difficult to evaluate in terms of quality assessment: this choice gives our analysis more stability and better guarantees about the quality of the selected papers. Table 2 shows the used databases. To identify our set of documents, we submitted some "free keywords"-based queries in 4 Research Documents Portals, namely Google Scholar Search Engine, Elsevier Search, Springer Open Access, IEEE Explore Digital Library (IEEE E.D.L) and The ACM Digital Library. All the queries were submitted filtering documents in the English language only. As first, we tried with the only keyword "ns3": all of these search portals provided research papers dealing with a unique topic related to medicine and chemistry domain; almost in the most of these documents, the keyword "ns3" is an acronym standing for a multifunctional inhibitor protein for the hepatitis C virus. Conversely, adopting the bi-gram "ns-3", which is the correct spelling of the name of the simulator, we just observed different behaviors for the considered search portals. A numerical comparison between the number of documents retrieved by adopting the "ns3" and "ns-3" keywords over the 4 considered sources, for the English language and in the time interval in 2009-2019, is summarized in Table 3. Elsevier and Springer exhibited the worst behaviors, since they were not able to provide a fine-grained selection of documents, even if the keywords changed. Springer retrieved 21 documents for the "ns3" keyword and only 2 over 21 were related to Computer Science (e.g., we retrieved the 2019 Lecture Notes in Networks and Systems), even if the research category filter "Engineering" was applied. Conversely, looking for keyword "ns-3", the number of retrieved documents amount to over 70 thousands but no matching documents to the network simulator were found yet. Documents retrieved by Google Scholar well represented the discrepancy among the "ns3" and the "ns-3" keywords. In the first case, the most of retrieved documents were related to the inhibitor protein; in the latter case, the most of documents matched the network simulation domain and were compliant to our research aim. Also results obtained by IEEE E.D.L. were interesting. Unlike Google Scholar, it was able to retrieve a significant number of relevant documents to the network simulator ns-3 both with "ns3" and "ns-3" keywords. The ACM Digital Library reported respectively 394 and 290,550 papers. Finally, we applied a cross selection strategy over all the relevant documents we retrieved by submitting the described queries to the 5 sources; we adopted, in order to select the 112 analyzed documents, as selection criteria the number of references and citation, the novelty of proposals for ns-3 modules extensions and the applications detailed in the following Sections.
The final search strategy followed to form the searching query makes use of the traditional 1998 version of the ACM Category classification as in Table 4 (available at https://www.acm.org/ publications/computing-classification-system/1998/ccs98). We decided to use this categorization to leverage an authoritative, well known and widely accepted set of reference criteria and to avoid any kind of bias that might have been influencing the selection process because of our knowledge about the topic or our personal research experiences, points of view and perspectives. We wanted to avoid a search strategy that could be too subject to our personal understanding of the topic, which is driven by the way in which we approached it in the past, and may have suggested biased keywords for queries. For the same reason, we decided to give a primary role to Scopus as main reference for retrieving the starting set of papers, as Scopus indexes papers from different publishers, from different kind of sources and from different disciplines, while others are limited to own publications and tend to suggest more papers than the ones that seemed actually relevant to our purpose for us. Analogously, we decided to use the old version of the ACM classification with respect to the new one because it is used by other publishers and institutions than ACM, and it does not especially privilege papers published by ACM in the considered period, as the new version has been adopted in 2012.
To obtain the standard set of keywords for the definition of queries, we selected Topic C, "Computer Systems Organization", with sub-topic C.2, "COMPUTER-COMMUNICATION NETWORKS", so we performed our queries referring to the sub-sub-topics of C.2 listed in Table 4, using the most significant words as keywords. Based on this classification, equivalent queries have been defined for each index with the needed syntax. For example, the query defined for Scopus is: "( TITLE-ABS-KEY ( ns3 AND computer AND communication AND networks ) AND (( network AND architecture AND design ) OR (Network AND Protocols) OR (Network AND Operation) OR (Distributed OR System) OR (local OR wide AND networks) OR (internetworking) ))".
The query has been obtained by considering the keyword "ns3" and relevant keywords from Table 4. As stated previously, Scopus has been chosen as the main reference because of its primary role, because it indexes papers from different publishers and because its filtering mechanisms was a better fit to obtain an appropriate coherence between resulting papers and relevance to the focus; the other databases have been then manually searched with similar ad hoc queries to look for relevant results not indexed by Scopus, as they tend to expand the results set with borderline or non-relevant papers.

The WNS3 Workshop on ns-3
The WNS3 workshop (https://www.nsnam.org/research/wns3/) is a yearly meeting that gathers developers and users of ns-3, with proceedings published by ACM in The ACM Digital Library. This venue is of paramount importance for the community that develops and supports ns-3, it is the flagship meeting of the ns-3 Consortium and hosts important training sessions. We carefully considered all 90 papers published by the workshop as from the web site, in order to find out if they match our needs, and one of the authors participated in 2019 to get to know the community and have an hands on approach to ns-3, its applications and the spirit behind it. A first analysis shows that the most of the papers deal with ns-3 specific implementation of features or standard modules, extension of existing modules or improvement of features: out of 90 titles, 33 explicitly mention implementations, 11 improvements or extensions, 2 integrations, for a total of 46 (see Table 5). These papers are not the only ones that are about implementations, improvements, extensions or integrations. In general, the focus of the papers are very specific to ns-3 internals and technical aspects, and rarely present other research related aspects, with some significant exception, so they are more bound to the ns-3 development community than to a general audience, with respect to the rest of the papers collected for our study. Out of the list of papers of the conference we isolated , that showed to extend significantly the domains coverable by means of ns-3 or to show noticeable research-oriented case studies.

Selection Criteria
We defined a general inclusion/exclusion criteria set to be applied to the papers obtained from the queries. These criteria are summarized in Table 6.

Data Extraction Form
The data extraction form is used to categorize and analyze the papers in an organic and objective way. It consists of two different parts: • the first part collects all general information about the paper, as from Table 7; • the second part collects the information that is more related with the research questions, to address the research questions (see Table 8): for RQ1 the questions characterize the field in which ns-3 is used in terms of networks technologies, area of application and kind of problem; -for RQ2 the answers are binary (yes/no), and describe if the paper presents a new module or extension for ns-3 and if it uses an already existing module.

Included and Excluded Studies
After the first search in Scopus and an integration by other search campaigns in the databases from Table 2, as described in Section 5.2 we selected 128 papers [1,. Then we applied to those papers the inclusion/exclusion criteria as described in Section 5.2, and after this phase we excluded 8 papers. More details are provided in Table 9 and Table 10, which reports the number of papers selected by the query and of the remaining papers after the selection process, divided by years.
Selected 2% 0% 2% 7% 6% 6% papers (2) (0) (2) (8) (7) (7) After reading abstract and conclusion we excluded 2 papers of them based on exclusion criterion 2 [25,26], 3 based on exclusion criterion 3 [27][28][29], 2 based on exclusion criterion 4 [30,31] and 1 based on exclusion criterion 1 [32]. In the end, we included in the study 94% (120/128) of examined papers . Figure 3 shows that in recent years there has been a significant growth of the number of paper talking about ns-3 (the query was performed in October 2019, so the number of paper of year 2019 must be considered partial because of the typical delays in indexing and actual availability of published papers).   Figure 4 show that the selected papers mainly aim to developing applications or to perform conceptual study.  Figure 4. Type of content.

Data Synthesis
Once we selected the papers that are more pertinent and in line with the aim of our research, we evaluated the general quality level of the picked papers. A first analysis did show not a polarization of the number of papers on specific venues (as, for the sake of space, readers can directly verify by browsing the bibliography of this paper), suggesting that the adoption of ns-3 is not specifically proper of any sub-community nor restricted to specific groups of users or research groups at large, so we decided to omit a summary table and to avoid further analysis about venues. Table 12 shows all details of the adopted Quality Assessment Scores form. Scores have been used to verify the validity of the selection and to support answering the research questions presented in the next Subsections.

Id Quality Assessment Criteria Score
A1 What is the relevance of the paper according to the conference/journal where it was published?
A2 What is the relevance of the citation according to its related citations?
A3 How clearly is the problem of the study described?
A4 How clearly is the research context stated?
A5 How explicitly are the contributions presented?
A6 How explicitly are the insights and issues for future work stated?

RQ1
We wanted to address different side of this research question, basically to help the researchers when they must choose a network simulation tool and help to evaluate if ns-3 is a good choice. To achieve this goal, we examined the fields of application in which ns-3 was most used, both the application field, intended as topic, and specific network technology applied, and also as specific problem addressed. Table 13 summarizes the information extracted from the papers that are related to this question. Results reported in Table 13 suggest that the most of the application domains in networking are covered, with a (unsurprising) peak for ordinary networking, which is anyway not prominent, followed by VANET (Vehicle Ad Hoc Networks), MANET (Mobile Ad Hoc Networks) and common Mobile Networks on similar positions (see Figure 5). The lowest percentage of application, corresponding to WSN (Wireless Sensor Networks), yet a popular application field, may be due to the fact that support for energy management, which is of paramount importance in this subdomain, is still under development, and quite novel as a topic in the literature per se, and to the fact that the implementation of the simulation stack for typical real-world configuration is recent and still to be completed: anyway, the aggregation of all wireless networking related subdomains (WSN, Wireless Networks and Mobile Networks) that share the most of the low layers of the ISO/OSI stack, constitutes more than one third of all papers. The flexibility of the tool is witnessed by the composition of the group of papers labeled as "others", as it includes very peculiar and specialistic domains such as Delay Tolerant Network, Named Data Network, Optical wireless, SDN, Visible Light Communication (VLC), while domains that are still emerging as autonomous from more general topics have been included in the relevant general topic (e.g., a paper about 5G mobile networks has been reported as belonging to Mobile Networks). Variety of problems is also rich: anyway, here the analysis shows that the detail of the simulation stack fosters application to network assessment and protocol design, analysis and verification, and minor attention is paid by researchers and practitioners interested into physical layer problems, probably because of the closeness with the telecommunications domain, that also has proper specific tools and simulation support capturing physical issues in deeper details (see Figure 6).

Wireless Networks
Unfortunately, the analysis did not provide enough information to answer Q1.2 with a significant level of details, as the most of the papers were not especially focusing on specific application fields that were not implicitly stated in the application domain.

RQ2
The aim of the second research question is to address another kind of issue about simulation in general. The question investigates the need and the availability of additional modules to cover issues that are not natively dealt with by ns-3, to understand how much it is a good choice to approach problems that exceed the original extent of modeling support included in ns-3. The research question is articulated in two different yes/no questions, the first of which is oriented to quantify in how many papers a new extension or module is proposed for ns-3 to cover a specific need (see Figure 7), the second to quantify how many papers document cases in which additional modules or extensions have been actually used to reach the goals (see Figure 8). In Table 14 we show the results, in Table 15 the distribution of papers over years.      2  0  2  8  7  7  19  23  27  15  10 According to results, the practice of extending ns-3 is established, but not especially needed. There is a non-negligible number of papers that present new modules or extensions, but in a good majority of the cases, assuming that researchers that adopted this simulator consider it suitable for their purposes, the standard package seems to be sufficient to satisfy the most of the needs. In fact, considering the number of the papers that use extensions or contributed modules, almost one third of the examined literature actually exploits additions. Jointly with the percentage of positive answers to Q2.1, this seems to us to prove that the tool is easily and profitably extensible and needed, but the standard package is sufficient for the most of the needs.

Conclusions and Future Works
In this study, we documented the large success of ns-3 as a network simulator, its popularity and its flexibility. Data show that the scientific community considers it a useful tool in different fields, and dedicate third-party resources to extend it and develop add-ons for new adjacent application domains.
A further useful contribution may be provided by analogous analyses on literature on the main "competitors", e.g., ns-2 and OMNET++, based on the same research questions and an overall meta-analysis to obtain a general view on the topic, to spot what is still not covered and would be useful for the scientific community and practitioners and develop focused built-in extensions or design a new, comprehensive simulation tool.
Author Contributions: All authors participated with the same effort and contribution to all phases in the preparation of this paper. All authors have read and agreed to the published version of the manuscript.
Funding: This research received no external funding.