MIT Technology Review has included Reinforcement Learning as a top 10 breakthrough technologies for 2017 in its latest issue.
​
What is reinforcement learning? It is about to let machines learn by experimenting. As it is explained in the magazine, reinforcement learning copies a very simple principle from nature. The psychologist Edward Thorndike documented it more than 100 years ago. Thorndike placed cats inside boxes from which they could escape only by pressing a lever. After a considerable amount of pacing around and meowing, the animals would eventually step on the lever by chance. After they learned to associate this behavior with the desired outcome, they eventually escaped with increasing speed.

Reinforcement learning algorithms can help, for example, to improve the "driving skills" of self-driving cars. Today’s driverless vehicles often falter in complex situations that involve interacting with human drivers, such as traffic circles or four-way stops. AI engineers can collaborate with simulation engineers to integrate a digital model of the driverless car in a simulated environment, replicating the most complex traffic situations without any risk for people and real traffic. In the virtual space, the control software can perform the maneuvers over and over altering its instructions a little in each attempt. Applying deep learning techniques, the system can extract the patterns from the best performed maneuvers, learning from experimentation.
​
We see then that high fidelity simulation is going to be a key lever to reinforcement learning but, what kind of simulation we need to do an effective learning? In a former post, we have already discussed about the many applications that simulation brings to the development of smart and connected devices, including the virtual experimentation and design of IoT's products. In that article, we were also discussing about the kind of simulation that it is needed to integrate in an effective way digital models of the new products in simulated environments: we need to work with Net-Centric and interoperable models and simulations. In the same way that the physical product evolves to the Internet of the Things, its digital model will need to evolve to an Internet of the Simulations or IoS, in which heterogeneous simulations of the diverse physical and cyber subsystems of the smart product can interoperate without restrictions.

One of the main challenges to solve before we can evolve real time distributed simulations to the network, fully connected in the Internet of Simulations, it is how to deal with in an effective way with multiple architectures and protocols. Lead users of distributed simulation, the military simulation & training community, has been working to solve this huge challenge for a long time and the main solutions has been mainly focused on the development of complex gateways solutions that connect the individual simulators' architectures. Gateways based solutions are hard to scale and introduce many restrictions in the interoperability, because the flow of information between the simulators is limited to the capabilities of the gateway. 

This problem has been already a big one when trying to connect only a few man-in-the-loop (virtual) simulators in a common synthetic environment but it is becoming in a much bigger one now that simulation is also requested to integrate with real and live systems in the network. In this case the potential number of architectures and protocols involved is enormous when you compared with the few protocols and architectures in use in the  military sim & training domain. For example, the Internet of the Things is opening new exciting opportunities for the real time simulations, as a type of Cyber Physical System or CPS connected to the IoT compliant systems. (to know more take a look to my former post in this blog). 

To solve this challenge, in Simware, we are taking a more holistic approach to this problem and we are founding our technology in our Data-Centric and Layered Simulation Architecture : LSA. LSA is the first distributed simulation architecture designed to meet the specific simulation requirements of IoT systems dealing with multiple architectures and protocols. 

LSA will not only allow to expand the applications of real time simulations to the IoT but also will enable new business models for the traditional markets of simulations as flight simulation as it is discussed in this blog.

Our Simware product-line  leverages LSA to converge multiple standards and protocols in a common data-centric simulation platform. Simware, just out-of-the-box, is offering integration between DDS, HLA and Web applications, but it can be extended easily using its APIs to support other protocols and standards, as for example JAUS to connect with robotic systems (to know further about the capabilities provided by this kind of integration take a look to the Citius use case). You can find many resources in our website to know more about LSA & Simware but a good starting point is this post in the blog.
​​
​Are you having issues dealing with multi-architecture distributed simulations? Do you want to move toward the Internet of Simulations? Please contact us and we will help you to find the right solution to your pains. 

Jose-Maria Lopez
​jmlopez@simware.es
​General Manager Simware Solutions 

I have read an interesting paper included in the volume 9848 of the SPIE proceedings , "Modeling and Simulation for Defense Systems and Applications XI". This paper (you can downloaded here proceedings.spiedigitallibrary.org/volume.aspx?volumeid=17674) titled "Internet of the Things, a possible change in the distributed modeling and simulation architecture paradigm" by Mark Riecken, Kurt Lessmann and David Schillero, proposes to consider LVC simulation as a type of Cyber Physical Systems or CPS in the Internet of the Things (IoT) as defined by NIST (know further about CPS concept at www.nist.gov/cps/ ).  

Authors recognized many similarities between LVC simulation and IoT/CPS and propose a closer collaboration between both communities to improve both. LVC Simulation can benefit from IOT-/CPS to refresh and sustain its core technologies, because much of the technology employed at LVC simulation was developed prior to the proliferation of internet.  IOT/CPS can leverage LVC simulation to experiment and test complex scenarios in synthetic playgrounds. Paper presents two specific use cases of IOT/CPS that would benefit from distributed simulation :

1. LVC simulation as a decision support tool to decide the best course of action in an emergency in which various biosensors (the CPS's connected in a smart city emergency network) located throughout one or more healthcare facilities detect a pathogen and the authorities must manage the outbreak of the pathogen event in a dense urban area.
2. Testing new IoT capabilities for a product line of appliances. Manufacturer uses LVC simulation to test virtual prototypes of the new capability in a synthetic scenario integrated with live systems (real applicances already located at homes and in the test facilities of the manufacturer )

At a conclusion, authors are proposing to expand the collaboration between both communities using special sessions or forums at SPIE and similar venues that could evolve to permanent structures in which both communities could work together on common protocols, standardization processes, shared data models, LVC in CPS and modeling cybersecurity. This paper uses as references of the work to be done our study group at SISO for the Layered Simulation Architecture and the integration of different standards in our Simware platform. 

We do support this proposal because we fully agree with the vision of the authors.  Our R&D in Simware platform and our work at SISO and NATO CoIs related to LVC simulation are already pursuing this collaboration and a seamless interoperability between simulation and IOT technologies and processes. We called this vision the Internet of Simulations and it will enable the evolution of a niche technology, as it is distributed simulation nowadays to the mainstream, useful for exciting new applications as the use cases explored in the paper.

What do you think? Are you ready to collaborate with us on the future of simulation, the Internet of Simulations? If you are, please send me an email to jmlopez@simware.


Jose M Lopez