TrafficControl: Exploring Qt WebAssembly

I've seen that it is possible to create executables that can be run from within a web browser. This is very interesting for my pet project Traffic Control and I'll spend some time exploring this.

What is web assembly and why bother?

Desktop applications are a bit inconvenient for the end user. He/she needs to find an installation package that goes with the particular operating system - that requires some competence and also some administrator rights. Further, there is a risk of viruses. After using the software, the software lies on the computer until removed. It will take quite a bit of persuation to make people install my software. 

With a Web Assembly app, it is different. The program will be linked from a simple web page that loads the executable to a sandboxed environment inside the web browser the user is using. This will be independent on the hardware architecture and operating system. No installation and no cleanup.

Qt supports web assembly - this is the key point of Qt: that it is a programming framework that can be used for different platforms. If I'm able to port TrafficControl to web assembly and deploy on a web server, the program will be able to run from any connected device with one of the bigger web browsers.

What would that mean for me? 

For me, the program would be deployable to any unit that runs one of the major web browsers. To deploy the software, I would need to set up a web server, or use an existing platform that hosts the binary. 

Traffic Control would need some refactoring:

• The program depends on python for converting files. That needs to be in C++
• I need to consider how to access local files and examples
• I need to look into the user interface - should I migrate to QML for all user interface?

Step-by-step

1. Install an updated Python version and Emscripten 
2. Setup environment for Qt Web Assembly
1. Specify C and C++ compiler
2. Other settings?
3.  Configure for simple C++/QML application - QML Demo
1. Download from GitHub
2. Compile for desktop
3. Compile for Qt Web Assembly
4. Open in Web Assembly
4. Adopt TrafficControl to WebAssembly
1. Find out how to store the TNF/TNM/TNO files when using webAssembly? Initially, use read-only and disable the conversion features?
2. Handle loading of traffic network files
3. Port the conversion from KML to Traffic Map File/Traffic Network file to either
1. Python in-project code, or
2. C++ Code
5. Deploy application on web server

Emscripten is used for compiling to webAssembly and it is based on Python and needs a late Python version. Installing Python is straight forward. 

I installed Emscripten from the git repo:

Yes, I'm using a screen shot.
If you're copying commands, you'd better copy from the formal documentation.

As I'm using Qt 5.15.2, I need to install the corresponding Emscripten version (1.39.8) and activate it permanently (use the same version for each instance:

However, after activating emsdk, the python path is missing. I found out that I need to run the shell as an administrator.

In Qt, I currently use the auto-detected "Local PC" as a target for the binaries. I added a web assembly device where I pointed to the Emscripten folder.

Once the device is configured, I need to set up a kit for the compilation. 

Finally, I specified that the project should have the web assembly option in the "Projects" tab.

This time, I used an old QML demo project that I have on my Github account. In that project, I explored some basic QML features and how to interact from C++. 

During the build process, I saw a warning - my installation of Qt was actually built with Emscripten 1.39.7, not 1.39.8 as the documentation said. The build was still successful.

After the build was completed, I started it by loading the local web site:

There it is! A C++/QML application that is running inside a web browser. The html page can be embedded into another web page on a web server. For my simple QML application, the binary used ~20 MB for the release build and ~27 MB for the debug build.

In the next blog post, I'll explore if it is possible to port TrafficControl to web assembly. For Qt 5.15.2, Qt Location seems not to be supported for web assembly. For Qt6, QtLocation and QtPositioning didn't work out of the box, maybe it is possible to find a workaround.

TrafficControl: Doxygen

In order to get a better understanding of the code structure of my project TrafficControl, I need to use Doxygen.

Doxygen is one of the most popular tools for documenting source code. The developer adds Doxygen-style comments to the source. When Doxygen analyses the code, it uses the structure along with the comments to generate a set of files with an overview of the program, and the relations.

Step 1 - Select how to use Doxygen

At first, I wanted to integrate Doxygen into Qt Creator. After digging into how to do it, it seemed to be quite difficult. I would either need to find pre-compiled binaries that matches my system and my Qt installation, or recompile Qt Creator. I tried to follow the video below, but the URL they used was broken.

I'm not the only one to pay too little attention to my pet projects.

Instead, I installed the program on my computer from the Doxygen web site. I also installed GraphViz for creating graphs - that will be useful for getting an overview of my code.

Step 2 - Configuring Doxygen

The next step was to use an appropriate set of parameters for my project. The most important ones are listed below:

• QT_AUTOBRIEF           = YES - Assume that the first line of the class descriptor to be the brief description.
• EXTRACT_ALL            = YES - Assume all entities to be documented. Maybe not needed?
• EXTRACT_PRIVATE    = YES
• EXTRACT_PRIV_VIRTUAL = YES
• EXTRACT_STATIC      = YES
• EXTRACT_LOCAL_METHODS = YES 
• INPUT          = C:/Users/gusta/GIT/trafficControl-dev/src \ - All folders to be documented. IMPORTANT!
•                          C:/Users/gusta/GIT/trafficControl-dev/inc \
•                          C:/Users/gusta/GIT/trafficControl-dev/test
• RECURSIVE                  = YES - Include sub directories
• HAVE_DOT                    = YES - Use Dot tool from GraphVix to generate graphs - IMPORTANT!
• UML_LOOK                   = YES - Make graphs look more like UML
• DOT_UML_DETAILS    = YES - Add bytes and info to UML graphs
• CALL_GRAPH               = YES
• CALLER_GRAPH          = YES
• DOT_PATH                     = C:/Program Files/Graphviz/bin

Step 3 - Insights from Doxygen

My first insight is that there is lot's of more to learn about Doxygen, and that it will be useful to check the relations between different parts of my program. Listing everything will be too exhaustive for a blog post, and I don't know yet know how to publish Doxygen output in a reasonable way. 

I'll show some examples of the graphs below:

Train::move() will call different functions depending on the current state.

A lot of functions are calling NetworkControl::parseCmd()

A lot of functions are calling parseCmd. This means that bugs that are introduced hwere will have a huge impact - extensive testing will be needed.

TrafficControl: Migrating Back to QtTest

After seeing new strange issues with all test cases, (compilation issues, I guess that it was a mismatch in compilation for the test files), I've decided to try moving back to QtTest for my project. 

I had some issues with Qt's Meta Object Compiler, but after a while I got that working (the name of the moc file must match the source file).

It took me several hours to refine the test cases, and I saw that I could merge several test cases to one. For example, some test cases check parameters after initialization, and there is no reason to have several test cases for that. I was also able to simplify the syntax and use the command parser.

The next step will be to use Doxygen for my code. I need to get a better overview of the structure, and I think Doxygen will be the best tool for it. 

Qt: Upgrading from Qt 5.12.2 to Qt 6.3

After a couple of years exploring Machine Learning, IoT and Python, I'll return to TrafficControl. The current version of my project is Qt 5.12 - more than three years old. 

Upgrading from Qt 5.12 to Qt 6.3.2 seems to be easier said than done. It is recommended to start upgrading to Qt 5.15 and after that upgrading to Qt 6.3.2.

The steps for this project are:

Build and test the program as it is, with the installed version of Qt. 

There are sometimes changes to external services, such as map provider. I may also have done changes that I forgot about several years ago. 

Further, my antivirus program scans the binary that I compile, and that takes some time.

https://forum.qt.io/topic/90617/avast-alarm-to-qt-5-10-1-static-build/4

I solved the issue by adding an exception to the anti virus program. 

Push the current version of the program to GitHub. 

I saw that I had some staged changes that I had to check. After generating an access token, I was finally able to push the changes to GitHub.

Test program for Qt 5.15

The recommended way to port an application from Qt5 to Qt6 is to upgrade to/compile for the latest Qt5 version (Qt 5.15) first, and resolve any issues. After that, one can upgrade to Qt6.

I started with disabling any Qt modules that was deprecated between Qt 5.15 by adding the following line to the pro file:

DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x050F00

After some minor fixes, I got the program to run and almost all tests to pass.

Upgrade Qt 5.12 to Qt 6.3.2

I used the online installer and I found some strange issues when trying to run the program. 

The issue was that the module QtLocation (a module that is enabling maps on Qt applications) hasn't yet been ported to Qt 6. The work is ongoing and can be tracked here.

This means that I need to postpone the migration to Qt 6 and use Qt 5.15 instead.

Using Qt 5.15 - a new crash

After installing Qt 5.15, I got a software crash when running the application. 

The program worked fine in Qt 5.12 so I assume there are changes in Qt between 5.12 and 5.15 that made my code fail.

My classes trainListModel, stationListModel and trackListModel are Model/View implementations for the network and are used to simplify the data presentation. They all inherit the trafficDataModel, which in turn inherits the QAbstractTableModel, which inherits QAbstractItemModel, which inherits QObject.

There is a flaw when I call the insertRows() function, where the position where I want to add rows is calculated as a number that I just incremented. When changing the position to the actual list of the train/track/stationListModel, the program works fine. 

The next step will be to review the code and push it to GitHub. After that, I'll investigate what to implement next.

Hiding Information in Repos

When adding my codebase to Github, I want to avoid telling the world about my login credentials. To do this, I'll save the information in separate files that will be imported during runtime. I won't push those files to my Github repos and my credentials will remain secret.

I'll focus on Python and Bash in this blog post.

Bash scripts:

Create a file, for example "secrets.cfg" with the content:

password=testa123

In the bash script, source the file. Now, the parameters will be available as $password:

$ source settings.cfg
$ echo $password
testa123

Python scripts:

I'll use dotenv to set environment variables from a .env file that is ignored by Git.

I'll create a file ".env" with the content

DOMAIN=fatsug.example.org

The python script will use dotenv, so you need to install it using pip.

pip install python-dotenv

I'll make the python script look for a .env file in the users home folder. This is how the python script looks like:

import os
from os.path import expanduser
from dotenv import load_dotenv

load_dotenv(expanduser("~")+'\.env')
print(os.getenv('DOMAIN'))

For convenience, I can merge the secrets.cfg and .env files to one. Thus, I'll have all my user credentials at one location. 

New Repos to GitHub

 After some months of inactivity, I've been looking into my codebase again, with some challenges: 

• It is easy to forget where I've placed the code on my computer. Sometimes, there are different versions of the scripts at different locations. I want to have the code at one location. 
• It is easy to forget the syntax and order of the scripts. I want to document the projects at the same location as the code itself.
• I need to centralize the API keys, instead of having them directly in the code.

The solution is to add everything to different GitHub repositories. The repositories worth mentioning are:

cutetrains/hiddenProjects - Projects that I won't share with others at all. These projects include web scrapers and tools for price prediction for financial assets. None of these projects are illegal (this goes for all my repos). No link for obvious reasons.

cutetrains/IOTProjects - Projects for home automation, facial recognition and robotics. I'll use this repo for downloading code into my Raspberry PI. Some of the projects communicate with Arduino boards. For the Arduino part, I'll have the code at my Arduino.cc account.

cutetrains/trafficControl-dev - TrafficControl, a C++/QML train network simulator.

cutetrains/TravelTimeCalculator (Discontinued) - A small Android App that I wrote to get some understanding on Android App development. I'm working in the 4G/5G base station industry with experience from 2G/3G/4G mobile phone verification. I wanted to have some clue about what app development is about.

cuteTrains/StockToDatabase (Discontinued) - A web scraper that collects key numbers for stocks, converting data into a database and performs some machine learning on the data.

I will add instructions and documentation for the non-discontinued public repos.

I'll spend some time on the IOT projects in the near future. 

IOT: First Steps for the Robot Project

One important part of my IOT project will be to have a small robot that can move around in my house and stream video/pictures with an acceptable delay.

I spotted a platform/robot car that I can start with at a Swedish electronics store and I've spent a couple of evenings with my four year old son building the basic design. In the original setup, the robot uses an ultrasound sensor to determine in what direction it will move.

The robot car consists of:

• An Arduino board (a very limited computer)
• A H bridge that controls the engines
• A pair of simple analog engines that takes 5-10 volts as input.
• An ultra sound sensor
• A servo motor that moves the sensor in different directions.
• A battery pack of 6 AA/LR6 batteries that gives 9 volts.

I've disconnected the original battery pack and replaced it with a power bank.
Later, I've removed the battery pack.

Connecting a Raspberry PI Computer.

The original setup feeds 9 volts both to the engines and to the Arduino board via the DC jack. However, I want to use a Raspberry PI computer with a camera module. 

With the Raspberry PI computer, I can connect to the internet over Wi-Fi, setup a video stream or a web server and do some imaging. An USB/Serial connection can handle the interface between the Arduino and the Raspberry PI.

However, it is not that straightforward to connect a Raspberry PI to the Arduino in this setup. The manufacturer recommends that the battery pack is disconnected before connecting to the USB port on the Arduino. There are two options to communicate between the Arduino and another computer:

1. Keep the batteries and connect a cable to GPIO pins on the Arduino and Raspberry PI. Since the boards are operating on different voltages (5 V and 3.3 V respectively), I would need a level shifter in between.
2. Remove the battery pack and connect the Arduino to the Raspberry PI. That would mean that the motors will get only 5 V. This is the easiest solution, I'll maybe explore the other solution later. Another advantage is that I'll reduce the weight of the vehicle.

I've replaced the battery pack with a LEGO rig for the Raspberry PI computer. 

In the next blog post, I'll modify the code of the Arduino so that I can send commands to it via the Raspberry PI.