RPI: Setting up Raspberry Pi 3B+ to be accessible from anywhere

I have a simple trading bot that I currently run from a WSL Ubuntu session in my home computer. In this post, I'll set up a headless Raspberry Pi that can host that session. I still want to be able to control it from my mobile, so I need to do the following:

Set up RPI headless, Install and Disable screen saver

After installation, the board became unresponsive after some 10 minutes. I found

https://forums.raspberrypi.com/viewtopic.php?t=286000

Now, the RPI works fine for at least a couple of days.

Install app for accessing RPI from mobile

I found RaspController that is available for both Android and Iphone that gives a good interface to the RPI. 

Set up Dynamic DNS using NoIP

I activated a Dynamic DNS session using NoIP with both a host name and ipv4 IP.

As my router was too old, I decided to configure my RPI instead using the instructions below

https://www.noip.com/support/knowledgebase/install-linux-3-x-dynamic-update-client-duc

Make NoIP start automatically when booting

https://www.noip.com/support/knowledgebase/running-linux-duc-v3-0-startup-2

Configure Python and setup venv

There is plenty of documentation online

Nohup to run detached

Finally, since I need the program to run in the background, I used the nohup command to start the script,

https://www.digitalocean.com/community/tutorials/nohup-command-in-linux

TrafficControl: Bind Python into Program, or Port Scripts to Code?

Currently my program expects the user to have python installed on his/her computer. This will be a severe blocker for any portability and I will need to handle this, either by binding Python into C++ or porting the scripts into C++ code. 

Binding:

Advantages: I would be able to use the existing code

Disadvantages: The binaries may be bigger and that will be a disadvantage when deploying on web assembly

Porting:

Advantages with porting to C++: Everything will be in C++

Disadvantage: Porting the application to C++ may be time consuming.

For now, I'll try porting the code to C++/Qt/Regular expressions.

Legacy Code:

The program creates a shell process that starts python.exe (hard coded for Windows), with a hard coded path to the script. 

In my last commit, I added code to src/networkDesigner.cpp. For the XML/QML handling, I used some code that I found at stack overflow.

This change isn't visible but it is a big step towards Web Assembly.

TrafficControl: Exploring Qt WebAssembly

I've seen that it is possible to create executables that can be run from within a web browser. This is very interesting for my pet project Traffic Control and I'll spend some time exploring this.

What is web assembly and why bother?

Desktop applications are a bit inconvenient for the end user. He/she needs to find an installation package that goes with the particular operating system - that requires some competence and also some administrator rights. Further, there is a risk of viruses. After using the software, the software lies on the computer until removed. It will take quite a bit of persuation to make people install my software. 

With a Web Assembly app, it is different. The program will be linked from a simple web page that loads the executable to a sandboxed environment inside the web browser the user is using. This will be independent on the hardware architecture and operating system. No installation and no cleanup.

Qt supports web assembly - this is the key point of Qt: that it is a programming framework that can be used for different platforms. If I'm able to port TrafficControl to web assembly and deploy on a web server, the program will be able to run from any connected device with one of the bigger web browsers.

What would that mean for me? 

For me, the program would be deployable to any unit that runs one of the major web browsers. To deploy the software, I would need to set up a web server, or use an existing platform that hosts the binary. 

Traffic Control would need some refactoring:

• The program depends on python for converting files. That needs to be in C++
• I need to consider how to access local files and examples
• I need to look into the user interface - should I migrate to QML for all user interface?

Step-by-step

1. Install an updated Python version and Emscripten 
2. Setup environment for Qt Web Assembly
1. Specify C and C++ compiler
2. Other settings?
3.  Configure for simple C++/QML application - QML Demo
1. Download from GitHub
2. Compile for desktop
3. Compile for Qt Web Assembly
4. Open in Web Assembly
4. Adopt TrafficControl to WebAssembly
1. Find out how to store the TNF/TNM/TNO files when using webAssembly? Initially, use read-only and disable the conversion features?
2. Handle loading of traffic network files
3. Port the conversion from KML to Traffic Map File/Traffic Network file to either
1. Python in-project code, or
2. C++ Code
5. Deploy application on web server

Emscripten is used for compiling to webAssembly and it is based on Python and needs a late Python version. Installing Python is straight forward. 

I installed Emscripten from the git repo:

Yes, I'm using a screen shot.
If you're copying commands, you'd better copy from the formal documentation.

As I'm using Qt 5.15.2, I need to install the corresponding Emscripten version (1.39.8) and activate it permanently (use the same version for each instance:

However, after activating emsdk, the python path is missing. I found out that I need to run the shell as an administrator.

In Qt, I currently use the auto-detected "Local PC" as a target for the binaries. I added a web assembly device where I pointed to the Emscripten folder.

Once the device is configured, I need to set up a kit for the compilation. 

Finally, I specified that the project should have the web assembly option in the "Projects" tab.

This time, I used an old QML demo project that I have on my Github account. In that project, I explored some basic QML features and how to interact from C++. 

During the build process, I saw a warning - my installation of Qt was actually built with Emscripten 1.39.7, not 1.39.8 as the documentation said. The build was still successful.

After the build was completed, I started it by loading the local web site:

There it is! A C++/QML application that is running inside a web browser. The html page can be embedded into another web page on a web server. For my simple QML application, the binary used ~20 MB for the release build and ~27 MB for the debug build.

In the next blog post, I'll explore if it is possible to port TrafficControl to web assembly. For Qt 5.15.2, Qt Location seems not to be supported for web assembly. For Qt6, QtLocation and QtPositioning didn't work out of the box, maybe it is possible to find a workaround.

TrafficControl: Doxygen

In order to get a better understanding of the code structure of my project TrafficControl, I need to use Doxygen.

Doxygen is one of the most popular tools for documenting source code. The developer adds Doxygen-style comments to the source. When Doxygen analyses the code, it uses the structure along with the comments to generate a set of files with an overview of the program, and the relations.

Step 1 - Select how to use Doxygen

At first, I wanted to integrate Doxygen into Qt Creator. After digging into how to do it, it seemed to be quite difficult. I would either need to find pre-compiled binaries that matches my system and my Qt installation, or recompile Qt Creator. I tried to follow the video below, but the URL they used was broken.

I'm not the only one to pay too little attention to my pet projects.

Instead, I installed the program on my computer from the Doxygen web site. I also installed GraphViz for creating graphs - that will be useful for getting an overview of my code.

Step 2 - Configuring Doxygen

The next step was to use an appropriate set of parameters for my project. The most important ones are listed below:

• QT_AUTOBRIEF           = YES - Assume that the first line of the class descriptor to be the brief description.
• EXTRACT_ALL            = YES - Assume all entities to be documented. Maybe not needed?
• EXTRACT_PRIVATE    = YES
• EXTRACT_PRIV_VIRTUAL = YES
• EXTRACT_STATIC      = YES
• EXTRACT_LOCAL_METHODS = YES 
• INPUT          = C:/Users/gusta/GIT/trafficControl-dev/src \ - All folders to be documented. IMPORTANT!
•                          C:/Users/gusta/GIT/trafficControl-dev/inc \
•                          C:/Users/gusta/GIT/trafficControl-dev/test
• RECURSIVE                  = YES - Include sub directories
• HAVE_DOT                    = YES - Use Dot tool from GraphVix to generate graphs - IMPORTANT!
• UML_LOOK                   = YES - Make graphs look more like UML
• DOT_UML_DETAILS    = YES - Add bytes and info to UML graphs
• CALL_GRAPH               = YES
• CALLER_GRAPH          = YES
• DOT_PATH                     = C:/Program Files/Graphviz/bin

Step 3 - Insights from Doxygen

My first insight is that there is lot's of more to learn about Doxygen, and that it will be useful to check the relations between different parts of my program. Listing everything will be too exhaustive for a blog post, and I don't know yet know how to publish Doxygen output in a reasonable way. 

I'll show some examples of the graphs below:

Train::move() will call different functions depending on the current state.

A lot of functions are calling NetworkControl::parseCmd()

A lot of functions are calling parseCmd. This means that bugs that are introduced hwere will have a huge impact - extensive testing will be needed.

TrafficControl: Migrating Back to QtTest

After seeing new strange issues with all test cases, (compilation issues, I guess that it was a mismatch in compilation for the test files), I've decided to try moving back to QtTest for my project. 

I had some issues with Qt's Meta Object Compiler, but after a while I got that working (the name of the moc file must match the source file).

It took me several hours to refine the test cases, and I saw that I could merge several test cases to one. For example, some test cases check parameters after initialization, and there is no reason to have several test cases for that. I was also able to simplify the syntax and use the command parser.

The next step will be to use Doxygen for my code. I need to get a better overview of the structure, and I think Doxygen will be the best tool for it. 

Qt: Upgrading from Qt 5.12.2 to Qt 6.3

After a couple of years exploring Machine Learning, IoT and Python, I'll return to TrafficControl. The current version of my project is Qt 5.12 - more than three years old. 

Upgrading from Qt 5.12 to Qt 6.3.2 seems to be easier said than done. It is recommended to start upgrading to Qt 5.15 and after that upgrading to Qt 6.3.2.

The steps for this project are:

Build and test the program as it is, with the installed version of Qt. 

There are sometimes changes to external services, such as map provider. I may also have done changes that I forgot about several years ago. 

Further, my antivirus program scans the binary that I compile, and that takes some time.

https://forum.qt.io/topic/90617/avast-alarm-to-qt-5-10-1-static-build/4

I solved the issue by adding an exception to the anti virus program. 

Push the current version of the program to GitHub. 

I saw that I had some staged changes that I had to check. After generating an access token, I was finally able to push the changes to GitHub.

Test program for Qt 5.15

The recommended way to port an application from Qt5 to Qt6 is to upgrade to/compile for the latest Qt5 version (Qt 5.15) first, and resolve any issues. After that, one can upgrade to Qt6.

I started with disabling any Qt modules that was deprecated between Qt 5.15 by adding the following line to the pro file:

DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x050F00

After some minor fixes, I got the program to run and almost all tests to pass.

Upgrade Qt 5.12 to Qt 6.3.2

I used the online installer and I found some strange issues when trying to run the program. 

The issue was that the module QtLocation (a module that is enabling maps on Qt applications) hasn't yet been ported to Qt 6. The work is ongoing and can be tracked here.

This means that I need to postpone the migration to Qt 6 and use Qt 5.15 instead.

Using Qt 5.15 - a new crash

After installing Qt 5.15, I got a software crash when running the application. 

The program worked fine in Qt 5.12 so I assume there are changes in Qt between 5.12 and 5.15 that made my code fail.

My classes trainListModel, stationListModel and trackListModel are Model/View implementations for the network and are used to simplify the data presentation. They all inherit the trafficDataModel, which in turn inherits the QAbstractTableModel, which inherits QAbstractItemModel, which inherits QObject.

There is a flaw when I call the insertRows() function, where the position where I want to add rows is calculated as a number that I just incremented. When changing the position to the actual list of the train/track/stationListModel, the program works fine. 

The next step will be to review the code and push it to GitHub. After that, I'll investigate what to implement next.

Hiding Information in Repos

When adding my codebase to Github, I want to avoid telling the world about my login credentials. To do this, I'll save the information in separate files that will be imported during runtime. I won't push those files to my Github repos and my credentials will remain secret.

I'll focus on Python and Bash in this blog post.

Bash scripts:

Create a file, for example "secrets.cfg" with the content:

password=testa123

In the bash script, source the file. Now, the parameters will be available as $password:

$ source settings.cfg
$ echo $password
testa123

Python scripts:

I'll use dotenv to set environment variables from a .env file that is ignored by Git.

I'll create a file ".env" with the content

DOMAIN=fatsug.example.org

The python script will use dotenv, so you need to install it using pip.

pip install python-dotenv

I'll make the python script look for a .env file in the users home folder. This is how the python script looks like:

import os
from os.path import expanduser
from dotenv import load_dotenv

load_dotenv(expanduser("~")+'\.env')
print(os.getenv('DOMAIN'))

For convenience, I can merge the secrets.cfg and .env files to one. Thus, I'll have all my user credentials at one location.