StockReader: What Data I Want and the Final Script

I want to acquire the following data for stocks on Swedish markets:

• Name, DI
• Price, DI
• Earning per share, DI
• P/E, DI
• Book value per share, DI
• Price per adjusted equity, DI 
• Yield, DI
• Yield per share, DI
• Profit margin, DI
• RSI, DI
• Date of next report, DI
• Date of next yield, DI

(In Swedish: Namn;Kurs;Vinst per aktie;P E;Kapital per aktie;P JEK;Direktavkastning;Utdelning per aktie;Vinstmarginal;RSI;Nasta rapport;Utdelningsdatum;TA)

I'll start with fetching 

https://www.di.se/robots.txt

If that allows, I'll fetch the entire stock list:

https://trader.di.se/index.php/stocklist/index/35206/none/pop

For each stock in that list, I'll use the hyperlink to get the stocks unique id number:

https://trader.di.se/index.php/quote/kurs/730

This is done using re.findall("/index.php/quote/kurs/[0-9]{1,10}", stockListResponse)

I'll use that number to fetch the key numbers of that stock:

https://trader.di.se/index.php/quote/nyckeltal/730

The code is shown below:

The new script is 24 lines of code, compared with the original C program with 300+ lines of code.

The data is encapsulated in table tags. The regex command generates a list of all matches, and I cherry-pick the data I want.

The next step was to add StockReader to scheduled tasks in windows.

StockReader: Respecting Robots.txt and Initial Scraping

Robots has some rules to follow, or break if they are nasty. Those rules are defined in the robots.txt file on the particular web site.

I'll start by creating a stub python script that will access a web page and print that to a console.

First, I need to install a module: requests. To do that, I need to install a Python package manager, PIP:
In https://bootstrap.pypa.io/, download and run getpip.py.

To install a package,
Navigate to the folder where pip.exe is located.
Run pip install requests

Now, requests can be imported and used

In my case, I'll use DI.se. The corresponding robots.txt file will be analyzed. If that page allows, I'll download the stock list and analyse the stocks in the list

TrafficControl: Locking Tracks for Trains - Tests and Visualization - QML

The visualization of locks in QML will be dynamic, with a green/red dot at the start/end station if the track is locked:.

• FREE: If the track is free, no dots will be shown. The track will be green. 
• END/START: If the track is locked in both directions, red dots will be shown at both end and start station. The track will be red.
• LOCKED: If the track is locked at start or end station side, a red dot will appear near that station and a green dot will appear at the eother station. The track will be yellow.

When the lock status is changed. a signal is sent to QML with the new state of the track.

If the state goes from FREE to any other state, two dots are created close to the start and end station. To avoid overlayed dots, the QML part will search the QML tree for existing dots and add new signal dots only if there are no existing dotsfor that particular track..

If the last train leaves a track, that track will be set to FREE and the corresponding signal dots will be removed in QML.

When removing QML items, it takes some time for those changes to take place. That caused some strange behaviour when a train arrived to a station, where another train is waiting for the same track:
Train 1 arrives to station 1 from track 1, existing dots are removed from track 1.
Train 2 enters track 1. QML looks for signal dots for track 1 and finds them as the removal is still in process. The signal dots are repainted.
The signal dots for the track 1 are finally removed.

StockReader: Re-implement in Python

One of my first larger software projects was StockReader. That is a C++ web scraper for stocks.

The final result of StockReader

Now, I need to re-implement the program for four reasons: 

1. My firewall doesn't like blocks executables that sends several get requests
2. The source code is horrible and needs to be refactored completely in order to be more maintainable
3. I should be able to use regular expressions.
4. The program should be more automatic, selecting stocks automatically

I will not publish the program on Github, but I'll post some issues that I will discuss.

The first step will be to implement a script that gets a web page from an URL. From that html code, I'll extract the URLs for the key number sections for the stocks.

When creating code, it is crucial to understand what one is writing. It is important to resist the temptation to copy some code from stackoverflow and instead focus on really understand what one is writing.

In the Python documentation, there is a section about regular expressions.

TrafficControl: Locking Tracks for Trains - Tests and Visualization

After adding locks for tracks, I need to both visualize them and test them.

The tests are quite straight-forward, but they require some code.

• The first test case, trainListAddRmUpstream verifies that is it possible to add two trains in upstream on the same track, with a distance of more than 3000 m between the trains. It also verifies that the first train is removed first.
• The second tedst case, trainListAddDownstream verifies that a train that wants to enter a used track, where a train is moving in the opposite direction, will wait until the first train has left the track.

For the visualization, I'll change both model/view and the QML part.

Model/View:
A new column will be added for each track: Lock. The different states will be: Free, Start, End and Locked.

When something is changed for a track, the method Track::emitChangedSignal(trackID) is called. That method compiles a QStringList that contains the messages that shall be sent to the datamodel. The lock status is added as a string to the string list that is sent to the track data model. Also, a column is added for the data model.

Three tracks are locked. For example, dHja_Lun_E is locked at the end station (LundC). This means that trains can enter at Hjarup but not at LundC.

The next blog post will describe how I added the locks in the map.

TrafficControl: Locking Tracks for Trains - Code

Preventing trains to collide is an essential of any traffic controller. In this blog post, I'll implement some basic functionality for that purpose.

I'll use a simplistic model with two boolean variables: lockedUpStream (end station locked) and lockedDownStream (start station locked) with corresponding get and set classes.

When the train is in readyState (in a station), it will try to find a suitable track to the target station in Station::findLeavingTrackIndexToStation(int targetStationID).

If the train finds a suitable track and enters it, it will set a lock on the track:

• When a train enters a track in upstream direction, the lockedDownStream will be set to true.
• When a train enters a track in downstream  direction, the lockedUpStream will be set to true (see above). This happens in track::findLeavingTrackIndexToStation.
• When the last train leaves track in upstream direction, the lockedDownStream will be set to false.
• When the last train leaves track in downstream direction, the lockedUpStream will be set to false.

If all suitable tracks are locked, the train will wait one second and search for tracks again. 

As a safeguard, the track will also consider the locks when adding a train:

The locks are reset only if there is no train on the track.

The next blog post will cover tests and visualisation of track locks.

Future improvements:

For a certain time after a train enters the track, the track should be marked as red and no trains should be able to enter from any direction.

TrafficControl: Making User Interface Flexible with DockWidgets

Some readers may have seen that the user interface has a new look. Now, the user can select which widgets should be visible at run-time, how big they should be and where in the program they should be.

The user interface is designed in the file TrafficControl.ui, that is based on XML. That file contains the different widgets that makes up the user interface. 

When compiling, TrafficControl.ui generates a corresponding header file that  is called from the program (trafficControl.cpp/h):

The file ui_trafficControl.h is generated at runtime from the file trafficControl.ui

I've added qDockWidgets to the form that contains the elements. A qDockWidget is basically a container that can dock to different borders of the applications interface. The user interface assigns some space on the form, depending on the size of the contents, other widgets and the user's decisions.

A qDockWidget can be closed (set to invisible) and floating (presented in a separate window) if the user wants to. A list of the qDockWidgets is shown if the user right-clicks on the title for a qDockWidget.

This is needed as my program grows and there will be much more information to present to the user.

I've recently learnt how to use qDockWidgets and I'll explore more of its abilities in a future blog post. 

Future work: 

Add a panel of pushButtons, one for each qDockWidget that is available in the program. The buttons should have icons that describe the contents, for example Map, Train List, Basic Controls, Log window...