StockAnalyzer: Validating the Raw Data - Broken input data and Missing Report date

My Stock project is progressing and I'm currently adding fixes for flaws in the input data that I want to add to a SQL database. In my previous blog post, I added a fix for records that has no information about the stock price.

In this blog post, I address some more issues:

Input Data Replaced by HTML Tag

The program tries to parse ass="BrodTextWhite" width="12">? to a float number. I don't know why this string has replaced the data.

I will handle this by simply removing the bad record. I haven't seen that for other stocks and there is no information in the string that can recover the missing data.

Invalid Data for Dividend or Report Dates

Another issue that I saw was that some records had an invalid date for the report. I solved that by checking that the string can be parsed to a date. If not, I'll feed the NULL value to the database.

Now, the database has a NULL value when the date is missing.

Empty Profit Margin

Now, I was able to scan three stock records for almost five years (2013-07-28 to 2018-06-13). For AAK, the program is trying to parse an empty string to a value.

There are some empty values: Earnings per share, Capital per share, Dividend per share and Profit Margin. I can recover everything except for the Profit Margin, that requires information about sales in relation to the production costs. Since this information is important, I will add the stock information but set the profit margin to NULL.

Populating Missing Data?

I will create functions for filling in some data that is missing, if corresponding data can help calculating that data.

For the AAK record of 2018-06-13, the Earnings per share is missing.

I will handle this in two steps (next blog post):

First, I'll add try-catch clauses to handle parsing errors from string to floats.

After that, I'll add functions to verify and populate null entries of the key numbers.

TravelTimeCalculator: Fixing Bugs, Testing on Physical Phone and Closing the Project

So far, I've been testing the app on a simulated device from Android Studio.

The next step is to test the app on a real device. I'm using a Samsung SM-J500FN (Android 6.01, API23). That phone is some four years old and I wasn't sure how responsive the app would be on that phone.

Once I installed the app, it worked like a charm. The app is responsive and runs smoother than when running in the virtual devices. That makes sense, since a virtual Android device is a full-stack emulation of the application processor of a smartphone. Having that running on a Windows PC will require quite a lot of system resources.

The last bug to fix was the apps inability to find valid values from the database when using reversed directions. I fixed it by checking whether the directions are in direct or reversed mode in the database.

Why TravelTimeCalculator Won't be Released
The purpose of this project for me is to learn Android - not to release an app that can be downloaded.

An app that is available on Google Play must have been tested thoroughly on several phones, different settings and lots of configurations. And once released, there should be someone to fix the incoming bug reports. I don't have time and motivation to do that.

Further, If I would release it on Google Play, I'd need to provide a Directions API key so that the user can get any directions. And if the app would be popular and generate lots of requests, I'd get a bill from Google. In any case, I need to consider whether to make that app free and risk getting a bill or charge a small fee for the app. The quality of the app can't justify charging for it.

Therefore, I'll keep the app as it is. Anyone can download it from Github and build it him/herself.

In the spring, I'll continue with TrafficControl and also have a small project on AI/Machine Learning.

TravelTimeCalculator: Applying Settings

In this post, I'll use the values in the settings to estimate the cheapest travel mode.

The DBHelper method getAllDistanceDuration is expanded to return cost as well. It is using the following parameters:

• Origin 
• Destination
• Emission cost (from settings)
• Time cost (from settings)
• Start and stop time for bicycling (from settings)
• Start and stop time for driving (from settings)
• Cost per kilometer for driving (from settings)
• Emissions per kilometer for driving (from settings)
• Ticket cost for transit (from settings)
• Emissions per kilometer for transit (from settings)

I query the distances and durations from the database:

The values are stored in an array of integers.

The settings are retrieved from SharedPreferences.

I calculate the relevant costs and compare them with each other. The smallest cost is returned from the function,

Calculation of costs from the database. 

Finally, I update the marker with the shortest cost and the corresponding mode of transport.

Finally, some notes about bugs for this project:

One issue was unrealistic calculations for travel mode; "DRIVING", that is solved now.

Two issues are remaining:

The second one will be tricky and the first one probably needs some Googling.

Once those two issues are solved and I've cleaned the code, I'll stop developing this app. In the next blog post, I'll discuss the journey of learning to build an Android App, and why this app won't reach Google Play for now. 

TravelTimeCalculator: Implementing Settings

The first step is to add the setting menus for the app by adding the elements to the xml file:

I'm keeping the settings from the example for this moment, since I may need them later to understand how to bind the settings to the app code.

After that, I want to add the current setting below the setting title. This already works for the comparison mode setting (Distance/Duration/Cost), so let's look at that setting.

When selecting a new comparison mode (Cost, Distance or Duration), the new value is sent as an integer.

When changing that setting, onPreferenceChange is trigged with two parameters: "Compare duration, distance or Cost? Duration" and an integer value that corresponds to the new value (1 for Cost, 2 for Distance and 3 for Duration).

The first part of the string corresponds to the title tag of the item and the second tag corresponds to the value of the setting before it is changed. The integer corresponds to the new value.

Defining settings isn't enough - something must happen when changing them. When a setting is changed, the corresponding entry in the settings screen is changed. This is done by binding the values.

When any setting is changed, a method is invoked that saves all the settings to variables inside TravelTimeHandler. Changing parameters directly in another class is a dirty solution, but it will do for now.

The next step is to actually use the parameters.

TravelTimeCalculator: Understanding the Settings Activity

In the last blog post, I added some default preferences to my app. Now, I'll try to change the code in order to understand how it works.

The first step is to add logs to see what values are parsed when a selection is made:

The log output above shows what happens when navigating the preferences. When selecting preferences, the corresponding strings that are defined in the preference files are found and printed to the console.

The strings are defined in the res/valuses/strings.xml file.

The settings themselves are defined in the settings xml files. Listeners are defined in the SettingsActivity.java file.

The pref_headers file encapsulates the sub screens for the different settings.

Here, the settings and some of their properties are set.

 Finally the source code with listeners are defined as listeners in the SettingsActivity.java file.

The second step is to modify the existing settings to two settings screens: One for general settings and one for settings related to the travel modes. I'll add suitable controls depending on what kind of input I need from the user.

• General Settings:
• Mode: Distance, Duration, Cost
• Currency: A short text string, for example SEK or USD
• Cost per Emission: SEK per kg CO2e
• Time to cost ratio: SEK per hour
• Travel Mode Settings:
• Bicycling Start/Stop Time: seconds
• Driving Start/Stop Time: seconds
• Driving Cost per kilometer: SEK per km
• Driving Emissions per kilometer: kg CO2e per km
• Transit Ticket Cost: SEK
• Transit Emissions per km; kg CO2e per km

I'll implement those settings in the next blog post.

TravelTimeCalculator: Adding a Preference Activity

My app can now query Google Directions for the best travel mode with respect to duration and present that on a map, when the user is tapping on it.

I want to let the user define what the best travel mode is, by selecting some parameters.

There will be three options for estimating the best travel mode:

• Duration
• Distance
• Cost

Cost will consider the following parameters:

• Travel duration and distance
• Start and stop time (unlocking and locking the bike, finding a parking space for the car)
• Travel costs per km (fuel cost and vehicle depreciation)
• Travel costs per travel (transit ticket price)
• Emissions (CO2e per km)
• Time cost (SEK/s)
• Environmental cost (SEK/CO2e

The app will calculate the cost for each travel mode:

For transit and driving, I'll use the estimates from klimatsmart semester (climate-smart vacation) to estimate the emissions per person-kilometer.

Adding a Settings Activity to the App

Users will need to change the settings above. I started by adding an activity to the project:

Specify the parent class

A couple of new files are created automatically, with the necessary code to make the new activity appear. Now, I need to understand the anatomy of the new files.

The preferences are defined in xml files in the java/res/xml folder

Currently, the code is autogenerated. The settings are divided into three submenus.

Preferences for ringtones are located in a submenu:

A listener is declared in the SettingsActivity class

The settings are designed in the XML file and accessed from the SettingsActivity class.

In the next blog post, I will explore the settings and its listeners further.

TravelTimeCalculator: Adding Multiple Markers for Directions

TravelTimeCalculator shall illustrate the best travel mode for different coordinates, related to the target of the map. I'll add the markers to an array of markers, either when the user taps the map or when the user moves the map to a new location.

Adding Markers After Tap

I store the markers in an array of markers.

Step 1: Adding markers dynamically when user taps on map
I added two lines of code to the onMapClicked method:

Step 2: Changing the added markers
CheckDirectionsFromDb will check if the direction has data for all transport modes. In that case, it will call updateMarker for that coordinate:

Since the method doesn't care whether the center of the map equals orig or dest,
I'll let the method updateMarker do that check,

To find the marker to change, the app searches all existing markers and compares with the tapped location:

First, the script determines which coordinate is connected to the marker.
After that, it changes the connected icon. Note the icons to the right.

The result is markers added to the map.

For some strange reason, the title is hidden when another marker is added.

The next blog post will discuss how the program shall add when moving the map. The markers should be cleared and existing directions shall be added, if found in the database.

TravelTimeCalculator: Adding Markers With Custom Image

The last step (for now) is to add markers that represents the quickest/shortest transport modes between the center of the map and some other coordinates.

I'll start with adding a marker at the center of the map (target). I'll round the coordinates to the closest three-decimal number (36.2123321->36.212). The marker will be updated every time the map moves.

There will also be an arbitrary number of markers representing the existing directions from the current origin coordinate. Those markers will appear and disappear, depending on the current location.

A snippet will show the best distance/duration and an icon will show the best mode of transport

Single Marker

It's easier to implement the single marker, so I'll start with that one for now.

The first step is to simply add the marker at the center of the map. After changing the icon of the marker is now changed in two steps:

Step 1: Add an icon to the resource file

Step2: Let the marker object point to the icon resource 

To make the marker update when the user moves the map, I made a small change in the onCameraMove listener in the map object.

The code rounds the coordinates to the closest 3rd decimal. 

The purpose of this is to reduce the number of possible coordinates. Otherwise, I'd have zillions of coordinates in the database, many of them only feet apart.

TravelTimeCalculator: Tying it All Together

TravelTimeHandler is now working in the sense that it collects directions from Google Directions (or casched from the internal SQLite database) and select the quickest option.

Implementation
I started with skeleton code that indicated what it was supposed to do using printouts to the logs.

The stubbed method. I'll replace the
console print statements with actual
code later.

After having the overall structure in place, I implemented the needed methods:

• scanDirectionsForTap is activated when the user taps the map. It starts with a query to the database for any directions between the center of the map and the tapped location. If no results are found, it queries Google Directions for bicycling, driving, transit and walking.
• getShortestDirectionFromDb queries the database for the shortest duration (or distance) between two points. It returns the value and an integer representing the mode.
• saveDirectionsToDb saves the directions to the database
• clearMarkers will clear the markers from the map (stub). I'll call this method when the map is moved.
• addMarkers will add a marker to the map (stub)
• getDirectionsFromUrl queries Google Directions for distance and duration for a couple of coordinates and a mode of transport. The response is handled in the over-ridden method onResponse. onResponse adds the result to the database. After that, it will query the database for the shortest distance/duration using getShortestDirectionFromDb.
• readDirectionsFromJson extracts the relevant data from the JSON response from Google Directions.

Since the database handles only integers for the coordinates, I'll multiply the coordinates by 1000 and round them to integers when querying the database. 

The output from some queries is:

Initially, the app had no direction data for the coordinates.
After the query, the database was populated with the data.

Context 

One issue that I saw was applicationContext. When having code that uses context in another file than the mainActivity.java file, the app can't find it. The solution was to get the app context explicitly:

Remaining Work
I want to visualize the results with markers. I'll add one marker in the middle that represents the current location.

When the user taps on the map, I'll add a corresponding marker on the map. The marker will have a color or symbol that indicates the quickest transport mode (driving, transit, walking or bicycling).

When the user moves the map, the existing markers are cleared. The app should look for existing directions in the database and add corresponding markers automatically.

TravelTimeCalculator: Selecting Coordinates to Scan for Directions

I'll simplify the selection of destination coordinate to scan for. Instead of having the app selecting some coordinates around the center, I'll let the user select the coordinates by tapping on the map.

The work flow will be like:

• User re-zooms and pans the map to a new origin:
• Clear existing markers (clearMarkers)
• The app searches for any existing directions starting or ending at the coordinates (getDirectionsFromDb)
• If found, it adds corresponding markers (addMarkers).
• If the user taps the map,
• If coordinates are not found in database, (getDirectionsFromDb is null)
• Send Google Directions requests, (getDirectionsFromUrl)
• Save response to database (saveDirectionsToDb)
• Add marker to map
• If coordinates are found, add marker to map

Step 1: Add a OnMapClickListener
I followed the instructions regarding detecting a tap on map from the official documentation. As soon as the map was installed, it crashed. The crash was my mistake, since I added the code in "onCreate". After adding the code to "onMapReady", it worked.

Step 2: Create a new class, TravelTimeHandler that handles the requests and decides whether to send a Directions query or query the database. It shall also add data to the database.

I moved the methods getDirections and readDirections to that class.

Step 3: Define what methods new methods to use.

• readDirectionsFromJson
• getDirectionsFromUrl
• getDirectionsFromDb
• saveDirectionsToDb
• clearMarkers
• drawMarker

Step 4: Implement those methods.
I'll cover this step in the next blog post

2018 in Retrospect

The last year, I had quite some progress on my projects. A normal week, I spend two hours on coding, two hours on learning and two hours on blogging.

TrafficControl (C++/Qt):

• I added locks for tracks in order to avoid trains to travel in opposite directions on the same track. I also added red and green circles to visualize the lock status of the tracks.
• Further, I added an icon to the executable 
• The Python script that parses the KML map to a network in TrafficControl was modified: Now, it ignores hidden tracks and the tracks/stations doesn't need to be in any specific folder.
• Improving the model for acceleration of trains
• User Interface Improvements: I added system load and simulated time to the user interface.
• Adding buttons for the controls 
• Adding menus and toolbars
• Adding a Help menu that points to some Help sections on this blog.
• NetworkDesigner is a new tool that helps the user design a network

StockReader (Was: C, now: Python):

• I re-implemented the web scraper in Python. Now, it has only 24 lines of code, including a check of the robots.txt policy.

TravelTimeCalculator (Android/Java):

• Setting up and learning Android Studio
• Sending a Google Direction API request
• Work with databases in SQLite

HouseReader (Python)

• Google Directions Request and JSON responses
• API keys

and some AI book reviews and one movie review

I've also started another blog that will discuss society-related issues.

In 2019, I'll make TravelTimeCalculator demo-able. After that, I'll continue with TrafficControl, adding passenger handling.

TravelTimeCalculator: Further Improvements to the Database (TDD)

Test: Check if dbHelper can add information to an existing entry (continued)

Currently, it is only possible to add information to the database if there is no entry at all for the coordinates.

If I add a direction for the mode "WALKING" to an empty SQLite database using INSERT, it will look like this:

There is data only for the mode "WALKING" in this entry. The other ones are "null".

If I'd like to fill in information about the mode "DRIVING" using INSERT for the same coordinate, the constraint that I used to define the database will prevent me from doing that.

Instead, I need to use the SQLite "UPDATE" command when there is already entries for that direction. 

Using the method checkIfCoordinateExist, to decide whether to insert or update entries, with or without reversed coordinates, I was able to add directions in reversed diretions.

With those changes, I was able to pass the new TDD test case.

Test: Get Duration and Distance for All Modes for Coordinates

I made similar changes in the code to be able to get reversed directions for all modes.

Finally, I added a method for printing the contents of the database to the console.

Next step

Soon, I'll be able to continue working with the app itself, instead of focusing on the database. 

First, I need to define which coordinates the app should scan for, given the center and zoom of the current map view.

TravelTimeCalculator: Improving Database

After being able to add a simple entry to the database, I want to add more modes of transport to the same origin and destination pair.

The first thing I saw was that the database isn't cleared automatically between the two test cases. I needed to add a method for tearing down the database between the test cases.

The setup method doesn't do anything for now.
The tearDown method clears the database.
If needed, I'll add more code later.

Test: Testing reversed directions
In order to reduce the number of Google Directions Queries that are sent, I'll assume that the transport durations and distances are similar in both directions. For example, I assume that the duration/distance between Trafalgar Square and Picadilly Circus in London is the same in both directions. When checking on Google Maps, it is not exactly the same, but close enough.

The test case looks like this:

The first test checks

In dbHelper, a query is first done in the intended direction (Trafalgar Square to Picadilly Circus) . If no entries are found, a query is done in the other direction (Picadilly Circus to Trafalgar Square). If nothing is found after that, 0 is returned.

Test: Check if dbHelper can add information to an existing entry.

Currently, dbHelper can add one entry for one mode. When trying to add a second entry for the same coordinates but with different mode, the database rejects that query since there is already an entry.

The second entry is currently rejected, since there is already one row in the database. 

The dbHelper needs to check if the entry exists. If it does, it shall run an UPDATE command, otherwise it shall INSERT into the database. I'll describe this in the next blog post.

TravelTimeCalculator: API Issues, TDD and Checking Records In Database

After making the app target API level 28, I got a strange crash. I found that I need to specify the requirement for Apache HTTP Legacy library, and after adding that dependency, the app was running as before.

The next issue that I saw was that my database method dbHelper:onCreate wasn't called as intended. dbHelper is inherited from SQLiteOpenHelper.

That behaviour was actually correct. The onCreate method is only called when Android needs the database.

Many of the current constants in dbHelper are hardcoded. I need to replace them with strings that I can change in the top of the file.

First TDD Test
The first TDD test checks that I can add a simple entry to the database (containing only one direction) and that I can read it out from the database.

The test intially failed since I've stubbed the addEntry method. After fixing that, I got an exception when querying the database.

That issue was caused by a typo. I missed to add a white space in the SQLite query.

Once that was fixed, I was able to pass the first Test Driven Development test case. I've updated the Github repo.

The next TDD test cases will cover:

• How to add data to an existing entry
• How to handle queries for non-existing entries
• Ensuring that the database won't add entries in the opposite direction. If "From A, B to C, D" exists, "From C, D to A, B" shouldn't be added.