Manual

Unofficial manual version 2.0

September 2018

If you have no idea about this work, please start from the paper

https://arxiv.org/abs/1803.00788

All the code are stored in: C:\Users\CNNplace\Documents\MATLAB\Code

All 227x227 images are stored in:

Small tip: if you do not have any idea where I put the said functions or scripts, use the

command edit functionOrScriptname or highlight the name in the file → right click → select

‘open *filename*’, and make sure you add all the path above before search it, if not, it will ask

you whether you want to create a new file of not.

•••••

Where to start

The basic processes are: read OSM, get the tag of each object, get road points, find intersection

of road and generate connectivity table, for each point, interpret the OSM information, and then

download the corresponding images from GSV.

After that the connectivity table will be used for generating the DB.

● Load OSM

To get .osm file, go to https://www.openstreetmap.org/ and select the region you want → press

‘Export’ → ‘Overpass API’. Then rename the loaded file to .osm

MatLab will crash.

Note: Make sure you check the appearance of the OSM map before download. Sometimes

you’d see the blank city with only road; that means there is no information about the building at

all, so you cannot use this for ‘gaps between buildings detection’ process as there is no building.

Also, you can check the tag and label I used from OSMtagSetupInit. Be careful if you would like

to try to load OSM of non-english usage country, there might be a problem caused by the label.

I had tried Japan before and found out that my code could not work due to the face that the

OSM map contains Japanese label and my function cannot read it.

find where I missed. The effect of this bug is sometime the connection looks funny, they jump to

the next road. This probably might cause by the raw OSM.

● Load Google Streetview

The concept of Google streetview is you send the <lat,lon> → get Panoramic ID of that

panorama and then select where to crop out.

The process from 1-3 are in Streetview\src\streetImagery\checkUsageLL. If you’d like to get

separate folder for saving the large images (1240x480) and cropped images (227x227).

However, for the future test, do the train by capture in 227x227 directly. No need to keep in two

separate location. To change this, just get the input for sizeIm2 = [227 227] and remove the

cropped part & filepathS variable from the function.

Note that sometimes by running the GSV scripts/functions in the different time, you might get

the different number of GSV-uniqued road. If there is a crash like the data cannot be download,

maybe you should stop and wait for another day. Sometimes there will be a weird crash like this

using Google API.

There is also the Bing version, check realZhu\BING\osmretrieveBBG4_NEWSMALL to see

how I download images. The basic process of Bing is similar to Google. Also, there are some

other useful functions that you can use all in Streetview folder. This is the legacy code, so there

are things that I have not discovered yet.

latitude and longitude are highlighted in red.

You can retrieve the XML information by copy and paste this to the web browser:

http://maps.google.com/cbk?output=xml&ll=51.4568999,-2.6021452

If you want to get the panoramic image, use:

I = downloadPanoImage(panoID, zoomfactor);

%default = 3

For example, from osmelem parameter, you can call:

● Generate DB

For DB, I would suggest you to try the normal construction first. Be careful when you use dense

version, this one is not optimised, it could take days to generate the whole dataset. I encourage

you to look for more practical solution.

In my code I set the sub-section = 4; you can adjust them as you like, or make it more flexible

(for example, the auto-generate function/script that do the auto loop instead of manual like this).

If you change the number, make sure to go and modify the search process.

Note that the generated routes are not allow to perform revert direction (for this version). You

can (and should, in the future) add them by turning the direction of the descriptor. Please check

genFullBoxNew3 and genFullBoxDesc functions.

Also, please be careful about the format of the descriptor (binary/decimal). If you put it wrong,

Raw format: 1 1 2 3; when 2 = 0 and 3 = unknown (sometimes the info on the map can be

missing.)

Binary format: 1 1 0 0; I replace 2&3 as 0

Decimal format: 12

For the important parameters: rconnTable

maximum number of locations within the route you need (try 40), descList

descriptor for each location; in my database: descList_n2

descList_n3

is the number of

locations those are out of range (says, some location might not appear on the map, and it

Go and check OSMFirstprocess for more information about how each parameter has been

genereated.

2. Train the dataset

Each classifier trained separately. The data will be arranged into table format. For more details

of using CNN in Matlab: https://www.mathworks.com/help/nnet/ref/trainnetwork.html

the ggSet. For ggSet, you generated them in the previous step (while extracting the OSM).

If you want to use my image data, go and check

C:\Users\CNNplace\Documents\MATLAB\SceneImage\OSM_MAPtest folder, there are tons

of it; their corresponding ggSet has been stored in the

C:\Users\CNNplace\Documents\MATLAB\DataBackup folder and look for _allGG file (ex.

'C:\Users\CNNplace\Documents\MATLAB\DataBackup\20170413_BBLSM_allGG.mat'). It might

be a bit hard to find. If it is too confused, and you need to re-download/retrain it anyway, just do

the step 1 and make sure you properly save the ggSet parameter.

You can also rewrite everything, the training process is simple, you only need matlab table with

To test it, follow the code in testing section. Note that I am not sure whether this one is the

newest version or not. You can also write a simple function to check the results. The easiest

way is to turn the test set into the table, put the image into the classifier by:

I = imread(‘Your file path’);

%Optional I = imresize( imread(‘Your file path’), [227 227]);

res = predict(net,I);

The res parameter will return the score for each category, you can add ans_cate = find(res ==

max(res));

to get the answer. In my work, 1 = junction 2 = no junction, and the same as gaps.

To turn the answer to binary, convert 2 into 0.

Also, there is some ways you could obtain the backup for each round of the training. From my

setting, see the parameter ‘CheckpointPath’

, in the demo, I put them in 'D:\checkPoints'

. You

can change it to any directory you prefer. This is also loaded, you should go and clear them

when you don’t want them anymore.

load('C:\Users\CNNplace\Documents\MATLAB\DataBackup\20180330_NETonly')

3. Sampling + Test + Results

Note: andddd again, I would suggest you to be careful with the size of the test set; having too

large would also freeze the machine. The smart move is 25 or 50, and then you can run several

times to gain numbers.

SamplingAndSearch is the the naive version (run from 2-40 big BD)

SamplingAndSearch_DYNM is the dynamic version; it would start by searching on the full

While searching, if the program just closes itself (or crash/ get debug prompt) without any sign, I

advise you to check the balance of the descriptor and the location you load the DB file. The

common problem is the size of descriptors in the database is not the same as you query; this is

the crash of c program. Another possibility is the wrong length of threshold list (ex. You test for

50 keys but put 40 threshold list).

This part still opens for the optimisation process. I’m sure it could do faster.

Note that the turn process has been included after the search, you can find the code in there.

Please keep in mind that sometimes Google frontal angle might have an error (ex. → → ← →

→ ; one location gets reverted frontal angle). I have not remove this yet as it is not a major

problem at this state of work. However, in real world environment, this might affect your result

when applying the turn direction process (as this might cause a fake turn). I’d suggest you to

find a way fix this. Small tips: check the direction along a route to see the sudden changes and

magic number; I’d tried my best to reduce them.

The convertScores_frameCCGT is the ground-truth integration; this one will guess the current

location by using the information from the previous steps. The original is counting the

n-consecutive overlapped routes.Says, if they appear to be the best answer around the same

location for n times, we guess that’d be our current location.

Sorry that I cannot find the old version, it should be somewhere in my code. This one is the

modified version, so it might be slightly different operation, but you can do the modification

based on this; it is kind of straight-forward process. I left the test route for you. If you would like

if numHops > 1

%not the first round

%Remove the handle of the previous round

delete(AJ);

delete(SJ);

end

[AJ,SJ] =

displayBKCCC_anim_IN(mroadcoord2,convAns2,pathSet2,thresSet(keynum,numHops),numH

ops);

%Go to take a look, the basic operation is similar to my example code

temp=['Your file path\frame',num2str(keyframe),'.png'];

saveas(gca,temp);

close(gcf);

To calculate the size of the whole OSM area: [w,h] = getOSMboundM(osmelem.bounds);

To see the whole OSM map: displayOSMinit(osmelem)

•••••

Small tips

I’m sure that you would see a lot of ‘crash’ or ‘freeze’ in this document. It is because each

process is memory consuming; try not to open the other programs or run things in parallel to

reduce the memory usage.

more stable (but, yes, you need to rewrite everything).