tool for analysing hero item (forge step)

hello,

the last days i tried something: i wanted to program a tool that analyses screenshots of items that are forged. a kind of OCR (optical character recognition). never did that before and i was curious if/how it could be done.

therefore i used the free software Octave (download and infos: https://www.gnu.org/software/octave/ ). it works well for me - it recognises owner, quality, perks and values, of screenshots ( designed for screenshots i get from an apple ipad air 2 of resolution 1536x2048 px and actual OR version 4.2.0 ) placed in a given folder, makes a reduced snapshot-image with identified values of the analysed regions and also an excel-worksheet. here some screenshots from a result. 

IMG_5721_mask.PNG



image.png

 

here a screenshot of the short description inside the main file “run_OR_itemForgeOCR.m”  and what it looks like when/after running the tool (=this file):


 

i dont want to keep this on my own, so i added all files that are needed to work in the attached zip file. (see actual version below) 

you need to:

  • install octave
  • unzip the files in a folder of your choice (e.g. you get “<your_path>\OR_itemForgeOCR” with the listened files and 2 subfolders (see above) inside
  • run octave and change to that folder 
  • type “run_OR_itemForgeOCR” into the command window
  • if the tool works with the given screenshot (“IMG_5721_mask.PNG” inside subfolder “screenshots”) you can try to analyse your own. but remember that they have to be of the described resolution and i do not give any warranty that it will work or the values are detected correctly.

 

i hope it works for you guys.

actual version: v1.1OR_itemForgeOCR_v1_1.zip

 

EDIT: 2019/03/03 … v1.0: even if i tested it with 50 screens i did some more now and (ofc) some troubles have occoured: e.g. problem detecting the lower line, the area of identifing the right perks was too small from the left (so  problems detecting an eight on those occur), and i forgot the movement speed perk… (omg).
tested on another 153 screenshots, made some changes, more stable now on detection. -> removed v1.0 and released v 1.1: 

% &nbsp;v1.1 ... &nbsp;seek for tag "v1.1" &nbsp; % &nbsp; &nbsp; &nbsp; ... 2019/03/03 ... improoved L1-detection and bigger L1 height (maybe just fix them...) % &nbsp; &nbsp; &nbsp; ... 2019/03/03 ... problem of detecting first 8 ud rigth perks -\> area to small!! -\> increased % &nbsp; &nbsp; &nbsp; ... 2019/03/03 ... forgot speed perk, added % &nbsp; &nbsp; &nbsp; ... 2019/03/03 ... added more damping and threshold for NoS detection % &nbsp; &nbsp; &nbsp; ... 2019/03/03 ... &nbsp; my\_simple\_OCR: added overall min threshold for OCR analysis (prevent "ghost 1") and shift area to be analysed 1 to left (want a 0 column 1st) and reduced width 20-\>19; reduced dot shading area 1px added possibility of show selections % &nbsp; &nbsp; &nbsp; ... 2019/03/03 ... remember: unicates do not have stars, -\> they produce error

 

 

Hey @NaN, that’s a nice tool… I might get into that. My company uses OCR for document’s management, and I know how it works actually, it’s pretty awesome, but most of it is based on text. Here is my important questions to you!

How can it recognize color (of item and of star)? And just to be sure, I assume it can generate a report with all your items, based on hero/slot/perk or whatever, correct ?

hey archimides, thx.

i dont know how commercial OCR programs work, i asume nowadays they use techniques also found in computer vision or deep learning algorithms.

what i use is a normalized cross correlation method (see e.g. https://www.mathworks.com/help/images/ref/normxcorr2.html ) for finding a pattern in an area of a greyscale picture. here i dont mind if its a full name, a perk or a digit (for detecting all digits i simply repeat it).

for color detection of an item i calculate the 3 mean RGB-values of the colored pictures (i marked them as white squares in the screenshot below) and compare those w.r.t. to “normal ones” by a defined threshold
image.png
image.png
image.png

 

color detection of a star is similar, but i need to know where a star is located (and we also need to know how many)…
here i take the areas of a greyscale image (the y-position is rather fixed) and calulate the max values of the center-line through all stars (the red line in the first row in the screenshot below) and also the [edit] mean min y-values (blue line). the difference of those two [edit]  minus mean value of the 10 most left and right values is filtered and compared to threshold. from this signal the amount of stars can be determined either by positive rising edges or by distance between the far left and right.
image.png image.png
color of star can be determind as before analysing a small rectange that lies inside the star and compare of what should be, or just by the maximum value of the center line:
bronce star is below 180, gold between 180 and 220, silver is >220.

That’s all.

actual i have analysed 1650 screenshots but i need to have a closer look at the results and also make a postproceccing validation (e.g. range of values or if a 4 star is produced from a 2 star, or if the color is detected properly (first two have to be the same, and e.g. green+green->blue, forging values can only increase (beside refining),…) some kind of apriori knowledge of how the forge works to detect those items where the automatic recognition might have failed. (i do already set star color to unknown, if NoS~=NoS2

e.g. those manually marked red lines

but this, or reports can then also be done e.g. in excel

Can you let me know of your progress, and how accurate your results are?

One more question, say if I make a screenshot with 6 items, can it read all 6 items? Or do I have to setup the reading boxes, and then I also have to make the same screenshot every time, so that the boxes are in the same place, to be able to OCR the data ?? Otherwise it’s 1 by 1 item, which is not helpful much.

Also, is the tool ready to use, or are you tweaking settings in backend?

Hi NaN,

Just wanted to say this is insane (in a good way)! Good work! :slight_smile:

i will share infos about that when i did some more checks. maybe next weekend.

in this version only screenshots taken before you hit the forge butten (the “dismatle”+“before forge”->“after forge” screens) can be analysed and it also does not work for uniques (they dont have stars). i made it for those bc that are the screens i take bc they contain the information about the forge boost.
but you can change in the init-file the location (if they are at a fixed and know place) of the info boxes
image.png
and give it a try, but i doubt if it will work if the size hase changed. maybe a rescale can help, but in general i guess this would need some work.

i used this version 1.1 as it is for analysing my 1650 screens without tweaking, but i have to look at the results and see at which screens / items the algorithm had problems and after that maybe i can do some further modifications. but sure you can give it a try, just remember:  it is designed for screenshots i get from an apple ipad air 2 of resolution 1536x2048 px and actual OR version 4.2.0 

hey archimides.

unfortunately only 1401 screenshots out of the 1650 have been transferred to an excel (i forget to set the flag to identify a screenshot again even a previous result exists… but its ok).

i did some automatic checks like out of range (only up to 200k allowed), perk values have to increase, owner check (before and after forge need to be the same), item color check (dismantle and before forge need to be the same), and so on (but did not check everything like color change etc.)

result: 47 out of the 1401 screenshots seem to have a problem s.t. at least one of all those information have not been identified correctly (did not pass all checks).

here a short graphics w.r.t. bronce forge cycle, selected “everything” on normal heros (not trinkets) and forged with 4 stars. displayed values are “before forge” vs. “after forge”. the second line for the perk is due to a color change of an item.

short: there are still (beside the obvious ones) some small errors that passed “out of range” check - i should do this w.r.t. the specific perk or w.r.t. its relative increase.
but: it looks all pretty linear through all possible perks and can maybe also extrapolated to higher values. (5star forge of titan items is excluded here)

 

@Jewelmania here is my conclusion to this on how to predict values when all 4* forged… for green 1* values i took max numbers i had of my green item collection at end of lvl 141. (i did not investigate 1* forging… but it should be easily adapted)
@Archimides does this also fit to your observations?

@NaN is it fixable? Do you need to increase the box range of retreiving data or is the picture quality a problem? We used to have this problem, and I told the guys to make the OCR in effect before it makes the image in worst quality. But not sure if capable to conduct the OCR before a screenshot.

its not a bad “start”. I’ll give it a try this week I think.

@Archimides i need to sort out those screens where i detected that something was not coreect and lokk loser what was the problem. maybe i can make improovements, i have some ideas (but no pressure to do so)

but in general it should work fine, just remember that the are plenty of informations per screenshot (you can put them all in the folder and it processes them (hopefully) all). per screenshot there are 3 items and for each item you have 4 general informations (Item color “IC”, forge cycle color “FC”, Number of Stars “NoS” and owner), min 2 and max. 4 symbols “perks” (depending on FC) and each didgit seperately to be identified. So in the example above we have 3x4=12 general infos, + 9 symbols and 34 digits… so it works pretty good but i think it can be done better and more reliable. i let you know when i have found some time and also improvements.

But the informations i hoped to get are that the mechanism works for all the same (we get lines regardless of perk, item color or player level). so your observation that with each level upgrade you get 1 forgestep is valid from my point of view too. this means: if you max forge you should be able to predict item stats several levels ahead (as you already do) - you should have a multiplier of 1.0655 (at least for bronce forge cycle, did not check refinings) per level. e.g. if you can make a 100k item at a certain level you can make a 100k*1.0655^5=100k*1.373=137.3k item after you climbed 5 levels only due to the normal forgeing steps. but remember: you also have better improvements due to color changes that get better the higher the values are, so it will be even more. and important: you get a better boost on items than you have on base resistances - so when you loose approx. 5% on quality per level on most perks, you can achieve items with >6.55% better. not much per level (and you need to find good items again also), but it summarizes… 

 

edit: i dont think it is a problem due to screenshot quality: it is designed for pictures from ipad air 2 with 1536x2048 px resolution (~5MB each)…
[comments:  i do know it sometimes has troubles with the “dot” or finding a “1” inside a “4” with a better correlation or a 2 instead of a 7 (as a result from the action of ignoring a “dot” (which leads to a double identification of a digit)… but i have an idea in my back head which should work better… sometimes also the shining effect (those rotating rays) may cause troubles detecting the correct number of stars.]