Extracting Gameplay Data from Videos

Topics: computer vision, machine learning, games analytics
Supervision: Günter Wallner
Contact: guenter.wallner@jku.at
Type: BSc Practicum, MSc Practicum, MSc Thesis

Games Analytics relies on large behavorial datasets gathered in-game during play. In many cases, data of commercial games cannot be publicly accessed which does hinder research with respect to such games. A workaround is to develop solutions such as neural network-based methods to extract gameplay data directly from live footage or replay videos. Therefore, the goal of this project is to develop a solution to extract relevant information from the HUD of Valorant on the fly for real-time use as well as for automatically creating a dataset of match data to be used for subsequent analysis. An important aspect here is to ensure that the extracted data is error free and reliable. 

Screenshot from Valorant

Game Event Summarization during Streaming

Topics: games analytics, streaming, text analytics
Supervision: Günter Wallner
Contact: guenter.wallner@jku.at
Type: BSc Practicum, MSc Practicum, MSc Thesis

Streaming services such as Twitch have become a major attraction, gathering millions of viewers worldwide. To automatically summarize matches, automated extraction of interesting events is necessary. One possibility is to derive these events from spectator opinions by analyzing the chat. In this project, a system to extract events from the twitch chat for Fortnite should be developed. A time-line based visualization of the extracted events should be included.

Screenshot from Fortnite, showing landscape

AI-based Playtesting

Topics: games analytics, playtesting, information visualization
Supervision: Günter Wallner
Contact: guenter.wallner@jku.at
Type: BSc Practicum, MSc Practicum, MSc Thesis

Playtesting of games can be a time and cost-consuming effort. As games become larger, exhaustive testing of games becomes increasingly difficult. AI-based approaches can help to speed-up the playtesting of games. As such, this project aims to develop AI-bots that assist in the automated playtesting of games. Playtesting can be performed with respect to a variety of questions (e.g., navigation, difficulty, finding glitches, etc.) which makes it reasonable to develop dedicated playtesting bots for different tasks accompanied by a visualization to facilitate analysis of the results. As part of this larger area, two sub-projects are possible to be defined based on individual interest.

Concept Image of AI ©Photo by Tara Winstead from Pexels

Fast Motion-aware MRI Reconstruction

Topics: medical imaging, machine learning, magnetic resonance imaging
Supervision: Erich Kobler
Contacterich.kobler@jku.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

In the last decades, MR imaging has evolved into a routine imaging technique in clinical practice.
The underlying temporal constraint of the acquisition process limits the broad accessibility and drives the operational costs, though.
Therefore, numerous techniques (parallel imaging, compressed sensing, ...) have been develop to accelerate up the scanning time while maintaining reconstruction quality.
Recently, these approaches have been successfully combined with deep learning in various ways.

However, the majority of these algorithms are not able to cope with patient motion, resulting in sever artifacts that typically spoil the diagnostic value of MRI scans.
Hence, this project/thesis aims at developing an efficient reconstruction method that jointly estimates the motion along with the reconstruction image/volume.

mri_recon ©fastmri.org, https://fastmri.org/

Brain Tumor Segmentation in mpMRI Scans

Topics: medical imaging, machine learning, semantic segmentation
Supervision: Erich Kobler
Contacterich.kobler@jku.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

Gliomas are the most common primary brain cancer types, with different degrees of aggressiveness, variable prognosis and various heterogeneous histologic sub-regions.
Therefore, these cancer types are hard to segment and classify.
The aim of this project is to combine efficient discrete optimization algorithms (belief propagation, ...) with deep learning methods to semantically segment 2d and/or 3d MRI scans into enhancing tumor (ET), tumor core (TC), and whole tumor (WT) regions.

mri_seg ©Menze et al. 2015, https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6975210

Sampling to estimate Uncertainty in Optical Flow

Topics: computer vision, machine learning, uncertainty
Supervision: Erich Kobler
Contacterich.kobler@jku.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

Optical flow describes the apparent motion of pixels between a pair of images and is therefore an essential building block in numerous computer vision systems (video classification, video editing, temporal super-resolution, autonomous driving/flying, ...).
Currently, the most successful optical flow approaches use neural networks to extract and correlate local information to predict a spatially refined point-estimate of the displacement vector for each pixel.

However, the uncertainties of these predictions are hardly estimated, which is of course of high interest in safety critical scenarios such as autonomous driving.
The starting point of this project is to inspect different uncertainty estimation methods facilitated in machine learning (dropout inference, score-based generative modeling, energy based models, ...) for their applicability to optical flow estimation.
Then, these findings are condensed to develop a model and learning strategy suitable for optical flow estimation that yields results comparable to state-of-the-art while enabling an estimation of uncertainty.

Flow ©Hofinger et al. 2020, https://link.springer.com/chapter/10.1007/978-3-030-58604-1_46

Airborne Optical Sectioning

Topics: drones, image processing, robotics, object detection and classification, user front-end
Supervision: Indrajit KurmiRakesh Nathan, Oliver Bimber
Contactoliver.bimber@jku.at
Type:  BSc Practicum, MSc Practicum, BSc Thesis, MSc Thesis (multiple sub-projects available, see below)

Airborne Optical Sectioning (AOS) is a wide synthetic-aperture imaging technique that employs manned or unmanned aircraft, to sample images within large (synthetic aperture) areas from above occluded volumes, such as forests. Based on the poses of the aircraft during capturing, these images are computationally combined to integral images by light-field technology. These integral images suppress strong occlusion and reveal targets that remain hidden in single recordings.

Many researchers and students have contributed to JKU’s innovative AOS framework to explore novel solutions as search and rescue, wildlife observation, and wildfire detection. Several high-impact publications in Nature and Science have covered our scientific findings. See https://github.com/JKU-ICG/AOS/ for details.

In the course of AOS research we offer the following sub-projects (details on request):

  1. Implementation of an AOS front-end system for flightpath visualization and -planning (based on Google Earth).
  2. Investigation of novel adaptive AOS sampling techniques with drone swarms (based on our AOS simulator: https://aos.tensorware.app/)
  3. Investigation of parallel-sequential AOS sampling strategies (based on our AOS simulator: https://aos.tensorware.app/)
  4. Implement AOS on NVIDIA Jetson (https://developer.nvidia.com/embedded-computing)

 

We would like to point out that when playing the video, data may be transmitted to external parties. Learn more by reading our data privacy policy
Data protection information

Understanding Analytical Strategies using an Image-based Approach

Topics: image-based, machine learning, visualization
Supervision: Conny Walchshofer, Marc Streit
Contact:  conny.walchshofer@jku.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

Users of interactive visual analytics tools naturally come up with analysis strategies . Deriving those strategie is a challenging research topic. Current approaches primarily log interactions, such as click events with the system, to replicate the analysis workflow. However, visual analytics tools often do not make the raw data easily accessible.

Therefore, this project/thesis first aims to reverse-engineer screenshots of the visualizations to identify text elements such as chart title, x-axis, or y-axis of a visualization by using text localization, recognition, and classification (similar to this paper). To further understand the analytical strategies, a recurrent neural network should be developed to identify the changes between each interaction based on images.

Understanding Analytical Strategies using an Image-based Approach

Pinocchio: Uncovering False Positives from Multiple Testing

Topics: statistical tests, comparison, visualization
Supervision: Patrick Adelberger, Marc Streit
Contact:  patrick.adelberger@jku.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

Statistical tests are basic measures to infer information from data. Applying multiple statistical tests simultaneously has the risk of obtaining false positive results, as in the images below. Although this risk is being controlled by the level of significance, the probability to get a false positive result increases with the number of tests.

The goal of this project/thesis is to make users aware of this problem and look into methods to correct for multiple testing. As a first step, a website that makes automated comparisons and highlights the problems can be created. Following this, however, integration into tools for biomedical data analysis is also possible.

 

Spurious Correlations ©http://tylervigen.com/spurious-correlations
Significant ©https://xkcd.com/882/

Exploration of Semantic Image Segmentation Network Explanations

Topics: interpretable machine learning, semantic image segmentation, visualization
Supervision: Christina Humer, Marc Streit
Contact:  christina.humer@jku.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

There are recent advances in generalizing explainability methods of neural networks for classification tasks to the task of semantic image segmentation networks [Vinogradova et al]. To understand the inner workings of segmentation networks, static explanations alone do not provide sufficient information. 

The goal of this project/thesis is to implement a tool to facilitate the exploration of explanations for semantic image segmentation networks. The tool should allow users to upload an input image of interest, apply a semantic image segmentation model, and then interactively show which areas of the input image were important for a certain area of the segmentation mask.

SEG-GRAD-CAM for all pixels and class Sky © Vinogradova et al: https://arxiv.org/abs/2002.11434

Dashboard Onboarding

Topics: graph traversal, user interface, visualization
Supervision: Vaishali Dhanoa, Marc Streit
Contact:  vaishali.dhanoa@pro2future.at
Type:  BSc Practicum, BSc Thesis, MSc Practicum, MSc Thesis

The use of dashboards containing multiple interlinked visualizations is quite prevalent these days. However, they often lack a clearly defined way of onboarding a user about their intent and use. Standard procedure often involves a textual description or a human narration.

In this project, we aim at implementing an onboarding strategy as a custom application which enlists the dashboard components as a graph and applies a traversal strategy to onboard the user. The custom application needs to be implemented as a client-side application which also takes user input(s) into account and adapts the graph traversal accordingly.

Dashboard Onboarding

Churn Prediction for Rail Nation

Topics: games user research, visualization, analytics
Supervision: Günter Wallner
Contact: guenter.wallner@jku.at
Type: MSc Thesis

Game development is increasingly relying on data analytics. Among the many use cases, predicting and modelling player churn – i.e. predicting players that will leave or stop playing at some point – based on tracked in-game data has become of key interest. This is especially true for new business models that rely on the continued engagement of players such as subscription-based services or free-to-play games where monetization is, e.g., happening through micro-transactions. Customer churn prediction can thus be pivotal for the success of games.

The goal of this thesis is to develop a churn predication model for the round-based, free-to-play, online strategy game Rail Nation from Travian Games. In the game, players build up a railway company and manage transports to increase their prestige and wealth in order to achieve victory over the competitors. A game round is scheduled to last between 3 – 4 months. It consists of six consecutive railway eras followed by an endgame, in which all players compete with their grown railway companies against each other in the ultimate battle.

The thesis will be conducted in close collaboration with Travian Games who will also provide access to the game and in-game data. Travian Games is located in Munich and focused on developing and marketing PC and browser games.

Tasks

Your job is to develop a churn prediction model for the Classic Mode of Rail Nation based on tracked in-game behavioral data (possibly in connection with survey data). Due to the large number of available metrics this will likely include selecting an appropriate subset of the metrics. The churn prediction model should be useable for real-time monitoring and be complemented with a browser-based visual dashboard to visualize data from the churn model. The model should be implemented in Python and/or R.

Requirements

  • Good knowledge of predictive analytics and unsupervised learning
  • Good knowledge of Python or similar
  • Knowledge in data visualization is of advantage
  • Willingness to crack problems and to show self-initiative
  • Interest in games
  • Knowledge of English language (source code comments/documentation and final report should be in English)

Real-time Training Visualization for League of Legends

Topics: games user research, visualization, analytics
Supervision: Günter Wallner
Contact: guenter.wallner@jku.at
Type: MSc Thesis

Training visualizations for players are getting increasingly important in the competitive gaming scene to learn and improve in-game skills. League of Legends (LoL) has emerged as a popular but at the same time difficult to learn game. The goal of this topic is to develop and evaluate solutions (e.g., visualizations, tools) that help players learn and reflect upon in-game activity in real-time by utilizing the Live Client API offered by LoL. The topic can be approached from different perspectives and interested students are encouraged to propose their own ideas in relation to it. Other games than LoL could be focused on as well.

Tasks

The overall goal is to develop visual solutions that can be used in a live setting for training purposes. Overall the steps include 1) requirement analysis with players, 2) develop a solution, and 3) evaluate the solution with players.

Requirements

  • Knowledge analytics or related domains
  • Knowledge of information visualization
  • Good programming skills
  • Interest in exploration of novel solutions
  • Knowledge of League of Legends is of advantage
  • Knowledge of English language (source code comments/documentation should be in English)

Encounter Visualization for DOTA 2

Topics: games user research, visualization, analytics
Supervision: Günter Wallner
Contact: guenter.wallner@jku.at
Type: MSc Thesis

Dota2, a multiplayer online battle arena game, is enjoying wide popularity in the competitive gaming scene. The goal of this thesis is to develop a visualization based on the concept of battle maps to convey information of encounters/events happening within Dota2 matches. Battle maps have long been used by historians to concisely convey the course of a battle. This includes 1) detecting interesting events and encounters and 2) condensing movement patterns and contextual information into battle maps. The work can draw upon previous work on encounter detection and battle maps. This topic is in collaboration with …

Tasks

Work will include developing or including existing encounter detection algorithms to extract events of interest and develop a visualization front-end which conveys information surrounding these events. This can initially happen post-play but extension to real-time detection and visualization should be accounted for. Developed solutions need to be evaluated with the target audience to assess its usefulness.

Requirements

  • Background in analytics or related domains
  • Knowledge of information visualization
  • Good programming skills
  • Willingness to crack problems and to show self-initiative
  • Knowledge of Dota2 is of advantage
  • Knowledge of English language (source code comments/documentation should be in English)

Presentation of Application Examples for ProjectionPathExplorer

Topics: web design, visualization
Supervision: Andreas Hinterreiter, Marc Streit
Contact:  andreas.hinterreiter@jku.at, marc.streit@jku.at
Type:  BSc/MSc Seminar

In a recent publication, we introduced ProjectionPathExplorer [1], an interactive visualization of high-dimensional processes as trajectories in a low-dimensional embedding space. In the paper, we describe how this type of visualization can be applied to a wide variety of domains. So far, the online prototype contains examples from Rubik’s cube, chess, interaction data, neural networks and Go. We have recently collected a number of additional application scenarios from student projects in the Explainable AI course. We would like to make all the different examples available and easily accessible as part of the ProjectionPathExplorer paper website. This task includes consolidating the different datasets and making them individually accessible via dedicated links, as well as potentially adapting the prototype with domain-specific summary visualizations for those applications that do not yet include them. The final result should be a neatly arranged collection of links and short descriptions that showcase the generalizability of the tool and can be used for publicity and presentation purposes.

  1. Hinterreiter, Andreas, Christian Steinparz, Moritz Schöfl, Holger Stitz, and Marc Streit. 2021. “ProjectionPathExplorer: Exploring Visual Patterns in Projected Decision-Making Paths.” ACM Transactions on Interactive Intelligent Systems. Special Issue on "Interactive Visual Analytics for Making Explainable and Accountable Decisions" (To be published): 1-30. https://arxiv.org/abs/2001.08372 . 

Automatic detection and visualization of patterns in projected paths

Topics: visualization, dimensionality reduction, (trajectory) patterns
Supervision: Andreas Hinterreiter, Marc Streit
Contact:  andreas.hinterreiter@jku.at, marc.streit@jku.at
Type:  BSc Thesis, MSc Thesis, BSc Practicum, MSc Practicum

Many real-world problems can be viewed as sequences of high-dimensional states. As an example, each color configuration of Rubik’s cube can be encoded in a vector, and solving the cube can be viewed as transitioning from one high-dimensional vector to the next. In order to better understand patterns of these paths through high-dimensional state spaces, the states can be projected to 2D using non-linear dimensionality reduction techniques such as t-SNE or UMAP. The paths can then be visualized as trajectories in the embedded space.

We developed an interactive visualization prototype to explore such groups of projected paths (see image and link above). We already obtained some interesting findings for different application areas, but we would like to further  expand our prototype in two directions:

  1. Automatic detection of high- and low-dimensional patterns of points and trajectories
    The visualization system should be able to autonomously detect interesting patterns for both points and paths. It should be able to determine whether these patterns in the low-dimensional states actually relate to the “true” high-dimensional data.
  2. Automatic and effective visualization of the detected patterns
    The visualization system should automatically highlight the detected patterns and visualize them in an effective way. This will likely include novel visual encodings for similarity of high-dimensional states. It is of particular interest how suitable similarity encodings compare between different application areas.

The work will combine traditional data analysis techniques (e.g., dimensionality reduction and clustering techniques), conceptual visualization research (novel similarity encodings), and implementation of the developed techniques in the form of a web application.

PathExplorer (https://develop--projection-path-explorer.netlify.com/)
Copyright Information