Farkle AI Simulation

Unity, Simulation, Reinforcement Learning

Overview

An AI bot that plays Farkle dice game, trained using Maskable PPO by Stable Baselines3. The project is a simulated Farkle game with the player going up against the trained bot.

Table of Content

Environment

Training

Gameplay

Background

Solo Project

The AI is trained using a maskable proximal policy optimization reinforcement learning algorithm by stable baselines3. The training space uses Gymnasium's custom environment interface. The trained model is then exported as ONNX, and imported to Unity to play against players in a simulated game of Farkle.

Tools & Skills

Tools Used:

Unity
Visual Studio Editor (C#)
Git/Github
Visual Studio Code

Skills Applied:

C#
Python
Unity Engine
Gymnasium API
Stable Baselines3 - MaskablePPO
Unity Inference Engine

Goals

The goal of this project is to learn how to train and apply a reinforcement model in a real example and game.

Enviornment

Gymnasium doesn't provide a Farkle environment, so I had to create a custom environment that defines the game rulesets, and a way to incorporate action masking.

Observation Space

An observation space describes what the model can read and learn from.

Space

The observation space is a single array of 10 elements, containing only information the model needs to see.

Normalize

From prior testing and training, I decided to normalize each element, to ensure the features are balanced and can converge better.

Action Space

The action space describes what actions the model are allowed to make.

Farkle is played using a maximum of 6 dice, and each player can choose any number of dice for scoring, in any order.

Given 6 dice, there are 63 possible combinations of selecting the dice, given that each dice can only be picked once, and not all of the dice needs to be picked.

Why is action space 128?

In Farkle, players can choose either to bank the selected dice, or reroll the remaining dice.

If there are 64 (0-63, including selecting 0 dice) actions, I can define that the first 64 actions are selection of dice AND BANK

and another 64 (64-127) actions, I defined as selection of dice AND REROLL

How does the model know what dice it selected from picking a number between 0 to 127?

Each number can be represented as binaries.

For actions 0 - 63, I can represent all actions as a 6-bit binary.

0 0 0 0 0 1 = Only selecting first dice

0 0 1 0 1 0 = Selecting 2nd and 4th dice

1 1 1 1 1 1 = Selecting all 6 dice

So when a model chooses an action, it returns an integer from 0-127

All I need to do is translate that action into "Dice Selections" and "Bank or Reroll"

Action Mask

In most scenarios, there will be only a few options a bot can choose, since most combinations are not scorable. So I need to introduce an action mask to mask out disallowed actions.

Why are the dice SORTED?

Notice that the dice are sorted,

Imagine rolling a 'straight' with 6 dice,

1 2 3 4 5 6

This is the same as

6 5 4 3 2 1

To the model, each of these is a different permutation - a straight alone has "6! = 720" possible orderings. Without sorting, the model would have to learn each scenarios separately.

By sorting the dice and eliminating positional randomness, I reduced the number of checks needed during training.

Farkle Ruleset

The next step is implementing the actual ruleset for Farkle.

Calculating Score

The next step is implementing the actual ruleset for Farkle.

The scoring function is split into two parts

1st, Account for all special combinations with 6 dice

2nd, Account for every other possible scoring combination

Step and Reset Function

The last thing is to actually implement the Farkle game.

In Gymnasium's API, this is implemented in the form of Step and Reset function

Step Function: step function takes a step in the simulation, in this case, it is the bot rolling a dice, deciding what to select, and chooses bank or reroll.

Reset Function: reset function simply resets the environment to get ready for the next game.

See the code here: FarkleSimulation.py

Training

API / Library

Stable Baselines3 is an algorithm library built to be compatible with Gymnasium's environment.

Algorithm

I chose Proximal Policy Optimization algorithm - more specifically a MaskablePPO - because of Farkle's discrete action-space, where in any given game state, most of the actions are illegal.

Actual Training

Hyperparameters

Hyperparameters are parameters I can control that tells the model how it learns.

Learning Rate:

How big a step the model takes each time it updates and "learn"

N Steps:

How much of the gameplay it plays through before it stops and learn

Batch Size:

After gameplay and its results are logged, how much of it does it learn from at a time

N Epochs:

How many times the model studies the same batch of gameplay

Gamma:

How much the model cares about future rewards versus immediate rewards

Gae Lambda:

How far ahead the model looks when determining how much an action is considered good

Ent Coef:

Encourages the model to try different approaches instead of settling on one approach

Time Steps:

How many steps to take for the entire training

Default Run (~20 minutes)

First I ran the model with 1,000,000 time steps, and leaving everything else as default

Then using TensorBoard, I can retrieve the training log:

This is a graph that shows the average rewards during its training, showing that it peaked at around 400k - 700k time steps.

Explained Variance: Measure how well the network predicts how much reward it can obtain

Value Loss: Measures how wrong the model is at predicting the outcome

Hypothesis:

Since both explained variance and value loss plateau around 100k timesteps, it tells me that the model is unable to improve, even with more training.

The first thing that comes to mind is to adjust the 'gae_lambda' parameter. Since Farkle is mostly a game of luck, lowering 'gae_lambda' should reduce the randomness of luck from the equation.

Additionally, to address the fact that the model stopped improving, I should adjust 'ent_coef' which encourages the model to try different approach.

Second Run

What Changed?

timesteps: 1,000,000 -> 700,000

gae_lambda: 0.95 -> 0.5

Comparing baseline and v2

Looking at the 'mean_reward' metric alone, I can see that it actually did worse than the baseline.

Lowering 'gae_lambda' reduced how much long-term luck is factored into the model's training signal, whereas in my baseline run, the model weighed almost the entire rest of the game, which, in a game of luck, introduced a lot of randomness.

The 'value_loss' graph shows the same result: it dropped from 0.2 in the baseline to 0.04, confirming that the model is predicting more accurately.

The 'entropy_loss' graph shows that with the new change, the network became more confident and deterministic faster than the baseline, likely a result of a more accurate value function. But this meant less time spent exploring other strategies before it converges.

Hypothesis:

Even though the value function improved, the reward results showed the model actually performed slightly worse. This can be explained by looking at the 'entropy_loss' graph.

Increasing 'ent_coef' should encourage the model to explore more strategies before converging on one.

Third Run

What Changed?

ent_coef: 0 -> 0.01

Comparing v2 and v3

Comparing the two results, increasing 'ent_coef' left the value function's accuracy unchanged.

However, it significantly improved the reward by ~0.2

Why?

With increased exploratory randomness, the model spent more of its training exploring instead of prematurely committing to one strategy.

Gameplay

Model Import

After exporting the model from Stable Baselines3 to an ONNX file, I can import it into Unity.

Model Setup

The most important function is the predict function, the parameters need to match what the model was trained on, which includes an array of floats (Observation) and an array of booleans (Masked actions)

Farkle AI Simulation

Table of Content

Solo Project

Tools Used:

Skills Applied:

C#

Python

Unity Engine

Gymnasium API

Stable Baselines3 - MaskablePPO

Unity Inference Engine

The goal of this project is to learn how to train and apply a reinforcement model in a real example and game.

Gymnasium doesn't provide a Farkle environment, so I had to create a custom environment that defines the game rulesets, and a way to incorporate action masking.

Observation Space

An observation space describes what the model can read and learn from.

Space

The observation space is a single array of 10 elements, containing only information the model needs to see.

Normalize

From prior testing and training, I decided to normalize each element, to ensure the features are balanced and can converge better.

Action Space

The action space describes what actions the model are allowed to make.

Farkle is played using a maximum of 6 dice, and each player can choose any number of dice for scoring, in any order.

Given 6 dice, there are 63 possible combinations of selecting the dice, given that each dice can only be picked once, and not all of the dice needs to be picked.

Why is action space 128?

In Farkle, players can choose either to bank the selected dice, or reroll the remaining dice. If there are 64 (0-63, including selecting 0 dice) actions, I can define that the first 64 actions are selection of dice AND BANK and another 64 (64-127) actions, I defined as selection of dice AND REROLL

How does the model know what dice it selected from picking a number between 0 to 127?

Each number can be represented as binaries.

For actions 0 - 63, I can represent all actions as a 6-bit binary.

0 0 0 0 0 1 = Only selecting first dice

0 0 1 0 1 0 = Selecting 2nd and 4th dice

1 1 1 1 1 1 = Selecting all 6 dice

So when a model chooses an action, it returns an integer from 0-127 All I need to do is translate that action into "Dice Selections" and "Bank or Reroll"

Action Mask

In most scenarios, there will be only a few options a bot can choose, since most combinations are not scorable. So I need to introduce an action mask to mask out disallowed actions.

Why are the dice SORTED?

Notice that the dice are sorted,

Imagine rolling a 'straight' with 6 dice,

1 2 3 4 5 6

This is the same as

6 5 4 3 2 1

To the model, each of these is a different permutation - a straight alone has "6! = 720" possible orderings. Without sorting, the model would have to learn each scenarios separately.

By sorting the dice and eliminating positional randomness, I reduced the number of checks needed during training.

Farkle Ruleset

The next step is implementing the actual ruleset for Farkle.

Calculating Score

The next step is implementing the actual ruleset for Farkle.

The scoring function is split into two parts

1st, Account for all special combinations with 6 dice

2nd, Account for every other possible scoring combination

Step and Reset Function

See the code here: FarkleSimulation.py

API / Library

Stable Baselines3 is an algorithm library built to be compatible with Gymnasium's environment.

Algorithm

I chose Proximal Policy Optimization algorithm - more specifically a MaskablePPO - because of Farkle's discrete action-space, where in any given game state, most of the actions are illegal.

Actual Training

Hyperparameters

Hyperparameters are parameters I can control that tells the model how it learns.

Learning Rate:

How big a step the model takes each time it updates and "learn"

N Steps:

How much of the gameplay it plays through before it stops and learn

Batch Size:

After gameplay and its results are logged, how much of it does it learn from at a time

N Epochs:

How many times the model studies the same batch of gameplay

Gamma:

How much the model cares about future rewards versus immediate rewards

Gae Lambda:

How far ahead the model looks when determining how much an action is considered good

Ent Coef:

Encourages the model to try different approaches instead of settling on one approach

Time Steps:

How many steps to take for the entire training

Default Run (~20 minutes)

First I ran the model with 1,000,000 time steps, and leaving everything else as default

Then using TensorBoard, I can retrieve the training log:

This is a graph that shows the average rewards during its training, showing that it peaked at around 400k - 700k time steps.

Explained Variance: Measure how well the network predicts how much reward it can obtain

Value Loss: Measures how wrong the model is at predicting the outcome

So when a model chooses an action, it returns an integer from 0-127

All I need to do is translate that action into "Dice Selections" and "Bank or Reroll"