For this project I was interested in exploring the registration of 2D images from a 3D world with a player-controlled camera and being able to recognize the elements photographed for gameplay uses. My goal was to be able to identify and categorize game objects, their types, qualities, current states and actions. Based on how visible objects appear on a photo, scores could be calculated which would represent how well the desired entities and characteristics were captured by the photographer.
In most games taking a photo is nothing more than a screenshot, capturing and storing pixels for aesthetic purposes. I’ve always wondered why so few games use this functionality as a part of the actual gameplay. One example of a game that does use it, and which initially inspired this project, is 2003’s Beyond Good and Evil. And even then, it’s only for a sidequest here or there, it’s not the game’s main mechanic. But what if it were? I can easily imagine games featuring things like espionage or the cataloguing of alien lifeforms being able to put a complex photography system to good use.
I decided to do this project in the Unreal Engine (4.24), having set a secondary goal for myself to learn how to use it since I’d never touched it before. I tried to do as much as possible in C++ and only use blueprints for things I consider unrelated to the main goal (such as materials and UI layout).
Taking a Photograph
After a couple days of learning the basics and familiarizing myself with the engine, I got the basic gameplay and camera in place. The next step was to add a Scene Capture Component 2D so that I could capture what the player was seeing and put it onto a material to be displayed in the UI. Then I had to make sure it didn’t update every frame, since that's the default behavior, capturing instead a single image when the left mouse button was pressed.
SceneCaptureComponent2D creation and use
To make this the actual photograph I was after, and not just a screenshot, I had to figure out which objects were inside the camera frustum at the time of the photo. More specifically, the objects with a Static Mesh Component, since I’m only interested in objects that can be seen.
Getting the objects in the camera frustum
Since I don’t want the photographs registering every single mesh in the frustum, I created a PhotographableObject component which can be added to any gameplay relevant actor. The component contains the name of the object and it’s types, qualities and actions bitmasks. These can all be set in the editor which makes it easy to add and edit the objects the camera should be able to identify.
Setting the Quality tag of a photographable object
Setting tags in the editor
UI showing tags in use
Each object receives a score based on its visibility: how distant it is from the camera and if it’s being blocked by other objects. An object starts out with a score of 100 which is multiplied by two multipliers, one for each of the visibility checks. If the final score is less than or equals to zero, the object is ignored and isn’t identified by the camera. Otherwise it gets a score depending on the score’s value: terrible, poor, good, great, excellent or perfect.
Objective #1: ”Take a perfect picture of The Core”
This objective was designed to show the scoring system. The Core can be photographed from different angles, but it is blocked by many obstacles, which reduces its score. To complete the objective the player must find the correct angle in which they can get a completely unobstructed view of the target and get the perfect score.
Different scores based on visibility
Object Visibility – Distance
Objects that are too far away from the camera shouldn’t be registered, even if they are inside the frustum. I calculate the distance between an object and the camera and if it’s greater than a set limit, the distance score multiplier is set to 0, which results in a final score of 0, making the camera ignore the object. On the other hand, if the object is closer than another set value, it is considered at a perfect distance (the multiplier is set to 1). Anything between these values is interpolated. At first I also had a distance which was too near, and the score would start to drop again. But then I realized that what I was trying to do was a matter of framing: the object being too close to the camera that you couldn’t see the whole thing. I realized that this was actually something the obstruction calculation (described below) should be taking care of.
One problem I had here was that this method of calculating the distance multiplier wasn’t really working for small objects. The coins, for example, were getting a perfect score even though they were very small on the screen (but technically within the correct distance). It just felt weird. I tried to change the calculation to use the visual angle by completely ignoring the actual distance and instead measuring the angle between the lines from the camera to the min and max points of the object’s bounding box. Though this seems like a much better solution, it wasn’t easy to get the near and far values feeling right. I ended up going back to the initial distance check, but making the near and far values proportional to the size of the object. This seems like a very simple solution, but it ended up working very well.
Objective #2: ”The three planets”
This objective was created to show the distance calculation in use. Near the beginning of the level you can see three planets outside a window, but only two are listed when a photo is taken because the last planet is just out of range. At the end of the level the player gets access to a balcony which is slightly closer than the initial window. From here all three are in range.
Objects aren't identified if they are too far away
Object Visibility – Obstruction
The second part of the score equation is obstruction. If an object is inside the camera’s frustum and close enough, it still shouldn’t get a full score (or be identified at all) if it’s being blocked by other objects. For this I send out rays from the camera towards the object's center and the eight corners of its bounding box. If the ray collides with another object, I decrease the score multiplier. If enough rays are blocked, the multiplier reaches 0, but only if all are unobstructed is it a perfect score (1). Precision can easily be added to this calculation by creating more rays to points in between the existing ones, but these 9 tests seem to be doing a good job for now.
Before even doing these checks there’s one more thing that needs to be done: checking if the points I’m raycasting to are inside the frustum. If not, they are treated in the same exact way as if another object were in the way. This is the framing check I was trying to do in the distance score calculation mentioned above.
Raycasting to determine how obstructed the object is
Objective #3: ”Find the One True Cube”
With this objective I wanted to show the obstruction calculation being used. The target object is floating behind four moving blocks. The player must wait for the blocks to sync up and move out of the way to get a clear line of sight to the target.
Objects aren't identified if they are blocked by other objects
One reason I decided to add objectives was to explore and present the different features and make sure the player learns of the uses, limitations, challenges and possibilities of this camera mechanic. Each objective was created to explore a different aspect in addition to showing the possibilities with different types of objectives.
Each objective has a number of subgoals. Every time a new photo is taken, each objective that hasn’t been completed yet is checked by seeing if the objects registered by the camera fulfill all of its subgoals. The objective creation interface permits adding subgoals that check for an object with a specific name or an object with certain tags. Objective #4: ”A small statue, a big statue”, for example, looks for one object with the Statue and Small tags and another with a Statue and Big tags, while Objective #3: "Find the One True Cube" mentioned above looks for an object called ”The One True Cube”.
It is also possible to define how many of these objects are required and if they are collectable. Collectable subgoals can be completed with multiple photographs. Objective #2: "The three planets" mentioned above requires a photo of three objects with the Sphere tag, and both Objective #5: ”Find all of the secret coins” and Objective #6: ”Find the source of the music” require the player to photograph six objects with the Coin and Secret tags, and two with the Radio tag, respectively.
Objective #7: ”The Red God watching over its two moving children”
The last objective requires the player to take a picture of two small cubes that slide around on the floor. The objects’ actions can change (technically, any tag type can be changed). The point of this objective was to make it clear that when the cubes aren’t moving they don’t have the Moving tag set, therefore forcing the player to time the photo correctly to catch them both in motion at the same time.
Tags being added and removed at runtime
I’ve learned a lot working on this project and I'm satisfied with both the final result and with how much I’ve learned of the Unreal Engine. That being said, there is of course so much more to learn, and I feel this project could still be greatly expanded. In fact, I look forward to working on it some more in my spare time. I still see a lot of potential here, so a few ideas of what could still be added:
- Photos could be stored and viewed at a later time in a photo album
- The obstruction score calculation could use more raycasts between the current ones for more precision
- The total score of a photo could be used as the value of the photo, perhaps sell them to NPC’s? There could even be objectives where the goal is simply to take a pretty picture, without any specific target
- Some objects might require being photographed from a certain angle, or with a certain part being visible. Humans, for example, might require the face to be unobstructed
- Lighting could affect the score
- If the score is too low the object could be only vaguely identified. A high score could translate as a monkey, while a low score as a mammal
- Camera focus, both as a visual effect and as a mechanic. Things out of focus would result in a lower score
- The camera could have upgrades such as zoom and night vision, which would allow the player to properly photograph previously unavailable objects
- Objectives could be created with blueprints and even added at runtime
- Consider, or even try to teach the player, actual photography concepts like the rule of thirds
- Improve and expand the UI
Click the Download button below to give it a try.