Combine 2D and 3D

Paper drawn Stickman placed in a 3D photogrammetry world generated from photos by using free open source software Meshroom

Combining 2D and 3D art, objects and animations - A general overview

There are many options for combining 2D images with a 3D world. The result can vary from hopeless and terrible to great.

Some examples:

Paper drawings or 2D work to 3D world.

If your drawing or image have high contrast and lacking small details, it may work to convert it to a 3D shape.

If you just want your drawings to appear in a 3D world, just make the background transparent and import them as planes and put them where you want. If you have an animated set of drawings, they can appear as a set of planes and you can let the animations take place on one spot or let the planes move around in the scene as you wish. This is pretty easy to do with keyframes in Blender, you can also use keyframing in Unity or move the planes with a script.

Photos. Real life photos are more difficult to convert to 3D objects or shapes due to their rich information of color nuances.

If you have a great nature photo, use it as a background, or create a skybox or perhaps even a HDRI image from it. Create 3D shapes in front of this background and use colors from the photo to create materials for the 3D objects to "melt" together with it. (Remember that you should probably not give the background a lit material, rather use an unlit or emissive material, since the plane background will not reflect light as an object.).

If you have a photo with a larger area of a similar texture or color, like sky, grass, wood, rock, then it can be a great source to create a texture material and use that on a 3D object.

If you really, really want to try to get a real environment into a 3D shape, then there is Photogrammetry, it is difficult, takes good planning and needs post work, and the meshes tends to get huge, but it might work. (Example below.)

Converting 3D objects and scenes to 2D images are on the other hand quite easy.

Some examples:

You would like to have a background for your 2D cartoon with more complex light and perhaps it should change over some scenes.

Create a scene in Blender or Unity, give it some nice lighting. If you're in Blender you can render the images to a chosen quality. If you're in Unity, you can build the game scene and from the built game scene, take screenshots of the scene.

Tips

⦁ Keep in mind that information in images and objects are for a computer totally different. Images are pixels (color information), objects are vertices (points in space). A 2D plane however is a similar object as a 3D object, it is just the number of vertices that may differ. You can "dress" an object with an image, then it is called a texture.

⦁ Keep it simple

⦁ Plan and examine what you can do and what may lead to fruitless work.

Bring a 2D paper Stickman into a 3D world - Tutorial

In this example a 2D paper drawing of a simple Stickman is placed in a 3D world created by real photographs generated to a digital mesh in the free open source software Meshroom. The process of creating a 3D object from photos are often called "Photogrammetry". (Of course, you can create your 3D world by modeling or use some of the great objects that come with Blender .)

The generated mesh can then be imported to Blender as an object (.obj format).

The Stickman can be animated by importing a transparent image sequence to Blender. The image sequence are then connected to a path in the scene together with the generated world mesh.

To view images below in full size, right click and choose "Open image in new tab".

Photogrammetry

Create mesh in Meshroom

Take a lot of photos that covers your object from different angles. The Meshroom process is fairly simple, you don't need to change anything if you don't wish.

If the process is successful you can now find both the object and its texture material in a folder where you saved your project. Normally you'll find it in the folder called "Texturing".

You may later want to change the UV of the object so that the object uses one material with one image. This can be done in Blender. If you saved your project from start, you can rerun only the texturing phase and use the modified object from Blender. The new textured object will then get a new folder in the "Texturing" folder. You can distinguish between the versions by looking at the date and time.

(If you are unfamiliar with photogrammetry and Meshroom, check out some Youtube tutorial and start with a simple object as a test.)

Import .obj mesh to Blender

In Blender, go to File - Import - Wavefront (.obj), locate the "Texturing" folder in the Meshroom project., select the .obj.

When importing and exporting between Blender and Meshroom, select Keep Vertex Order under Geometry Settings, at export from Blender you may also need to check "Triangulate Faces".

Chances are your object is quite large in triangles/polygons. Depending on what you created, it can be huge, like over a million triangles, and it could be difficult to edit. There are probably also parts of it you don't need, so the first thing you need to do is to get rid of a lot. Go to edit mode and remove vertices you don't need. Tip: Turn on X-ray mode to select in depth or it can take a very long time. Careful not to remove parts you want to keep, you can fine edit later.

After this first rough cleaning, go to Object mode and add a Decimate modifier (if you still have a large object). You can try to reduce the object by 0.8, 0.5 or more, this can help to make it easier to fine edit.

Viewing the object with its texture.

If the texturing phase came out all right in Meshroom, you should be able to view the result in for example Blender's Shading tab mode. The objects from Meshroom should be rotated from 90 to -90 on the X axis in Blender to be upright. Remember to do this or the object will look funny with Blender's default light. (Also remember to set it back to 90 if you need to export it back to Meshroom.)

(If you are unfamiliar with photogrammetry and Meshroom, check out some Youtube tutorial and start with a simple object as a test.)

Create a path for your actor

To let an actor, in this case Stickman, move in the new world, you can create a path by laying out a curve. You can then let the actor follow the path. Read below

Import Stickman sequence to Blender and animate to a path

In Blender

Note: Import Images as Planes simplifies the use of a plane as an image. If you don't have set Preferences to use this, you can do this by: Go to Edit - Preferences. Select Add-ons, scroll to "Import-Export: Import Images ad Planes", check that box. Close the Preferences Window.

File - Import - Images as planes (or Add - Image - Images as planes)

Select your images.

In the import window, under Import Options, notice a checkbox "Animate Image Sequences", check it, it will now treat the images as one object plane with a set of images.

In Shading mode, yoiu can see the Texture Image node with the Image Sequence, check Cyclic so the animation will loop.

In Object mode, notice under Transform window that the object are rotated.

Object - Apply Rotation will confirm the rotation and set the numbers to zero.

Import Image Sequences, check "Animate Image Sequence"

Set Image Sequence as Cyclic to loop

Notice rotation on imported image

Apply Rotation will set to zero, easier to work with

Set a path for Stickman

In Object mode

Add - Curve, try Path (With Edit - Preferences - Add-On, Add Curve: Extra Objects, you can get additional set of curves.)

In the little window that pop out down to the left, set Radius/Length

In Edit mode

Use Circle select to click on two Control points if you want to subdivide to more points between them.

Try Proportional Editing if you would like to move one or more points and would like their neighbour points to follow smoothly.

Try mark one point, press G plus one axis, X, Y or Z (height)

Notice a circle marker, use mouse wheel to make it larger or smaller, this will control how much neighbours are influenced when you move a point.

After you've created a Curve or path that you are satisfied with, Select the curve's object data properties to the left in the Properites window.

Path Animation should already be checked. Under that you can set how many frames traveling the path should take.

Next, Select the Image Plane object. In the Properties Window, select Object Constraint Properties. Click on Add Object Constraint. Choose the "Follow Path" alternative.

In the field called "Target", click to select the curve object as the target.

After you do this, you should see the Image Object jump to the start position of the curve.

Add Constraint

Choose Follow Path

Choose Target

Set Curve Object as Target

The easiest way now to create an animation is to click on "Animate Path" (it is in the same Constraint properties window), this will create an automatic animation along the path. There are other ways also which would give you more control right away, by using keyframes and ease in and out and more, but for now try click "Animate Path".

Now, if you press play on the timeline, the image plane will travel along the path and perform its image animation at the same time.

You can read more about the Follow Path Constraint in Blender docs.

Position the Camera

One way of doing this is to move your viewpoint with your mouse until you have a nice view of your scene.

Then select the Camera object, go to View - Align View - Align Active Camera to View. Now you should be able to see "through" the camera window. If you select Animation tab (or have two windows), have the Camera view in one window, and look at the camera from the "outside", select the transform tool (to the very left of a window, you may need to pull it out with the arrow). Choose "Local" up in Transformation orientation field (default it says Global). Now you can more easily finetune the position of the camera.

Since the "actor" is a 2D image, it needs to be rotated towards the camera to be visible. (After you have positioned your camera, rotate the image plane so it is facing the camera. If you don't check the option "Follow Curve" under Add Object Constraint, the image will not change its rotation along with the curve's direction, just move its position.) If you at some time want the image to switch rotation on the z-axis (from right to left, mirror like from 0 to 180 degrees), you can just key-frame that somewhere during the path.

One way to do this is by, in 3D Viewport, on frame 1, press I to insert keyframe, select Rotation. The Rotation part of the Transform information in 3D viewport will turn yellowish meaning on that frame there is a keyframe.

If you have a window with the Graph Editor, you can see an orange dot on the timeline where there is a keyframe (when the object is selected).

Continue to the frame before the frame you'd like to change the rotation, press I again to insert a keyframe. Now the rotation is "locked" until this frame. Go to next frame and change the rotation to 180 on z-axis. Press I - Rotation to confirm the rotation.

You can read more about Keyframes on Blender docs, Insert and Editing Keyframes, Keyframe Animation and more.

Set your material.

In Shading tab/mode (and with Material properties selected), if you need you can add some colour and Emission strength to your image.

Use a mesh with modified UV

If you have a large object, Meshroom may have created multiple image materials with a UV that stretches beyond a usual single image UV. If you're just going to use the object as a background or so it doesn't need to be a problem. If you want to some further work with the object you may want to edit the UV so that the object uses one material with one image. To do this, don't change position, rotation or scale in Blender. Select the object and select the UV Editing tab, then you should be in Edit mode.

Create a new UV with one material. (If Meshroom have created multiple materials you may want to remove all except one . If it is a large object, Smart UV may work best. Export the object from Blender, File - Export - Wavefront (obj). Under Geometry in Export settings, check Include UV's, Triangulate Faces, Keep Vertex Order.

In Meshroom, if it is not already open, open your previous saved project.

Remove the connection from MeshFiltering to Texturing node. Right click on the connection/wire and click Remove. Click on the Texture node to make it selected. Now you can see the texture settings in the window to the right. Next to the field "Mesh", write the path to where you saved your modified mesh. A simple way to copy path is to press Alt + Shift and right click on the file in Explorer and choose "Copy as path". Paste the path into the field and make sure to remove the quotation marks. Set the Texture size as high as you need. (Texture Downscale to 1 for full scale.) Then press Start, Meshroom will now run the texturing phase again and save it to a new folder in Texturing folder. (The green bar will be empty at the very right end when you disconnect the Texturing node and turn orange when you restart and turn green again when it is finished.)

After this, you can open this new textured version of the object in Blender and it should now have the texture with the new UV.