While AI-based tools like Sparc3D or Hunyuan 3D can generate genuinely impressive results from a single photo, they often rely on credits, paywalls, and non-so-transparent processing of data. In this video, we strip it down to basics: a smartphone, free software, measurement-based photogrammetry.

Free Photogrammetry Software

RealityScan Mobile

Agisoft Texture De-lighter

And of course, Blender is a hub for assembling, editing and previewing photoscanned content.

—

Photogrammetry: Best Practices and What Camera to Choose

The capturing practices described in this tutorial still apply to mobile photoscanning, so it could be worth (re)watching.

And here’s the basic guide answering the question of “what camera to choose for photogrammetry”. Or, understood more broadly – what makes capturing device better or worse for the purpose of capturing photographic data.

If you haven’t heard the term spatial fidelity before, make sure to watch this video.

—

Photogrammetry Course

If you’d like to go deeper into DSLR-based photogrammetry, advanced capture techniques, and desktop RealityScan workflows (again, way more advanced), check out our full Photogrammetry course.

On Superhive

On Gumroad

For this week only (till the 3rd of March) it’s -25% OFF (Winter Sale).

—

Sparc3D: An Alternative 3d-from-image Method (AI)

In this video, we have explored an alternative 3D reconstruction method – represented by SPARC3D here – and discussed what makes it stand out, along with its potential impact on asset creation and 3D workflows.

The post 100% Free Mobile Photogrammetry Workflow: RealityScan appeared first on Creative Shrimp.

https://lizhihao6.github.io/Sparc3D/ – Sparc3D – Sparse Representation and Construction for High-Resolution 3D Shapes Modeling by Nanyang Technological University. This method offers a robust alternative to traditional photogrammetry by generating detailed 3D meshes from a single input image.

https://huggingface.co/spaces/ilcve21/Sparc3D – live demo

Tencent’s Hunyuan 3D 2.1 (open source) alternative

—

What is Sparc3D? A Diffusion-based 3D Mesh Generator from a Single Image

Developed by Nanyang Technological University, Sparc is a new algorithm that builds 3D meshes from just A SINGLE IMAGE! It’s a significant leap beyond anything we’ve had before.

While traditional photogrammetry captures real-world objects using multiple photos to create detailed 3D models (something we dive deep into in our Creative Shrimp Photogrammetry Course), Sparc takes a different approach. This new tech can handle complex inputs – open surfaces, intricate geometry – all from one photo, powered by the latest in diffusion models. It’s pretty wild.

This technology has serious potential to shake up our industry. I don’t particularly like the word “disrupt”, as it’s overused in all sorts of silly contexts. But it feels disruptive.

Just browse through that comment section (1100+ comments, my god!), and you’ll see what I mean.

Now, let’s see what it means for us all.

—

How to Use the SPARC3D Demo

There’s a public demo online right now.

For how long? Who knows – on the one hand the authors said something about making it open-source upon receiving all permissions, but you can never rule out a sudden sale to a commercial company.

How to use SPARC demo?

• upload an image,
• hit “generate”
• and after a short wait (the queue is starting to get longer though, haha!),
• you get a full 3D model.

If you don’t get a full 3D model, then it’s most likely the fault of the source image. Try again, with a different one.

—

Use Cases: Game Assets, AI Art to 3D Props, and More

So, what’ll be the most popular use case (popular as in pop songs)? I’m betting on this: turning AI-generated images (from MidJourney, ChatGPT, Stable Diffusion, Rundiffusion etc.) directly into 3D props and assets. Not that we endorse that – but let’s be FRANK, it’s the first thing that comes to mind and I can already hear the sloshing of AI slop (with occasional amber) crashing onto our shores. I’m bad at metaphors though.

And before you asked – I’m writing this article mostly in an old-school way, with just Aidy GPT being my only helper.

What is so new and revolutionary about Sparc?

Well, it does something crazy: it fills in the blanks. The underside, hidden geometry – whatever’s missing from your single image. This workflow makes mass production of 3D assets remarkably feasible, compared to an older-school photogrammetry (where abundance of input images has always been necessary, to avoid gaps and general geometry syphilis).

—

Limitations of SPARC3D (And What Needs Improvement)

Is the resulting model a mess? Yeah – it’s not fully production-ready (yet). You still get dense meshes, no real topology flow, no textures (in Sparc3D at least – some other tools can do textures). It’s still a pain in the back to do retopo. It’ll probably take a few more research papers to change that.

But nevertheless, the surface quality is a generation ahead. Flat surfaces are actually mostly flat now. The connections are smoother, more coherent. The geometry actually doesn’t look that bad (I’m surprised to say that).

What sets Sparc apart from traditional photogrammetry, where meticulous coverage is key, is its ability to infer geometry on the hidden side of an object. It even tackles thin, intricate parts surprisingly well –something previous 3d reconstruction tools always struggled with.

To be fair, gaussian splatting solved thin and furry stuff to *some* extent, but this is a step forward (and it outputs good old polygons, unlike gaussian splatting).

—

The ‘Traditional’ Photoscan Apps Will Have to Upgrade Their Tech

I’m dying to see what Reality Scan’s (aka Reality Capture’s) response will be. I bet the devs are already speedrunning the ways to integrate a tech like this into their app – which is one of the best already, and just waits for that extra push. Gosh, I can only imagine the power it’ll give to the tech that was somewhat overdependent on physical world constraints (objects that are often impossible to shoot from all sides, or cars covered in birds’ excrements making it difficult to scan).

And how would it interplay with the latest advances in triangle splatting, 3d gaussian splatting and more – another tech is becoming mainstream really quickly?

—

The AI Slop Wave Incoming?

This blend of increased speed and quality of this new diffusion-based method makes it clear (to me) we’re heading for an avalanche of AI slop. Soon we’ll see a lot more models that are good enough, mass-produced and flooding the market. Like a viewer of our original video pointed out: the future will be good enough.

Putting aside the massive issue of the environment for a moment, this raises big questions about copyright, too, and how marketplaces like Sketchfab will handle it. Remember that user who uploaded 38,000 AI assets to Fab?

We’ll see more of that.

—

Blender’s Role: The Hub

So what is Blender‘s role in all that? With a huge influx of new 3D props (if and when it happens), Blender effectively becomes the hub – the central point where all elements are assembled and made sense of.

As we’ve mentioned, the Sparc-generated assets are similar to photogrammetry exported assets: messy, dense, high-poly. These will have to be prepared for exporting to Unreal Engine, Unity, Godot or other game engines, or just for using it within Blender for scene creation. From this perspective, very little will change for 3d artists… just yet.

Your skills remain valuable, YOU remain valuable and this will likely continue…

for some time.

—

If you’re looking to take these high-poly, often messy assets and optimize them for games or other production pipelines, our Game Asset Creation Course focuses specifically on retopology, optimization, UV maps and efficient asset preparation.

Just mentioning.

—

Creativity vs Complacency

This new accessibility that AI-driven tools bring to life is pretty exciting – and as many of you pointed out in the comments to our video, pretty disturbing as well. If you have an image – whether your own creation or something found online, or (god forbid), ai-generated – you are now significantly closer to realizing that thing in 3D.

From there, the possibilities to build your own worlds and tell your own stories are immense.

However, as Aidy pointed out when proof-reading this article: fast and easy does not always equate to better or more meaningful. Like processed fast food, that’s usually an easy option but still best used with caution. AI should probably serve as a tool to augment our creativity, not diminish the artistic skills that make our work distinctive. Think atrophied creative muscles.

The tech like SPARC will spark (ahem) new workflows and a lot of questions: Where does it fit? Kitbashing? Concept art? What about automatic retopo, for heaven’s sake? Expect tools like ZBrush’s ZRemesher or Quad Remesher to become even more valuable, as clean topology could be in higher demand than ever.

But why, oh why, AI-based automatic retopology isn’t a thing yet.

Also, will Blender cement its role as the hub where such content is being assembled?

—

So many questions!

So, what do YOU think? Do you feel threatened, or see this as an opportunity to level up your workflow? Let us know on Discord!

That’s it from us, Gleb Alexandrov and Aidy Burrows at Creative Shrimp, keeping you updated on the latest in computer graphics.

—

Links

https://lizhihao6.github.io/Sparc3D/ – Sparc3D

https://huggingface.co/spaces/ilcve21/Sparc3D – live demo

Tencent’s Hunyuan 3D 2.1 (open source) alternative

TrueDepth add-on for Blender – tutorial

Dr. Károly Zsolnai-Fehér – Two Minute Papers channel

—

Game Asset Creation Course

Photogrammetry Course: Photoreal 3D With Blender And Reality Capture

Creative Shrimp Discord

Gleb Alexandrov Youtube

—

Are We Screwed? The original Youtube video

PixelReconstruct Youtube channel – gaussian splatting and more

Photoreal Environments and Photogrammetry: an article

RealityScan

Blender

The post SPARC3D: Revolutionary Single-Image 3D Reconstruction Explained (with Live Demo) appeared first on Creative Shrimp.

Links:

Luma AI

Polycam

Nebula gaussian splats in Polycam (preview)

Nebula gaussian splats in Luma AI (preview)

Getting Started With 3D Gaussian Splatting for Windows (Beginner Tutorial)

GitHub Repo for Windows

Nebula course on BlenderMarket

More about photogrammetry workflows (including Gaussian Splatting)

Nebula course on Gumroad

The post I Photoscanned… a Nebula appeared first on Creative Shrimp.

—

As it turns out, we’ve been using photogrammetry quite a lot in our Creative Shrimp projects. We’ve been using it to digitize historical artifacts; for capturing details that would be a pain in the neck to model in any other way. We’ve built some miniature movie sets, like this Millennium Falcon, then photo-scanned it for fun.

Most importantly though, we’ve been using photoscanning to create life-like 3d environments. Mostly abandoned hallways and such, made of the shards of Brest (Belarus), Warsaw, Vilnius and other places we’ve been to.

—

A GENERIC PHOTOSCANNING WORKFLOW

There are a few really interesting things that we can do with photogrammetry as 3d artists and as environment artists…

But first, here’s a quick refresher of what a typical photogrammetry workflow entails.

• capture a bunch of photos of an object or an interior space from different angles;
  using the best capturing device you have; making sure there’s enough overlap between the photos and everything’s sharp and clear;
• load these photos into your photogrammetry software of choice;
  eg Meshroom, Colmap, Metashape, Reality Capture, etc
• let the software calculate similarities between the source photos and spit out a point cloud that can then be used to deliver a high density 3d surface;
• optionally, with texture;

That’s how it works. That’s the generic workflow anyway.

—

CAPTURING EQUIPMENT

What equipment do you need to start photo-scanning?

We often hear that you just need a device with a sensor and you can start doing photogrammetry. And that is true. But as a 3d artist, I find it somewhat frustrating how much photographic equipment has accumulated over time, in addition to my DSLR camera (I use a Sony A7iii with a Tamron 24-75mm f/2.8 lens or some of my primes because primes are generally sharper, but there was a time when we were using a Canon 80D with a Sigma 17-55 f/2.8 or pretty much any DSLR or mirrorless camera available).

Then you start thinking about getting a tripod. It is a must-have for keeping shots steady.

A ruler and a color checker come next.

And then: “Should we get a drone?”

It escalates quickly.

Let’s put it like this: a low or medium-range DSLR or mirrorless camera is probably all you need if you have just started.

But what camera is best for photoscanning? This video is as close as it gets to answering the question “What camera is best for photoscanning?”. There, we talk about the 3 parameters of any capturing device that make all the difference in photoscanning quality.

And in this one we talk about what are the optimal camera settings for capturing the highest quality photos for 3d reconstruction.

—

PHOTOSCANNING SOFTWARE

As for the software, there are many options to choose from.

What we’ve been using mostly is Reality Capture and Agisoft Metashape. Fantastic tools, really fast and good quality. And commercial as well. Reality Capture has an interesting pricing option called PPI or Pay-per-Input (here’s our video with some advice as for optimizing the cost).

Meshroom and Colmap are the open-source alternatives. Which are also quite capable, in fact fantastic considering the price, just not as obscenely fast as Reality Capture and Metashape just yet.

Somewhat counter-intuitively, the open-source tools are arguably *more* demanding about the quality of the photos (for now). In other words, you may need slightly more expensive equipment and a more powerful computer to get the reconstruction results that are on par with the aforementioned commercial software options.

Still, we should absolutely support open-source, because we know what happens to commercial enterprises.

It’s also worth noting there are also mobile apps like Luma AI and Polycam.

—

5 PHOTOGRAMMETRY WORKFLOWS FOR ENVIRONMENT ARTISTS

When it comes to environmental design and photogrammetry, there are different ways to use the toolset offered by 3d reconstruction software. Here are 5 workflows (not the exhaustive list!) that are distinct in their goals and the type of data that you gather.

Distinct enough so we can write about each of them separately.

1. THE WHOLE SEAMLESS ENVIRONMENTS

2. MODULAR ASSETS

3. CHARACTERS

4. PBR MATERIALS AND MAPS

5. LIGHTING

—

WORKFLOW 1a: (CAPTURING) THE WHOLE SEAMLESS ENVIRONMENTS

The first workflow on our list is capturing the whole seamless environments. This might seem silly, what would you do with such a model anyway? But it’s the one that has given us some amazing location snapshots.

When we arrived at the abandoned warehouse near Brest (Belarus), our goal was to digitize the Volga car that has been standing there for half a century, collecting dust and bird excrement.

And that’s what we did.

But then, before leaving the place, Nik proposed to bring a drone next time to capture an isometric view basically.

And when I saw it reconstructed in Reality Capture, I thought, damn! I can totally imagine this as a game level in some indie RPG that uses the isometric view.

Especially if you pepper it up with dramatic lighting.

—

Double so if you throw in a playable character (this is a quick demo put together in Godot by Aidy).

—

If you’re lucky enough to have some amazing historical locations nearby, like this 18th century Bernardine monastery in Brest, rebuilt into a military hospital, a gem in our collection of abandoned places, photographed some 15 years ago, you can totally get some nice environments captured almost in one piece. Given you have enough time to spend there.

Even though it’s not a super flexible workflow, capturing the entire space like that, walking with a wide angle lens and an on-camera flash through the dark and cold corridors, you can’t argue with the photorealistic output it produces.

It just looks… real. Which it is.

Even with a cinematic lighting treatment in Blender it still looks arguably better than many games. It’s hard to beat that kind of photorealism, which comes unfortunately at the cost of flexibility.

—

THE CONS

Aside from that, a notable downside of such approach is that as every polygon and texel in such model is unique, the meshes and textures end up pretty heavy.

• The good locations are hard to find
• The lack of versatility
• Performance

—

That being said, even the lower res environment scans found on Sketchfab like this funerary monument by Morbase | Museu Virtual can work really well as a base for your next Blender animation.

Once you start throwing in lights, effects and particles.

—

WORKFLOW 1b: (CAPTURING) THE WHOLE ENVIRONMENTS – RADIANCE FIELDS

When it comes to creating the complete snapshots of environments, the neural radiance fields (or NERFs) seem to do an even better job.

Unlike the traditional techniques, these novel view-synthesis methods can actually capture a lot of extra data: the wispy details like vegetation, or hair, but amazingly also the dynamic material properties like reflection and refraction.

The stuff that was unthinkable using more traditional photogrammetry tools now comes to life thanks to Gaussian Splatting and other NERF algorithms.

The demo below was created in the Inria’s open source Gaussian Splatting Github app. The beginner tutorial by The NeRF Guru, without which it would be impossible for me to make sense of it, is linked here.

A caveat: it requires a powerful GPU (I have RTX 2080ti, but NVIDIA GPU with 24GB VRAM or more is recommended).

Speaking about reflections, here’s a quick demo of the highly glossy scenery that we built and scanned just before going to Blender Conference 2023.

Honestly it’s pretty mind-boggling to see it captured by the means of photogrammetry at all.

I wonder what happens when the animation part of the puzzle falls into place. Will it be a photogrammetry bingo card completed?

—

This tech is becoming mainstream really quickly, too. We can already try gaussian splatting in the Luma AI and Polycam apps (for our smartphones).

—

THE CONS

• tricky to edit
• non-versatile

The problem with radiance fields is that it isn’t super clear how to edit this stuff just yet. The tools for editing gaussian splats and NERFs are still in their infancy, in other words.

But this tech is developing fast. We should watch this space.

P.S. Here you can see a demo testing the thin-structures capabilities of gaussian splatting in Luma AI.

—

WORKFLOW 2. (EXTRACTING) MODULAR ASSETS

A far more versatile workflow of course would be a modular one.

Meaning, finding a cool location and dissecting it for props. With the goal of creating the asset library that captures the essence of this environment. So we can make *any* kind of environment like the one we’ve just scanned.

Incidentally, that’s the industry standard.

—

Going back to our warehouse environment, we could have scanned:

• The hero assets like this Volga
  
• The medium and small-scale props
  
• Surface textures
  
• Decals

Whatever can be salvaged there. The goal is cohesiveness. Scanning such library around the same environment is like a locked-in art direction. Everything in this warehouse was equally dusty, forsaken and coming roughly from the same historical slice. Imagine fine-tuning such a look across dozens of models. Using this workflow, you get it for free.

Have a look at any of the Scans Factory packs to see how realistic an environment composed of such modular props can look.

—

Of course, then each asset would have had to be optimized; meaning, polycount reduced and the geometry details transferred from the high to the low res meshes (as normal maps for example);

—

Sometimes this process is a breeze.

Say, when you can take a prop back home or back to studio and scan it on a turntable, and maybe flip it during the scanning process, so the software reconstructs a full 360 degrees watertight mesh.

Then it requires almost no post-processing, maybe aside from some polycount reduction (and that can be done automatically) and texture baking (and that can be automatized as well, to a certain extent).

—

But it isn’t always like that. In fact, it’s almost never like that!

—

THE CONS

• Planning
• Post-pro and clean-up

More often than not, raw scans are pretty messy and require an enormous amount of post-pro to make them useable.

Like, imagine cleaning-up something like that.

—

But anyway, nothing is more satisfying than scattering the photoscanned assets that have been prepared in advance. Gathered in the same location. Carefully optimized. The resulting environments can look just as cohesive as the full location scans.

—

WORKFLOW 3. CHARACTERS

Speaking about modular props, characters also fall into this category and can be made with the help of photogrammetry. Let’s say, the background NPCs for your environments like those seen in Ian Hubert’s videos.

—

As a quick reminder, it goes like this. First you make a quick scan, add a few bones in Blender, then animate these bones to rotate the torso and the head a little bit while recording the motion. And you get a nice background character doing the background character stuff.

The advantage of this workflow is that we don’t have to use a proper A or T-pose for scanning, we can just scan the characters in the poses they’re supposed to be, like sitting or riding a bicycle.

—

A more advanced variation of this workflow would be to apply the real motion capture data to such quick character scans using one of the markerless motion tracking apps like Mixamo or Rokoko Vision. You can even record your own motion and transfer it to the skeleton with these apps, like Aidy is doing here.

Still, it’s way simpler if the character doesn’t need to move.

People tend to fidget and make the scanning process more complicated than it needs to be. It’s my motionless body you see in this scan by the way.

—

THE CONS

• people are tricky to scan

—

WORKFLOW 4. PBR MATERIALS AND MAPS

Yet another application for photogrammetry in environment creation is, of course, capturing the PBR materials.

The process is no different from capturing props (or characters for that matter!): first we take a bunch of photos of whatever surface we want to scan, hovering over the muddy ground like a fairy; then reconstruct a high-definition mesh in photogrammetry software.

But then, we bake the detail from that mesh onto a flat plane to derive such maps as:

• the diffuse color (or albedo)
• ambient occlusion,
• normals,
• and most importantly the height data that contains the detail about the surface displacement

Then, these maps can be made seamless in Gimp by offsetting and painting across them all at once using the clone brush. This is a fairly new functionality that imitates the Affinity Photo approach to multi-layer editing, and we’re proud to have influenced this feature developed for Gimp with the help of ZeMarmot.

And voila! We’ve got the ingredients for a seamless PBR material.

The shaders like that work extremely well with displacement. No surprise, because in the end we feed it the real heightmap derived from the real surface. The result looks extremely realistic (and runs in realtime thanks to Eevee Next).

—

Additionally, we can use the extracted heightmaps for blending between different materials to create a much-needed variation. In Blender via nodes or alternatively, in Quixel Mixer.

—

THE CONS

• less flexible than procedural workflows (in Substance Designer and so on)

Aside from that, variation isn’t the biggest pro of this technique. The photoscanned materials are quite limited by the source footage. By its resolution and other properties. And simply by the physical availability, unlike their procedural counterparts.

—

WORKLOW 5. LIGHTING

The last type of data that can be extracted from a real-world location, that we will have a look in this article, is lighting. Lighting can be extracted and thrown away or extracted and… cherished.

Let’s talk about removing it first.

—

Typically, the shadows, at least the indirect ones, are removed from textures as soon as possible with the help of Agisoft Texture De-Lighter or other de-lighting tools.

So we can set our own lighting in Blender.

—

Alternatively, the lighting can be flattened during the capturing phase, by using an on-camera flash. This also gives the supernatural ability to shoot in the dark.

—

Or we can go as far as removing both shadows and reflections using the flash AND the polarizing filters. In the process called cross-polarization.

We don’t have time to go into that. But cross-polarization happens when you put a polarizing filter on both the lens and the light.

Like in this video by James Candy.

—

THE CONS

If you were capturing for a virtual movie set though, the lighting data would become precious. Definitely worth retaining!

To capture a full range of light, each shot can be taken multiple times using different exposure settings, then merged into a singular HDR image in the process called exposure bracketing. In Darktable for example.

And then, the photoscan can be made from such high dynamic range images. And as a result, such environment would emit a proper amount of light. So that the 3d objects put into such scene would receive a proper amount of light. And integrate seamlessly.

Imagine a better HDRI panorama that is fully three-dimensional – this is it.

—

THE CONS

• Processing for HDR takes time
• Versatility!

Unfortunately, preparing the HDR images can be time-consuming and the use-cases for such a workflow are truly specific.

—

CONCLUSION

Alright! These are just some of the things we can do with photogrammetry, as 3d artists.

Despite producing a non-versatile output sometimes, the tech itself is actually pretty versatile nevertheless. There is something for everyone.

So no matter whether you’re a 3d artist, a director, a cinematographer or a level designer; or maybe you work as a scientist or archaeologist, photogrammetry gives us some powerful tools and workflows to bridge the gap between our physical world and the digital realms we create.

—

Before we wrap it up, I’d like to say thanks to everyone who helped us on the ground to gather the content: Eugeny, Nik, Pawel, Kaciaryna, Ryncuk, Aidy, Lena – thank you so much indeed!

—

Feel free to check our Photogrammetry Course. Thank you!

The post Photoreal Environments and Photogrammetry: 5 Workflows For 3D Artists and Environment Artists appeared first on Creative Shrimp.