More Reasons to be Excited about the Scatter Device

As I mentioned in my last post, I’m currently doing a  pass through the Instance Scatter device, making sure that it works as intended before release.

Why Fractal Distribution Rocks

Someone asked me in email what I meant about the placement device accepting a height object as a fractal basis function. Here’s a great example to illustrate:

We’re going to take this linear gradient and circular alpha mask:

Linear Gradient and Mask
Linear Gradient and Mask

And use the Instance Scatter device with its fractal distribution turned on (persistence : .5, lacunarity: .5, octaves: 8) to produce this:

Instanced Geometry | Erosion
Instanced Geometry | Erosion

When you get to examine this yourself in the next release, you’ll see it has all kinds of interesting features reminiscent of a tilted sedimentary type terrain.

This is a super quick example with pretty basic inputs, but it shows how the scatter device is useful for so much more than just placing craters and volcanos.

Here’s one more, which is just slightly modified from the above:

Directional Ridges and Strata
Directional Ridges and Strata

To be honest, I didn’t  plan this custom-fractal ability originally; but once the idea to allow fractal distribution happened, it emerged. You’ll find yourself creating all kinds of interesting rock types and textures using the scatter device. Another nice perk is that these fractals are stable across space, and thus are fully tile-able and explore-able.

I fully expect to see lots of user-created custom terrain type macros grow out of this!

Very Soon…

Progress Report, now with images!

Some folks on the forums were asking the other day about how development progress is going. Things are getting very close now for the next Dev Channel Release. I’m doing a final methodical trek through each new feature, and writing example files to help folks learn how to use them.

Something to whet your appetite

Here’s an example image from within World Machine itself demonstrating the “Instance Scattering” device, that takes a single impact crater (defined in localspace), and scatters it across the terrain.

Fractally scattered craters
Fractally scattered craters

Most powerfully, the instance scatter device lets you treat the local object to be scattered as the basis function for a fractal; in the image above you can see that the craters exist at many scales throughout the terrain, as you’d expect from impact with space debris.

Why it takes so darn long

I hoped to have Localspace support out about 3 months ago. There are many reasons why this release has been delayed, but part of it is the following process:

Each device slowly accumulates a wishlist as I both play around with it myself,  and also while I write example worlds using it. Oft times, the wishlist items would be nice to have but are not fundamental, so they go on the backburner. Other times, I’ve realized that I’ve missed a required fundamental ability. When I realize that, instead of putting it on the wishlist I dig in and add it to the feature ;)

For example,  what I noticed while building example files showing how to use the Instance Scatterer is that for many uses, it is extremely important to be able to specify an elevation offset for the placement object; This offset lets you control what parts of the object will be considered above the destination terrain level, and what is below. Without offset, the instance scatterer could either build up or cut away terrain, but not both at once. In the crater example above, we’re building up the crater rim but also carving away the inside impact area. You can do that now; but a week ago, not so.

That’s it for now. just wanted to write to show that things are still happening!

Cheers,

Stephen

mimic5

The Mimic Device

Time for a sneak peek at another new WM 2.9 device: The Mimic Device.

This has been in R&D stasis for at least the last six months; I finally got a chance to finish it up this week, and I am confident in its polish level enough to slate it for inclusion in this upcoming dev channel release.

What is the Mimic Device?

The Mimic device allows you to reshape a terrain to mimic the elevation profile curve of a model reference terrain.

What does that mean? A few images might make it more clear. Let’s say we have a basic perlin noise terrain:

Basic Perlin terrain
Basic Perlin terrain

And we have another terrain as a model. (Edit: I’ve updated this example to be more obvious!)

Highly modified Perlin Noise that we want to match
Highly modified Perlin Noise that we want to match

Quite different from basic Perlin Noise! Nonetheless, the mimic device transforms the input terrain shown in the first image using the above as a reference… into this:

Mimic Output
Mimic Output

You can see that it follows the same general shape as the original input, but it matches the elevation range, and more importantly, matches the elevation profile of the model input! Note that the Mimic device cannot introduce detail that wasn’t there to start with –But it does make what is there look the same.

Here’s another example. Let’s say our model terrain was terraced (in this case, by adding a Terrace device to a Voronoi:

mimic6

The terraced voronoi looks like this:

mimic4

 

While the input terrain looks like this:

mimic1

And sure enough, the terraces are replicated on the mimic!

mimic5

Hopefully you can start to see how how powerful this ability could be…

The immediate use cases that I developed the device for include:

  • Matching disparate terrain types together so that they  “fit” and transition to each other better.
  • Shaping terrain by exemplar (For example, provide a reference terrain from a DEM that has the kind of elevation profile character you want).
  • Making changes and modifications to an existing terrain that have to fit its characteristic shape.

An interesting side effect of the Mimic is that it can actually be used to “erase” changes made to a source terrain, as long as those changes don’t reverse the profile curve of the terrain. For example: In the above example world, if we moved the terrace device from the Voronoi chain to the Perlin Noise chain,  we could take that resulting terraced perlin noise input and actually remove the terraces from it by using the smooth-sided terrain as a mimic reference!

Removing Terraces Example
Removing Terraces Example
mimic7
The input terrain to the Mimic
mimic8
The output of the mimic: We’ve erased the terraces!

This surprising power works because the Terrace device doesn’t create completely flat ledges, and so can be reversed. If that was not the case (for example, if you were trying to reverse the terraces from an 8bit input file) then the Mimic would not be effective as-is — you would have to blur the input terrain first.

Of course, nothing is perfect. What can’t the Mimic do?

  • The mimic device cannot replace or insert surface detail. For example, providing an eroded model reference will not introduce erosion gullies to the target terrain.
  • The mimic device doesn’t play especially well with Tiled Builds. It will be flagged as such in the UI (with a !). With high blending percentages it should be acceptable however.

Balanced against those relatively few caveats, however, is a lot of utility.

Excited?

I am.

Latest Developments

Hi folks,

It’s been a little quiet here, so let me shine some light on what’s cooking at WM!

The next Dev Channel release contains major fundamental changes to the underlying device infrastructure in WM. Those changes started with multiple-res support…. but it wouldn’t be a bad idea to start giving some tours of the new features that will be in the next dev build.

What are they?

Localspace Support

A whole previous blog post deals with localspace, but in a nutshell this allows you to create “local” packets that are not referenced to the world.

Why is this useful?

Read the previous post for more details, but this really enabled powerful (and completely new!) workflows for WM. Things like creating hero volcanos or craters in isolation and then splattering them across your terrain randomly; or creating textures for your terrain; etc etc.

Lots more will be written about this as it comes nearer. In my playing with it, it really is a game changer, opening up novel workflows in World Machine for creating terrains in a way that you really couldn’t before.

But besides localspace, what else has been added?

User Interface Improvements

 

Since the UI needed lots more indicators to work well with Localspace, I’ve been taking the opportunity to fix a few UI issues that have been bothering me…

Groups can set resolution/space

It can be a bit laborious to set resolutions on each relevant device by hand. Now you can set them by group:

group_res

 

This ability also works well with the next item…

Hierarchical Grouping

Groups are a World Machine feature that I’m really happy with; they help you organize and make sense of your node world. I’ve seen a number of world files “in the wild” that have tried to use hierarchies of groups; unfortunately in WM 2.x it doesn’t work very well. Groups would double-move devices, not move their children, and all kinds of other headaches.

This has been corrected:

group_hierarchGroups now behave as you would expect when nested; moving a group moves its nested groups, etc. In general I’ve just given a second pass through group behavior to make it less likely to mess things up.

More Context Menus

The right-click behavior in the device view has been changed. Now r-click brings up a context menu for whatever you have selected; you can still flip views with the first menu entry, but you can also do other things like quickly add a device, access overrides, etc etc. I find this very useful!

rclick_menu

“Secret” New Devices…

They are here. They are secret for now ;)

Update: I’ve never been very good with secrets. Here’s a teaser image of the Scatter device distributing a simple localspace heightfield as a random fractal:

 sneakpeek_objchatter

 

Dev and Release: An Unexpected Tragedy

 Forwards and Backwards Compatibility

The Dev and Release branches have brought about a complication that in retrospect should not have been unexpected.

World files and macros created in the dev branch are pretty much incompatible with the release branch.

This is not good. It fragments the potential users of a macro or TMD file; Dev Branch users are unable to share any of their TMDs with normal users. We’ve already been seeing these situations crop up on the forums; what’s worse is that WM doesn’t give very good error messages when this happens, so the user doesn’t know what’s wrong — just that WM is having some kind of error for no good reason. frustrating.

This is the crowning annoyance of a longstanding problem — WM’s file-format is quite brittle. Missing a device? Can’t load the world. Saved from a new version? Can’t load it. And sometimes “can’t load it” actually means “stomp on memory and crash the program.”

I’ve been working on a solution to this problem this week,, and I’m happy to say that the solution is, in fact, already complete.

Dance of the Tags (Implementation Details)

The easiest (and most logical) way to solve this problem is to move from a versioned binary blob file to a tag-based file format, such as used by tiff or xml in text. The new file format is still a binary format, but adopts a tag-based structure that will be a bit familiar to anyone who has done html or xml document model work. Key to WM’s new file IO tags are that each tag is a hierarchical container; it contains a size field allowing us to skip to the end of the tag if we can’t parse it. WM’s old file format had absolutely no way to recover from this kind of lack of knowledge.

Anyways. What does this change buy us?

The Dev and Release builds will be able to open each others’ files and macros

Obviously any specific new features won’t work, but files that don’t use any new features will be 100% compatible with the old version.

‘Old’ World Machine versions will be able to open future versions successfully

Once this is in place, even if you are using an older version that can’t understand some new functionality you will be able to load the file. Of course, you’ll receive a warning that you’re missing some things…

Missing device plugins will never cause the world to fail to load

Instead of failing to load the world file or just plain crashing, a placeholder “Unknown Device” will be created and wired into the network.

missing1
Missing devices are marked in the workview

Of course, it won’t actually do anything and your world probably won’t work without the missing devices. But…

You may even be able to ROUND TRIP between different versions with most information intact

Devices and maybe features loaded that are not known to the current version will be kept and re-saved to disk; so that you should be able to successfully:

  1. Open a “too new” file that uses some new features. You can’t use em, but you are able to make some other changes anyways and…
  2. Save the changes to disk from your older edition. Then
  3. Open your changed file in the newer version … and have all of its original “too new to work” data still there!
Re-opened in a version with the missing plugin
Re-opened in a version with the missing plugin

That last point is really quite rare — and hard to pull off! Especially when devices, etc change their parameters and functionality. I can’t guarantee that this will always be possible for all features, but at least with current anticipated changes it’s looking like it will be.

The new file format will roll out to the Dev Build first for testing; once it’s been tested for a bit, after that it will also be incorporated into the Release build, at which point all of the above will become possible.

PS:

There’s lots of other exciting new changes afoot.

Some of them are in R&D stage so I’m not writing much about them for now, others (like rolling out localspace support to the dev build) will be coming in the next month!

What’s new on the Dev Branch

Hi folks! Lots of things are in flight right now on the dev branch — time for a progress report.

One of the things that goes hand-in-hand with the previous library and multiresolution support is the ability to create things in localspace. What does this mean?

Localspace

Localspace is a mode switch for your generators to allow you to create “free floating” individual textures, hero terrain elements, etc. These items are not located anywhere in the world; they don’t have a firmly defined scale or location. Think texture assets.

You can feel free to use all of the potentially-problematic devices you want (flipper, displacement, normalize, etc).  As the packet is guaranteed to exist only in isolation, these manipulations are completely OK — there’s no tiles to mess up or anything like it. And you can of course save or load them like any other terrain (the library I/O works great here for banks of textures, etc).

Into the World …

When you want to integrate the localspace items into the world, you use a Placer device to insert the localspace item at a particular scale, to a particular location. And of course, the Placer uses the same sampling framework as introduced previously, so you can use fractal resampling, etc when placing it.

I’ve talked to a number of people who have found the “Object Placer” macro very useful when creating their worlds. This is basically the same idea, but formalized and integrated into WM itself — and consequently more powerful.

… with Tiling

In addition, you can use the Placer to tile the texture across the entire world. Right now there is only the option of regular tiling, but eventually I could see the placer also having an option to randomly scatter the object throughout the world, with optional rotation, etc.

An additional device that will eventually appear to complement the above is a seamless tile making device, which takes any localspace packet and blends the edges so that it will tile seamlessly when inserted into the world. In addition, it’s technically straightforward (though UI-wise somewhat challenging) to allow periodic noise and cellular textures, so that those generators will natively tile.

 

So that’s the plan, how are things progressing? Functionally, we’re almost at 100%. As always, the lagging element is the GUI — everything from displaying what is local and what isn’t, to making it illegal to cross-connect local or worldspace items except through designated devices, etc. There’s a lot to be done in that department yet, and as usual for GUI work, it’s slow going.

In other news…

 

A New Interpolation Technique

Something neat that I ran across earlier today and I thought was work a mention.

Here’s the scenario: You are using a texture atlas or something similar, where each color has an encoded meaning (perhaps RGB(0,255,0) is grass, and RGB(127,64,64) is mud). Or perhaps you’re trying to segment a world into regions.

When re-scaling this texture or placement map, traditional interpolation techniques give very bad outcomes — they introduce new intermediate values that are no longer correct in your texture atlas index, or that don’t map to your region correctly.

What you really want to do is have the output use only the exact same colors as the input. Of the common interpolation methods in broad use, only Nearest Neighbor does this. And it looks terrible.

Here’s a different way of doing it:

Interpolation Techniques
(Click to expand)

From left to right, we have:

  1. The 64×64 Voronoi-colored source texture
  2. #1 resampled to full size using nearest neighbor. Ugh.
  3. #1 resampled using “A New Technique“!

As you can see, we’re using smooth interpolation techniques to find the edge, but then snapping the output color (or height value..) to the nearest (in a least-squares sense) of the existing inputs. This provides a very neat “set-preserving” interpolation that is very useful for texture atlases.

I haven’t decided yet if this interpolation technique will make an appearance in WM; it seems simple but useful,  so it has a strong chance.

 

 

Feature Release Soon

It’s definitely time to write a note to detail how things are progressing. The three major features below (multi-res, resampling options, and library I/O) are on track for a release very soon — I’m targeting end of this week for the the dev branch release.

There were some blocking items in their way that was holding everything up; some of them were code, some administrative, and the last one was figuring out how I wanted to deal with updates and the future development direction of World Machine.

The next post (coming shortly) will detail that and more!

Library Output and Input

The second in this time-life series of World Machine Dev Branch feature posts.

This pair of devices is intended to make it easier to compose multiple-world scenarios in World Machine, where you might create parts of the world in one TMD and put things together in another.

The Library I/O devices allow you to store and easily load multiple data packets.

multiout

Why would you want to do that?

  1. You might want to use World Machine to create a collection of things such as textures, etc that you want to create and use, but that don’t need to have regenerated all the time. In this case using a library helps tremendously.
  2. You might use this as a workaround for the problems that tiled terrains have with extensive context. You can do your large-scale work in one file and then pipe it into a second TMD for final detailing and splitting into tiles.
  3. You might have a situation where a number of world files share a base terrain and a bunch of associated masks, which you want to store instead of recalculating all the time.
  4. The more textures and mask outputs you create, the more trouble it is to configure all your outputs and store them on disk.
  5. The above are the target reasons for adding this feature, but I’m sure there are more workflows that this will help out with!

The name of this feature is still under decision. I’m considering either Multi, Library, or Stage Input|Output. What do you like better?

How it works

Much easier than creating seperate outputs for each datatype! Simply set the multi-output for the number of packets you need to capture, give it a filename, and hook all the inputs up. When you build/export the world, it will write all of them to disk in a single file.

The filetype is a new World Machine output file format. World Machine has not had a “native” output filetype in its history; I suppose this is actually the creation of that. 

When you want to load them into another world (or the same world, later on, perhaps…), you simply add a Library input into the world and point it at the file. It will create the appropriate number of outputs that you can then use in the new world.

More interestingly, packets will retain all of their world data, so they will be automatically placed at the appropriate coordinates in the world. Like the other file input devices, you will be able to choose the sampling method used on the data.

Why is this feature being added?

Mostly because it’s a simple feature addition that yields large dividends. Particularly when combined with the multi-resolution support described previously, and at least one more new feature that will be detailed in another upcoming post. For things like applying textures to the terrain, being able to load in a library file is a major help.

What’s left to do?

This device pair is about 95% done. The basic functionality is up and running. The only major bits left to do are some UI work and thorough testing!

 

Multiresolution Progress Update

A lot of my focus has been on bugfix/development issues lately, but support for Multires (and some other goodies) is coming along steadily.

Things are mostly complete — just lots of edge cases to fish out and fix where WM previously made some assumptions that the output resolution would always be the specified world resolution.

Here’s an example of a modified starting world, where the terrain has been downsized to quarter size (129×129), and the texture has been upsized 4x (2049) with fractal noise added. The Overlay View works seamlessly with these differently-sized inputs:

Multiresolution World

Note how the lighting and geometry is very low-res and smooth, while the texture is still high res. This is definitely not the worlds’ best example scene, but it shows generally how you would work in this environment.

I was afraid that this extra element of control would lead to counter-intuitive editing, but its mostly understandable and controllable. I think this is because of the following rules that WM uses:

  1. Devices operate on whatever resolution their inputs are seamlessly.
  2. If a device has two or more inputs of different resolutions, the smaller one is upscaled to match the larger (unless the device has flagged some of its inputs as allowing mismatched resolutions, like the Overlay View does)
  3. “Resolution Change Points” are marked in the device graph. (Note the grey resolution report under the Clamp and Select Convexity Devices).
  4. You can specify a resolution either as relative to the World setting, or a fixed value. There are good uses for both of these. In particular, if you are trying to create coarse geometry then upsample later in the network, a fixed resolution works much better — its disconcerting to have the geometry resolution change when you change the final output res.

The final bits of work required are making Layout View happy with the changes, and some final UI tweaking.

There are some additional enhancements I wanted to do, like allowing per-group resolution settings rather than just per-device, but that one will hold off a bit until some later changes to groups that makes them a bit more of a full-privledge member of the device world…

 

Lastly, there are some other improvements that will be rolling out at about the same time. One of them is the topic of the next blog post…

 

 

Progress Update

Just a quick update to mention that work is proceeding, although the feature branch is slightly lagging as more time has been put the last few weeks into another bugfix release which will be out shortly.

I’ve also been investigating some intriguing new devices that would provide additional terrain shaping abilities. Too early to talk much about that yet, but if they continue to show promise it might make an appearance sooner than later.

Anyways, what’s in place for the multires support:

Resampling method support has been added to the file input and tiled file input devices, and a generic set-resolution dialog implemented for all devices. What remains UI-wise is deciding on a way to clearly denote the resolution state of different devices in the world. There’s also some issues to track down with multires, particularly in layout view and with regards to macros.

 

News, Ideas, and Random Musings by the author of World Machine