mimic5

The Mimic Device

Time for a sneak peek at another new WM 2.9 device: The Mimic Device.

This has been in R&D stasis for at least the last six months; I finally got a chance to finish it up this week, and I am confident in its polish level enough to slate it for inclusion in this upcoming dev channel release.

What is the Mimic Device?

The Mimic device allows you to reshape a terrain to mimic the elevation profile curve of a model reference terrain.

What does that mean? A few images might make it more clear. Let’s say we have a basic perlin noise terrain:

Basic Perlin terrain

Basic Perlin terrain

And we have another terrain as a model. (Edit: I’ve updated this example to be more obvious!)

Highly modified Perlin Noise that we want to match

Highly modified Perlin Noise that we want to match

Quite different from basic Perlin Noise! Nonetheless, the mimic device transforms the input terrain shown in the first image using the above as a reference… into this:

Mimic Output

Mimic Output

You can see that it follows the same general shape as the original input, but it matches the elevation range, and more importantly, matches the elevation profile of the model input! Note that the Mimic device cannot introduce detail that wasn’t there to start with –But it does make what is there look the same.

Here’s another example. Let’s say our model terrain was terraced (in this case, by adding a Terrace device to a Voronoi:

mimic6

The terraced voronoi looks like this:

mimic4

 

While the input terrain looks like this:

mimic1

And sure enough, the terraces are replicated on the mimic!

mimic5

Hopefully you can start to see how how powerful this ability could be…

The immediate use cases that I developed the device for include:

  • Matching disparate terrain types together so that they  ”fit” and transition to each other better.
  • Shaping terrain by exemplar (For example, provide a reference terrain from a DEM that has the kind of elevation profile character you want).
  • Making changes and modifications to an existing terrain that have to fit its characteristic shape.

An interesting side effect of the Mimic is that it can actually be used to “erase” changes made to a source terrain, as long as those changes don’t reverse the profile curve of the terrain. For example: In the above example world, if we moved the terrace device from the Voronoi chain to the Perlin Noise chain,  we could take that resulting terraced perlin noise input and actually remove the terraces from it by using the smooth-sided terrain as a mimic reference!

Removing Terraces Example

Removing Terraces Example

mimic7

The input terrain to the Mimic

mimic8

The output of the mimic: We’ve erased the terraces!

This surprising power works because the Terrace device doesn’t create completely flat ledges, and so can be reversed. If that was not the case (for example, if you were trying to reverse the terraces from an 8bit input file) then the Mimic would not be effective as-is — you would have to blur the input terrain first.

Of course, nothing is perfect. What can’t the Mimic do?

  • The mimic device cannot replace or insert surface detail. For example, providing an eroded model reference will not introduce erosion gullies to the target terrain.
  • The mimic device doesn’t play especially well with Tiled Builds. It will be flagged as such in the UI (with a !). With high blending percentages it should be acceptable however.

Balanced against those relatively few caveats, however, is a lot of utility.

Excited?

I am.

Latest Developments

Hi folks,

It’s been a little quiet here, so let me shine some light on what’s cooking at WM!

The next Dev Channel release contains major fundamental changes to the underlying device infrastructure in WM. Those changes started with multiple-res support…. but it wouldn’t be a bad idea to start giving some tours of the new features that will be in the next dev build.

What are they?

Localspace Support

A whole previous blog post deals with localspace, but in a nutshell this allows you to create “local” packets that are not referenced to the world.

Why is this useful?

Read the previous post for more details, but this really enabled powerful (and completely new!) workflows for WM. Things like creating hero volcanos or craters in isolation and then splattering them across your terrain randomly; or creating textures for your terrain; etc etc.

Lots more will be written about this as it comes nearer. In my playing with it, it really is a game changer, opening up novel workflows in World Machine for creating terrains in a way that you really couldn’t before.

But besides localspace, what else has been added?

User Interface Improvements

 

Since the UI needed lots more indicators to work well with Localspace, I’ve been taking the opportunity to fix a few UI issues that have been bothering me…

Groups can set resolution/space

It can be a bit laborious to set resolutions on each relevant device by hand. Now you can set them by group:

group_res

 

This ability also works well with the next item…

Hierarchical Grouping

Groups are a World Machine feature that I’m really happy with; they help you organize and make sense of your node world. I’ve seen a number of world files “in the wild” that have tried to use hierarchies of groups; unfortunately in WM 2.x it doesn’t work very well. Groups would double-move devices, not move their children, and all kinds of other headaches.

This has been corrected:

group_hierarchGroups now behave as you would expect when nested; moving a group moves its nested groups, etc. In general I’ve just given a second pass through group behavior to make it less likely to mess things up.

More Context Menus

The right-click behavior in the device view has been changed. Now r-click brings up a context menu for whatever you have selected; you can still flip views with the first menu entry, but you can also do other things like quickly add a device, access overrides, etc etc. I find this very useful!

rclick_menu

“Secret” New Devices…

They are here. They are secret for now ;)

Update: I’ve never been very good with secrets. Here’s a teaser image of the Scatter device distributing a simple localspace heightfield as a random fractal:

 sneakpeek_objchatter

 

Dev and Release: An Unexpected Tragedy

 Forwards and Backwards Compatibility

The Dev and Release branches have brought about a complication that in retrospect should not have been unexpected.

World files and macros created in the dev branch are pretty much incompatible with the release branch.

This is not good. It fragments the potential users of a macro or TMD file; Dev Branch users are unable to share any of their TMDs with normal users. We’ve already been seeing these situations crop up on the forums; what’s worse is that WM doesn’t give very good error messages when this happens, so the user doesn’t know what’s wrong — just that WM is having some kind of error for no good reason. frustrating.

This is the crowning annoyance of a longstanding problem — WM’s file-format is quite brittle. Missing a device? Can’t load the world. Saved from a new version? Can’t load it. And sometimes “can’t load it” actually means “stomp on memory and crash the program.”

I’ve been working on a solution to this problem this week,, and I’m happy to say that the solution is, in fact, already complete.

Dance of the Tags (Implementation Details)

The easiest (and most logical) way to solve this problem is to move from a versioned binary blob file to a tag-based file format, such as used by tiff or xml in text. The new file format is still a binary format, but adopts a tag-based structure that will be a bit familiar to anyone who has done html or xml document model work. Key to WM’s new file IO tags are that each tag is a hierarchical container; it contains a size field allowing us to skip to the end of the tag if we can’t parse it. WM’s old file format had absolutely no way to recover from this kind of lack of knowledge.

Anyways. What does this change buy us?

The Dev and Release builds will be able to open each others’ files and macros

Obviously any specific new features won’t work, but files that don’t use any new features will be 100% compatible with the old version.

‘Old’ World Machine versions will be able to open future versions successfully

Once this is in place, even if you are using an older version that can’t understand some new functionality you will be able to load the file. Of course, you’ll receive a warning that you’re missing some things…

Missing device plugins will never cause the world to fail to load

Instead of failing to load the world file or just plain crashing, a placeholder “Unknown Device” will be created and wired into the network.

missing1

Missing devices are marked in the workview

Of course, it won’t actually do anything and your world probably won’t work without the missing devices. But…

You may even be able to ROUND TRIP between different versions with most information intact

Devices and maybe features loaded that are not known to the current version will be kept and re-saved to disk; so that you should be able to successfully:

  1. Open a “too new” file that uses some new features. You can’t use em, but you are able to make some other changes anyways and…
  2. Save the changes to disk from your older edition. Then
  3. Open your changed file in the newer version … and have all of its original “too new to work” data still there!
Re-opened in a version with the missing plugin

Re-opened in a version with the missing plugin

That last point is really quite rare — and hard to pull off! Especially when devices, etc change their parameters and functionality. I can’t guarantee that this will always be possible for all features, but at least with current anticipated changes it’s looking like it will be.

The new file format will roll out to the Dev Build first for testing; once it’s been tested for a bit, after that it will also be incorporated into the Release build, at which point all of the above will become possible.

PS:

There’s lots of other exciting new changes afoot.

Some of them are in R&D stage so I’m not writing much about them for now, others (like rolling out localspace support to the dev build) will be coming in the next month!

What’s new on the Dev Branch

Hi folks! Lots of things are in flight right now on the dev branch — time for a progress report.

One of the things that goes hand-in-hand with the previous library and multiresolution support is the ability to create things in localspace. What does this mean?

Localspace

Localspace is a mode switch for your generators to allow you to create “free floating” individual textures, hero terrain elements, etc. These items are not located anywhere in the world; they don’t have a firmly defined scale or location. Think texture assets.

You can feel free to use all of the potentially-problematic devices you want (flipper, displacement, normalize, etc).  As the packet is guaranteed to exist only in isolation, these manipulations are completely OK — there’s no tiles to mess up or anything like it. And you can of course save or load them like any other terrain (the library I/O works great here for banks of textures, etc).

Into the World …

When you want to integrate the localspace items into the world, you use a Placer device to insert the localspace item at a particular scale, to a particular location. And of course, the Placer uses the same sampling framework as introduced previously, so you can use fractal resampling, etc when placing it.

I’ve talked to a number of people who have found the “Object Placer” macro very useful when creating their worlds. This is basically the same idea, but formalized and integrated into WM itself — and consequently more powerful.

… with Tiling

In addition, you can use the Placer to tile the texture across the entire world. Right now there is only the option of regular tiling, but eventually I could see the placer also having an option to randomly scatter the object throughout the world, with optional rotation, etc.

An additional device that will eventually appear to complement the above is a seamless tile making device, which takes any localspace packet and blends the edges so that it will tile seamlessly when inserted into the world. In addition, it’s technically straightforward (though UI-wise somewhat challenging) to allow periodic noise and cellular textures, so that those generators will natively tile.

 

So that’s the plan, how are things progressing? Functionally, we’re almost at 100%. As always, the lagging element is the GUI — everything from displaying what is local and what isn’t, to making it illegal to cross-connect local or worldspace items except through designated devices, etc. There’s a lot to be done in that department yet, and as usual for GUI work, it’s slow going.

In other news…

 

A New Interpolation Technique

Something neat that I ran across earlier today and I thought was work a mention.

Here’s the scenario: You are using a texture atlas or something similar, where each color has an encoded meaning (perhaps RGB(0,255,0) is grass, and RGB(127,64,64) is mud). Or perhaps you’re trying to segment a world into regions.

When re-scaling this texture or placement map, traditional interpolation techniques give very bad outcomes — they introduce new intermediate values that are no longer correct in your texture atlas index, or that don’t map to your region correctly.

What you really want to do is have the output use only the exact same colors as the input. Of the common interpolation methods in broad use, only Nearest Neighbor does this. And it looks terrible.

Here’s a different way of doing it:

Interpolation Techniques

(Click to expand)

From left to right, we have:

  1. The 64×64 Voronoi-colored source texture
  2. #1 resampled to full size using nearest neighbor. Ugh.
  3. #1 resampled using “A New Technique“!

As you can see, we’re using smooth interpolation techniques to find the edge, but then snapping the output color (or height value..) to the nearest (in a least-squares sense) of the existing inputs. This provides a very neat “set-preserving” interpolation that is very useful for texture atlases.

I haven’t decided yet if this interpolation technique will make an appearance in WM; it seems simple but useful,  so it has a strong chance.

 

 

Feature Release Soon

It’s definitely time to write a note to detail how things are progressing. The three major features below (multi-res, resampling options, and library I/O) are on track for a release very soon — I’m targeting end of this week for the the dev branch release.

There were some blocking items in their way that was holding everything up; some of them were code, some administrative, and the last one was figuring out how I wanted to deal with updates and the future development direction of World Machine.

The next post (coming shortly) will detail that and more!

Library Output and Input

The second in this time-life series of World Machine Dev Branch feature posts.

This pair of devices is intended to make it easier to compose multiple-world scenarios in World Machine, where you might create parts of the world in one TMD and put things together in another.

The Library I/O devices allow you to store and easily load multiple data packets.

multiout

Why would you want to do that?

  1. You might want to use World Machine to create a collection of things such as textures, etc that you want to create and use, but that don’t need to have regenerated all the time. In this case using a library helps tremendously.
  2. You might use this as a workaround for the problems that tiled terrains have with extensive context. You can do your large-scale work in one file and then pipe it into a second TMD for final detailing and splitting into tiles.
  3. You might have a situation where a number of world files share a base terrain and a bunch of associated masks, which you want to store instead of recalculating all the time.
  4. The more textures and mask outputs you create, the more trouble it is to configure all your outputs and store them on disk.
  5. The above are the target reasons for adding this feature, but I’m sure there are more workflows that this will help out with!

The name of this feature is still under decision. I’m considering either Multi, Library, or Stage Input|Output. What do you like better?

How it works

Much easier than creating seperate outputs for each datatype! Simply set the multi-output for the number of packets you need to capture, give it a filename, and hook all the inputs up. When you build/export the world, it will write all of them to disk in a single file.

The filetype is a new World Machine output file format. World Machine has not had a “native” output filetype in its history; I suppose this is actually the creation of that. 

When you want to load them into another world (or the same world, later on, perhaps…), you simply add a Library input into the world and point it at the file. It will create the appropriate number of outputs that you can then use in the new world.

More interestingly, packets will retain all of their world data, so they will be automatically placed at the appropriate coordinates in the world. Like the other file input devices, you will be able to choose the sampling method used on the data.

Why is this feature being added?

Mostly because it’s a simple feature addition that yields large dividends. Particularly when combined with the multi-resolution support described previously, and at least one more new feature that will be detailed in another upcoming post. For things like applying textures to the terrain, being able to load in a library file is a major help.

What’s left to do?

This device pair is about 95% done. The basic functionality is up and running. The only major bits left to do are some UI work and thorough testing!

 

Multiresolution Progress Update

A lot of my focus has been on bugfix/development issues lately, but support for Multires (and some other goodies) is coming along steadily.

Things are mostly complete — just lots of edge cases to fish out and fix where WM previously made some assumptions that the output resolution would always be the specified world resolution.

Here’s an example of a modified starting world, where the terrain has been downsized to quarter size (129×129), and the texture has been upsized 4x (2049) with fractal noise added. The Overlay View works seamlessly with these differently-sized inputs:

Multiresolution World

Note how the lighting and geometry is very low-res and smooth, while the texture is still high res. This is definitely not the worlds’ best example scene, but it shows generally how you would work in this environment.

I was afraid that this extra element of control would lead to counter-intuitive editing, but its mostly understandable and controllable. I think this is because of the following rules that WM uses:

  1. Devices operate on whatever resolution their inputs are seamlessly.
  2. If a device has two or more inputs of different resolutions, the smaller one is upscaled to match the larger (unless the device has flagged some of its inputs as allowing mismatched resolutions, like the Overlay View does)
  3. “Resolution Change Points” are marked in the device graph. (Note the grey resolution report under the Clamp and Select Convexity Devices).
  4. You can specify a resolution either as relative to the World setting, or a fixed value. There are good uses for both of these. In particular, if you are trying to create coarse geometry then upsample later in the network, a fixed resolution works much better — its disconcerting to have the geometry resolution change when you change the final output res.

The final bits of work required are making Layout View happy with the changes, and some final UI tweaking.

There are some additional enhancements I wanted to do, like allowing per-group resolution settings rather than just per-device, but that one will hold off a bit until some later changes to groups that makes them a bit more of a full-privledge member of the device world…

 

Lastly, there are some other improvements that will be rolling out at about the same time. One of them is the topic of the next blog post…

 

 

Progress Update

Just a quick update to mention that work is proceeding, although the feature branch is slightly lagging as more time has been put the last few weeks into another bugfix release which will be out shortly.

I’ve also been investigating some intriguing new devices that would provide additional terrain shaping abilities. Too early to talk much about that yet, but if they continue to show promise it might make an appearance sooner than later.

Anyways, what’s in place for the multires support:

Resampling method support has been added to the file input and tiled file input devices, and a generic set-resolution dialog implemented for all devices. What remains UI-wise is deciding on a way to clearly denote the resolution state of different devices in the world. There’s also some issues to track down with multires, particularly in layout view and with regards to macros.

 

Feature Branch #1

I mentioned the other day that I would be posting more information about development direction soon. That post is still yet to come, but I thought I would start talking about what kind of features are being worked on for inclusion within World Machine in the relatively immediate future.

So here’s the first in the series!

Different Resolution Support

The current feature-in-development is the ability to have different sections of the device graph operate at different resolutions in the same world file. How does it work?

The basic rule of thumb is that you can set any device to operate at a fraction or multiple of the resolution set in the world file; That section of the graph will then flow at the new resolution until encountering device(s) that change the resolution again. If two different resolutions meet at a device, the lower-res one is upscaled to match the higher one. This allows existing devices to more or less ‘just work’ without requiring me to change anything.

Why would you want this?

This is very powerful and handy; you can do anything ranging from simply setting the resolution of your outputs independently (for example, for your heights vs normal maps vs bitmaps), all the way to having lots of different effects that are processed at different resolutions. For example, your high-resolution builds could run with most of the world-building done at a more modest resolution, then upsampling right before final detailing and output, saving lots of memory and time.

Resampling

Of course, having different resolutions all over the place requires improved support for re-sampling the resolutions of heightfields when needed. WM has always just done bilinear interpolation when required, which is the bare minimum but leaves a lot to be desired. I’ve just finished adding support for new resampling abilities, as seen in the graphic below.

Upsampling from a 64×64 source to a 512×512 heightfield in WM:

webimages

These will be the resampling filters implemented in the next feature release. Let’s go through them:

  1. Nearest Neighbor: This is not too useful for heightfields except if you’re going for a minecraft-look. However, this can be very useful for bitmaps as a method when dealing with material maps/atlases where each color actually is a code for a particular type of coverage; In this case blending doesn’t make any sense.
  2. Bilinear is what we’re used to. When applied to heightfields lots of creasing artifacts are in evidence, although in some cases this sharp look is very useful.
  3. Bicubic: (Soft) uses an approximating B-spline to create a smooth, C2 continuous surface. This is a great choice when you need to produce a very smooth output terrain, and will probably be the new default choice.
  4. Bicubic (Tuneable): This choice provides you with a “sharpness” tuning parameter, which ranges from the same as Cubic-Soft to quite sharp. This is immensely useful for images as well as heightmaps… As a ‘season-to-taste’ parameter, the correct value is according to your artistic judgement. For fellow math geeks, these are the Mitchell-Netravali spline family.
  5. Fractal: This adds fractal detail to the interpolated surface adding rougher details in high-variance areas. This is a simple and basic way to add some detail to a low-resolution input, and is easier than manually setting up the equivalent network in World Machine itself. It is tuned with a single roughness parameter. High roughness is also useful for things like material maps, which tend to get too blurry when upsampled from low resolutions. I’m still fine-tuning the algorithm here for best results, but the intention is to be a fast’n'easy solution, not the ultimate fractal upsizer; The Advanced Perlin device is the best bet for that, especially with some tweaks being made to improve the ability to quickly setup that device for detail-adding.
  6. I can’t resist a final image of the fractal resample with some erosion and texturing added!

I also experimented with some other ‘designer’ resampling kernels like Lanczos, but decided that they didn’t bring anything to the table that wasn’t already covered by the above.

All of these will be presented as options both for resampling within WM, and when importing from a regular or tiled file input.

What’s left to do

A fair bit:

The major work remaining on resolution/resampling support lies with the User Interface. The workview should mark all devices that are at non-default resolutions so that you can at-a-glance see where your custom settings are. Ideally the controls for setting the resolution (as well as a few other things as I will detail in upcoming posts) would reside directly within the dialog box for the device, but due to framework issues this is a mammoth task what with all of the custom device dialogs in WM. Instead, these are currently accessed through the right click context menu.

In addition, all of the various views and features (layout view / tiled builds / etc) that expected consistent resolutions will need to be modified to work within the new system.

Finally, the file inputs need to be modified to provide the additional options.

Final Thoughts

The update containing this ability should become available within the next month. I picked this as the first new post-2.3 feature to add as it is both highly requested and doesn’t require extensive R&D to implement. Have other things you’d like to see? Post about them on the forums!

 

The Taste of Progress

Yesterday I released 2.3.3, which marks the third revision to WM since the 2.3 release… which itself occurred only 2 months ago. 

I’m very satisfied with the new pace of progress! There are many reasons for it, a few of which I will detail below. But more than anything else, this characterizes how I really would like to develop World Machine — relatively frequent releases that bring new abilities and fix issues.

I’ve always been frustrated by the lengthy and slow release cycle that WM suffered from before. It’s not good for anyone to have the fixes for issues languishing for months or years(!) while the product is still in a beta phase before release. I think that 2.3 marked the end of the traditional development model for WM — more details to come.

The next section talks a little bit about what I’ve been able to change post-2.3 to make development faster. 

Development Details:

What did I change to be able to iterate versions much quicker? A few things:

  1. I changed to Mercurial for source control. The ability to branch and merge effortlessly has allowed me to keep feature and release branches seperate, something that was always far tougher than it should have been using the old style SVN (or nothing!).
  2. I automated much of the build process, so that kicking out a new version to the website takes maybe a half hour rather than most of a day.

Not a lot of differences, but enough to remove the friction from the process and make it easy to get changes out rapidly.

News, Ideas, and Random Musings by the author of World Machine