News | Forum | People | FAQ | Links | Search | Register | Log in
Multithreaded VIS For Windows
WVis is a modified version of Bengt Jardrup�s VIS tool. It�s the exact same program as the one you can get here�

http://user.tninet.se/~xir870k/

�except that it has multithreading turned on. Basically, you get 1 thread for every core/processor you have on your machine. This will speed up VIS compile times dramatically if you have a machine with multiple cores/processors.

Usage and syntax are exactly the same.

Enjoy!

WVis
http://www.quaketastic.com/upload/files/tools/windows/misc/WVis.zip
Windows Version Be Available? 
When will the windows version be available and not just the OSX version? 
 
Now? :) 
Hehe 
this is indeed a windows compatible version. i've been using it today and as far as i can tell, it works perfectly.

i said it before in GA, but thanks a *lot* for this. it's simple, but incredibly effective. 
Legendary. 
Makes me wish I had bought a quad core instead of a dual core... hmmm... wonder what the prices are like at the moment? 
Haha 
Willem, I was just messing with you.

I saw Willem + the word Windows and was like "no way!" 
 
wonder what the prices are like at the moment
is it sad i went to check prices too? ^_^; 
Yeah 
Nice work, thanks. Is it possible to manually set the number of threads to be used (i.e. -threads switch)?
I'll now have to buy a quad core too, just to get this map done! 
Now 
Who is gonna donate that time travel to send Willem and this VIS 7-8 years backwards in time? 
 
Is it possible to manually set the number of threads to be used

yes, CTRL+ALT+DELETE, select process, right click, Set Affinity.
check any core you want to use.
obviously, you meant from the program itself, but that's all i got. :P

incidentally:
original vis: 5h 43m
multicore vis: 3h 37m
additionally, the multicore vis today was while the computer was in use. i used below average priority, so i would only assume that the performance was lower than can be expected when running on a completely idle system.

that is, i believe, a 58% increase. if that remains consistent for all vis times (and the logic of my foolish brain says it should?), this is pretty damn awesome. 
It Means.. 
.. that CDA vis runtime could have been decreased from 1254 hours down to 725 hours... indeed, it is a serious enhancement ! 
Thanks Willem! 
I wanted one of these for ages! Anyone tried it on an i7 yet or a Core 2 Quad / Extreme? 
 
"Nice work, thanks. Is it possible to manually set the number of threads to be used (i.e. -threads switch)? "

It's possible and I guess I could allow it for people who want to experiment. In reality though, there's not a lot of use to setting more threads than you have cores/processors since then they'll have to start sharing time slices. It's best to have a 1-to-1 match up, IMO.

But i'll see about supporting the "-threads #" option for those who want to play around... 
 
Incidentally, I thought something was wrong with it when I was testing it at work because it was VIS'ing "Evil Exhumed" in 2 minutes and I was sure it was taking upwards of 20 when I was working on it originally.

I guess that's what happens with 8 cores. :P 
 
I was thinking more about setting it to a lower number if other programs need to be run at the same time.

Shouldn't you include the source and txt?

8 cores - what would I give for such a machine right now... :o 
Negke 
So, er, AguirRe's vis has an autosave function built in (i think it saves every minute by default or something).
So if WVis is the same only with support for more than one processor core, couldnt you stop your current build and then restart it with WVis?

Hmm, I wonder if that would work. If not, then you loose all of those hours, if so then you can cut our remaining time considerably. (if you have a multi-core processor that is) 
Hey 
Didn't inertia had access to some wicked cluster ages ago? 
 
Ricky

In theory, I would say you're right. That sounds like it would work. However, I don't want to be the one responsible for him losing 2 weeks of processing time or whatever it's been so I'm staying out of it. :P 
 
OK, I replaced the download with a new version.

This version contains a new argument:

-threads [n]

This lets you override the number of threads that will be used. This will let you do things like only tie up 3 of your 4 cores. Or try twice as many threads and see if it makes any difference. 
No Warren-ty, Eh? 
 
 
Har!

And for those interested, the source code:

http://www.quaketastic.com/upload/files/tools/windows/misc/WVis_Src.zip

This is pretty messy and includes a bunch of files not related to WVis but this was just a quick and dirty project anyway. "vis.sln" is the Microsoft solution file that I was using to build from. Most of the grunt work is in "vis.c". 
 
aguirRe's VIS does save after each portal I think. So you could simply make a backup of the files and play around.

I want an i7 so badly. :] 
Well I Could See How It Would Be Worth The Gamble, Even If He 
only has 2 cores!

What kind of processor do you have, Negke?

You see if yesterday the estimated elapsed time was only 28%, I would guess (at the speed he was going) that it would still only read like 30 or something, so if you think that it will only get slower, if he started again now going at twice the speed, there is enough remaining time for it to still end sooner with two cores then to keep going from this point with 1......

(gasp!) 
Benchmarks 
OK, so I messed around a little on this 8 core machine.

"Evil Exhumed"

1 thread - 14 minutes
2 threads - 7 minutes
3 threads - 5 minutes
4 threads - 4 minutes
5 threads - 3 minutes
6 threads - 2 minutes
7 threads - 2 minutes
8 threads - 2 minutes

Nice! There are obviously some rounding issues here in the time calculations provided by the VIS tool, but there is definitely a point of diminishing returns... 
 
nicec work Willem

petty that i just have a AMD 3200 :\

I might try this in a near future :) 2015 maby. 
Neg 
dunno if this is indicative of anything, as it wasn't an explicit test, but WVis was able to recognize a bsp file had already had it's Base Vis calculated from a prior run through with Vis's original .vis file.

you could just try copying the .bsp file and .vis file to a new location and try to run with WVis without interrupting the original's process. just set priority to low while you test.

also, you can set thread priority in aguire's (and WVis) with -priority. 0 = below average, 1 = average, 2 = above average. if you are concerned about bogging down your system. 
 
Thanks for the source.

I'll enable this in Linux bjptools, too, unless someone else does it first. Although I reckon it's done a little differently in *nix.

The base is already there. 
 
It probably is. You might want to grab the source code for the Mac version of the tools instead and use that as a reference. It's much closer to *nix than this Windows version will be.

I've lost the link for that source code and I don't have access to my FTP from work ... I'll try to remember to post the link though. I'll just post a new archive to Quaketastic when I get a chance.

The Windows stuff is fairly far removed. 
Cool 
Anyone tried it with wine ? 
Question 
Is Wine taking profit from CPUs as windows does? I mean: as Windows uses all the available CPU cores (4 for a Quad, 2 for a Dual), is Wine having the same ability to take profit of the multiple CPU core from a Unix/Linux system (e.g Sparc processor, etc...)? 
Of Course 
 
Seems To Run Good :) 
wine 1.1.26
threads 2

Full: 72.03%, Elapsed: 1h 9m, Left: 1h 58m, Total: 3h 8m, 37%

CPU usage 90% 
 
top shows every cpu you have as 100% I think. So if you have a quad, that would be 400% overall. I have 2 cpus and top goes over 100% sometimes for me. 
Cluster 
Anyone want to make vis use MPI (Message Passing Interface)? I can run it on a 100+ node cluster if y'all want. This would also pave the way for people to cooperatively use vis over the internets -- are we communists yet?

What's the the speedup per additional processor? 
Clarification 
I mean, more than just one data point (thanks Willem!). A run that with one processor takes an hour, is better than one that is already short & reasonable (e.g. Evil Exhumed). 
 
How complicated is it to make a distributed app like that? Seems like an undertaking... 
Another Lunchtime? 
 
Willem 
Just put in the MPI hooks in the source, and compile against the MPI libs. 
 
I'm not familiar with how any of that works at the moment ... so if I add the hooks to VIS this will all work magically? Doesn't there need to be something running on a server somewhere? 
Alternate Approach 
If you know how to split up the map into independent parts, you could apply the map reduce paradigm to the vis problem.

Re: MPI inclusion: magic doesn't exist. 
Inertia 
So you're one of the ones (like me) who just thought that the little girl in Pan's Labyrinth was schizophrenic? 
 
I disagree about magic. 
On Magic 
Yesterday I defined "faith" for a buddy:

Faith: Belief in irrational hypotheses. 
 
So I'm right then? It's a major undertaking? 
MPI 
I don't know. It's worth exploring. As I said, if you can restructure it to use MPI, you can make the leap to making it work over the internet.

Or go the whole hog and figure out how to separate the map into mutually-independent parts for vis. 
Or 
Black-Dog or Moaltz or Bambuz can port the thing to Haskell! 
 
It's something to think about but I wouldn't hold your breath. :) 
 
The source code is available if someone wants to get THAT ambitious, BTW:

http://www.quaketastic.com/upload/files/tools/windows/misc/WVis_Src.zip 
 
actually it would be nice to see a completely modern rethinking of vis and light. For example it should be possible to use your GPU to render lightmaps for you. As for vis, i'd like to see alternate implementations, maybe even tradeoffs such as 90% faster for 10% higher polycount.

The problem with this super long vis times is that the result is nobody vises their maps during development; they only do it once at the end, and if there's a problem, they're not willing to fix it and waste another 2 months for another vis.

I don't have any fully-developed ideas for vis, but i could imagine some sort of raycasting approach that sends rays out in a coarse grid, then subdivides that grid as needed. Or some sort of CSG method that starts with the half-space on the far side of a portal, then flows recursively through portals clipping that half-space as you recurse through each next portal. Or maybe a system where you use the standard algorithm but have a much smaller set of portals to work with, either though automatic selection of "important" portals (which would be awesome but i can't quite see how to do it), or through mapper-defined "portal brushes". 
 
Wait, this is the VIS for Quake(1), right? This thing was initially developed more than 13 years ago and it is still _this_ slow? I've never done anything for Q1, but a VIS compile of a (cleanly build) Q3 map typically took less than 2 minutes.

The Q3 VIS, as I understand it, recursively subdivides a map into a tree of convex shapes and then calculates which leafs can "see" each other. Does the Q1 VIS do any more than that, or why exactly does it take this long to compile a map? Maybe the algorithm of the Q1 VIS is just very naive (in which case one could probably adopt some code of the Q3 VIS)? 
 
I recall reading back in the day something Tokay or Romero said I think, that id's tools allowed you to build the maps incrementally. Implying that you could create one section, vis it, test it, build another part, vis that then add it to the already vised part. 
Chaos: 
If you made a q3 map with 7000 structural brushes, it might take a while. Detail brushes help a lot. 
How About An Editor With On-the-fly Compile In Other Thread(s)? 
Should be lots of idle cpu-time while mapping although the problem is of course working with a changing data set and figuring out how to recompute as little as possible when the geometry changes... 
Necros, Ricky, William 
That's what I do basically. Kept a copy of the bjp state file and resumed the process with Wvis. Both tools can be used interchangably. We'll see how it turns out in the end, though. I hope that c_chains discrepancy doesn't have any severe consequences. 
 
metlslime

I think having level designers place hand created portal sheets would be interesting. Have a texture called "PORTAL" or something that gets ignored during QBSP and LIGHT but is used to define the portal planes during VIS? I dunno. I don't know nearly enough about the internals of VIS to do this but it IS interesting. 
 
Not to make anyone cry, but I just got a new machine at work. 24GB of RAM and 16 cores. Good god... 
But Does The New Eight Cores Make Any Difference? 
I guess you can always make everyone jealous by compiling three maps at the same time. 
 
Well, it's for work, so yeah it'll make a difference. For Quake VIS'ing, probably not. 
16 Cores? 
you should sell your vising capabilities. :P 
What Is It Clocked At? 
I mean 4.5Ghz isnt uncommon AFAIK, for your Core 2's and i7's and the new phenoms. I wonder how long it would take to compile CDA, probably like a day or something......... 
 
Intel Xeon
X5560 @ 2.80GHz
2 Processors

So 8 cores per processor... 
Hyperthreading 
The intercoils indicate that your CPU has 4 real cores per processor, which with hyperthreading gives you a perceived 4 more. So, it's not quite 16 cores, maybe approximately 12, if you want to talk about it like that. I'm still jealous, though. 
 
going further off topic, but there will be 6-core Xeons soon (so 12 threads per core).

And about the hand-placed portals, isn't that what hint brushes do?

Relatedly, as I like to point out every time these topics come up, the codebase exists for adding hint, skip, and detail in Q1:

http://quake.chaoticbox.com/downloads/equakeutils.sit 
Grahf: 
i think i've seen you mention it before, but i simply don't believe that detail is possible in the quake1 bsp format. Maybe i should actually follow your link this time and see for myself :) 
Also... 
hint brushes supposedly indicate preferred planes to do major cuts in the BSP tree, from what i understand. And "skip" in quake2 is used on the other faces of a hint brush so all 6 sides don't have to be "hint".

"Skip" in quake1 has come to mean the equivalent of "nodraw but structural and casts shadows" in quake3 terminology.

Also: how do i open a .sit file in windows? 
Well... 
let me back up and say that perhaps detail brushes would be possible if you were willing to totally break software engines. 
Stuff 
.sit is Stuffit, a totally lame and outdated mac-only compression format. Here, this should be easier:
http://www.quaketastic.com/upload/files/tools/mac/misc/eUtils_r10.zip

The codebase is basically the Quest compilers, but enhanced and fixed up. Like, the original Quest utils had their own pointfile format, but pOx reverted it to standard quake style. Some higher limits too (but nowhere near bjp-tools' level), better handling of complex geometry, and probably some other features I am forgetting right now.

Truth time: the detail brushes in the eUtils don't work exactly like Q3 engine detail. I have done tests and they split the BSP tree exactly the same as regular brushes. However, and this is the important part for this discussion, they are ignored by VIS.

Both qbsp and vis have to be aware of the detail brushes, but they definitely work in all engines. I'm no programmer, but I had a look at the relevant parts of the source, and it looked to me like the main change was a new contents type (sky, lava, water, slime...and detail), and all brushes in am_detail entities were given that type. Then vis knows to ignore them, I guess. 
John Carmack On Parallelization 
There are opportunities for network level parallelization with some recoding. Qbsp uses fork level parallelization to utilize up to three processors, and it would be trivial to change to allow it to be run on three computers. Light is embarrassingly parallel, and can be divided arbitrarily. Vis is highly parallel, but it takes some advantage of order of completion in
the serial case, so scaling is not quite linear.


http://www.gamers.org/dEngine/quake/Qbsp/

I had to correct his maddening spelling mistakes. 
A Word Of Caution 
As it turns out, this multithreaded vis isn't safe. The problem lies with the autosave feature:

Each thread processes one portal at a time, that's why the fullvis is sped up on the whole the more threads a machine has. However, as we know, not all portals take the same amount of time - especially towards the end, their processing time increases greatly, some of them can even take days. The autosave option is geared for single-threaded processing, so it doesn't take into account which portals are done and which are still being processed on the other threads when saving the state. Therefore, pausing the process (ctrl+c) will leave these portals (that were being processed but couldn't be finished yet) open, so to speak. This will eventually lead to a situation where vis reaches the end of the processing line, but can't wrap up the entire thing properly because there're still those 'undone' portals in-between. It will then stop with a "portal not done" warning and the state file will be corrupted.

This means the multithreaded vis tool can only be used (more or less) safely if one leaves the process run from start to finish. Maps that take longer and thus have to rely on the autosave function should not be fullvised with this tool. 
 
Oh. OK, that explains why I wasn't able to VIS your map for you then (my resumes kept failing). Damn. 
 
Well, the source code is on Quaketastic so I'll see about making some time to look into it. It shouldn't be too difficult.

The solution, probably, is to turn off the threading when the autosave is ready to go, wait until the last portal finishes, save, and then start them all up again. We'll see... 
Thread Saving 
Ahoi,

aroused by Spirit's coding bounties over at quaddicted.com I thought I might take a look this one.

Honestly I haven't really dived into the code much. However judging by the comments I thought I try some blind thread synchronization upon saving.

Here is my first appalling version:

http://user.cs.tu-berlin.de/~tuna/WVis_thread_test.zip

I'm not even sure how to trigger the issue described. I tried ctrl-c-ing and continuing with the unmodified version and I didn't encounter the "warning" message..

If you have precise instructions how to make it fail please let me know.

Also feedback if the issue has changed or improved with my version is highly appreciated. 
 
Hooray! Thanks man. I took a few shots at it but always came up short. 
Awesome 
Could somebody explain how to properly test this? 
 
I think just get a map that takes a long time to VIS and stop/start it repeatedly. 
 
I tried gmsp3tw and it worked very well. 
 
But can you get a failure with the old version? The old version worked most of the time - until it didn't. :) 
Yes 
I set the -savetime to 10 and interrupted it a lot. Got the error in return at the end. 
Yes 
I set the -savetime to 10 and interrupted it a lot. Got the error in return at the end. 
Tuna 
Release the src, please. 
 
I wanted to wait for more feedback - or at least for no negative results. I will send the patch to Spirit later today or tomorrow when I have access to the machine again. I hope he can review it and decide whether it qualifies for the bounty or not :-) 
Test It With Coag3_negke 
 
Yay For Tuna 
 
 
Call for testing! Please re-download the test version from my location above. Its a newer version which should be more efficient and hopefully also has the bug fixed. 
 
When I use -nosave, I don't get %/time interval updates from the "full" part of the vis process. 
 
Updated the .zip file. I think this bug was also in the original version?

Now there is also the chance that if you have a corrupted .vis file that it gets repaired when it gets loaded. 
Yes 
It indeed loads my corrupted state file and seems to be continuing the process normally. 
Tuna 
Is this the 'idle' or the 'reset' version? 
 
This should be the 'right' version: No thread synchronization is done. Instead undone portals are marked as such instead upon saving.

If no bugs are found this should become the recommended version.

In the end the fix could probably be done by changing one line. But I think its ok to fix some other things while we are at it. 
 
Nice work, tuna! 
I See. 
I'm currently testing it on my coag map. The ultimate goal would be to get all portals done except for the long one, thus making the PVS data as complete as possible. Problem is that I can't tell what's going on as the -verbose display isn't as clear as in single-thread mode. Or dunno.. 
 
Hm. Not sure what the verbose messages are supposed to output. In theory just keep an eye on the CPU usage. When the "Full" Vis step is only using one core you are there.. Might be difficult to spot when you are on a single core CPU. When the task manager displays the threads per process you might keep an eye on them.. 
 
What Is It With Linux Always Screwing Up Line Breaks In Text Files? 
Please include the source with the (final?) build - better than refering to several authors and versions.

As for my vis experiment, it really seems to work as intended. I still need to test the final result in Quake to check for HOMs, but it looks good so far. 
 
Great news. Up to now I think that my link contains the latest version and also includes the patch. I asked Spirit to update his links to avoid people loading the older version.

In general someone who is going to maintain the application should take care of packaging and updating the relevant links.. .. Willem? ;-) 
 
I dunno man, I'm going to be really busy this year with work. I'd rather you just throw the source code and binary (with an updated filename/version number) onto Quaketastic and let the community sort it out. :) It's just an EXE after all...

Put it all in "tools/windows/misc" preferably, to keep it all clean. 
Question 
Could the same multi-threading be added to the light utility, or is that using the GPU? 
 
light utility does not use the GPU but it should; that is one of my wish-list compiler features (still need to work out the math to see if it's even possible. I think it should be but it's all new territory for me.) 
 
Yes, multi-threading can totally be added to the light utility because I added to the one I used on my Mac. It's about as difficult as adding it to VIS - which is not really difficult at all.

Someone get tuna to grab the source code for the most commonly used light.exe and splice it in. :) 
 
WE MISS AGUIRE!!!

Does anybody know if he stills in Q2 codding? 
Trinca 
yeah he is. you can still email him as before, hes just stopped coming here. 
 
I think he left because of travail secret map :\ when he saw negke face :(

he got disappointed with is love�

He saw negke genious is mapping as a sexy big huge German male. 
 
Aye, perhaps open up a bounty for the light multi-threading? ;-)

Anyway one would need to to which source to make patches to.. (although I'm quite busy right now so don't keep up your hopes too high)

On another note: I noticed that Spirit hasn't updated his bounty patch on quaddicted.com with the latest one. Can someone punch him when he is around?

Also I haven't copied my binary to any other location than on my web space. Perhaps someone can save it to some real location as Willem chickened out and I lack the power too. We don't want it to get lost, right? 
Ouch! 
I think I did not have a binary of the latest version and thus postponed posting it and postponed and postponed and ... spiderwebs in my brain. 
Done 
 
> Aye, perhaps open up a bounty for the light multi-threading? ;-)

Ummm - light.exe is already multi-threaded. You do need to use the -threads options to enable it though. "-threads 16" for 16 threads, for example. 
 
Sure you can pass it in - but it doesn't do anything. :) 
Not That Light.exe Really Needed It Anyway 
Besides, the -gate 1 switch in BJP's tool speeds up the process considerably already. 
 
You'd be surprised. I multithreaded my Mac version and even if you cut a minute build time down to 35 seconds, that speeds up iteration so it's totally worth it... 
 
Has anyone added the ability to use visdata (if present) to light.exe yet? I did it for MHColour and it was a HUGE boost, so it would definitely also be worthwhile for general lighting. 
 
I don't think so.

Go go go! That would be awesomely great. :) 
 
how does light work? i thought it was raytracing, so how would vis data speed it up? 
Necros: 
Light basically does a ray-trace from every light to every lightmap pixel on every surface in the map. I think there is already an optimization to reject surfaces that are outside the radius of the light (for linear falloff lights) but for lights with an inverse square falloff, a light theoretically has infinite range, so vis data could be useful to quickly reject polygons a light can't see. 
Oh I See 
i originally just sort of imagined it as blasting out rays and pasting a light value on any walls it hit. i didn't know it did the entire map. o_o 
Why 
do the light tools not use radiosity? 
 
> Has anyone added the ability to use visdata (if present) to light.exe yet? I did it for MHColour and it was a HUGE boost, so it would definitely also be worthwhile for general lighting.

Doesn't bjp light do that already? I seem to remember one that does it. 
Megaman: 
some of them do; for example i think lordhavoc's compilers do it. 
Post A Reply:
Name:
Title:
Body:
message
question
exclamation
idea
flame
noflame
error
skull
beer
moon
pent
rocket
sheep
pacman
pig
cheese
worldcraft
gauntlet
crate
pitfall
pimp
smile
cool
sad
frown
oi
yay
tongue
evil
wink
neutral
q1
q2
q3
ut
hl
cs
doom
dkt
serious
cube
Website copyright © 2002-2017 John Fitzgibbons. All posts are copyright their respective authors.