you don't need to use -fence, that is a light compile only thing for it to try and cast shadows based on your cutout textures.

alpha cutout textures simply need a few things to work: { prefix on the name of the texture; it needs to be applied to an entity (func_wall, func_illusionary, etc); and the parts which are going to be invisible need to be palette color 255. You don't need to compile specially, the engine is what does the work.

did you grab {swvine texture from the arcane dimension start map (or any AD map that used it)? if so, you should be all set, it would've been named correctly... open the compiled .bsp in texmex and be sure the texture is renamed, I suspect you've likely messed something up there either in the .map or in your .wad. Did you remember to change the brush in your map to the renamed texture? 

 

Oh, this is just pure masking ... OK. 

Make sure texmex isn't doing that thing where it replaces fullbrights with not-fullbrights. 255 is a fullbright. 

Along with its WAD version, your texture must be an external file, a TGA image with an alpha channel where black pixels are transparent and any other shade of gray (better use white) are opaque. This file must be in a <mod>/textures folder.

It works in MarkV and QS. 

another one mistaken here. 

I'm using that texture from the Wad of Daikatana ep1, taken from Quaddicted's wad bank. After checking the vine texture it had the wrong background color (it seems like this is a common problem in Quaddicted's wad bank), so I fixed it and run it. Darkplaces still doesn't recognizes the transparency (is there a commandline that can fix this?), but Quakespasm runs it like it should:
http://image.noelshack.com/fichiers/2016/02/1452527281-spasm0000.png
http://image.noelshack.com/fichiers/2016/02/1452527283-spasm0001.png

Also lighttorches I underestimate them once again. 2stronk4me.
Speaking of light, fog comes out fine in Quakespasm but is non present in DP. Same thing with the sky texture, it's all black even though I put lights (sunlights still have light and shadow issues with ericw's compiler it seems, or the small sky openings in the cieling make things fuzzy somehow). What the hell 

Does not support alpha textures. That is the reason you need an external tga file 

 

"14 new posts! This gonna be goo ... oh." 

So I don't know what the hell is going on. My start map is as for now divided by 2 sections: a short cave section, and the skill portal room. No matter what I do (and by that I mean changing light torches' light values, deleting some, up to just deleting all of them), the cave will always be flat-litted, but as soon as the skill chamber show its nose WHOOPS strong shadows and sunlight-quality shadows (At first I did add in sunlight2 but I decided to take it out because there wasn't any impact lighting-wise).
Just look (also in those shots light torches in the first section are supposed to be deleted, and worldspawn's light value is set to 170):
http://image.noelshack.com/fichiers/2016/02/1452545720-razenkrauft20160111214603-00.jpg
http://image.noelshack.com/fichiers/2016/02/1452545717-razenkrauft20160111214611-00.jpg
http://image.noelshack.com/fichiers/2016/02/1452545720-razenkrauft20160111214655-00.jpg
http://image.noelshack.com/fichiers/2016/02/1452545718-razenkrauft20160111214744-00.jpg
http://image.noelshack.com/fichiers/2016/02/1452545721-razenkrauft20160111214753-00.jpg
Also, somehow removing the fog key from worldspawn made the skybrushes visible again in DP.
I don't get it. 

are you using for your lights?
It looks like delay 3 lights(no falloff at all) from afar to me.

Use delay 2/5 for best results, also add the -addmin switch to your commandline for tyrlight.