I'm porting your MD5 code to Retroquad, but I have a few questions:

In R_MD5DrawModel, why do this:
md5header_t *hdr = (md5header_t *)currententity->model->cache.data;
Instead of this?
md5header_t *hdr = (md5header_t *)Mod_Extradata (currententity->model); 

...because I didn't send MD5 models through the Cache_Alloc system.

Note that this is the same way as Software Quake handles sprite models as well; they're not Cache_Alloc'ed so R_DrawSprite retrieves the data by directly accessing currententity->model->cache.data:

void R_DrawSprite (void)
{
int i;
msprite_t *psprite;
vec3_t tvec;
float dot, angle, sr, cr;

psprite = currententity->model->cache.data;

r_spritedesc.pspriteframe = R_GetSpriteframe (psprite);

To be honest, a modern engine should be getting rid of the Cache_Alloc system anyway; that was just complexity and overhead to get Quake running on MS-DOS with 8 MB of RAM and no virtual memory. 

Thanks a lot, I got it fully working.

There are a couple issues with the vertex normals, though. While they're not very noticeable with the standard MDL lighting, they look really bad with chiaroscuro lighting: screenshots.

Not only are the normals split (not smoothed) between the back and the front of the models, it also seems that the angles of the normals are wrong, because the angle of the lighting isn't coming from the camera as it should.

I've confirmed that in the KEX engine the normals are properly smoothed between the front and the back of the models. The only thing I can't confirm in it are the angles, since the KEX engine doesn't feature chiaroscuro lighting.

Any idea how to fix these? 

The MD5 files don't store normals so you need to recalculate them yourself. I grabbed the normal code from ID's original modelgen.c to do this: https://github.com/id-Software/Quake-Tools/blob/master/qutils/MODELGEN/MODELGEN.C#L768 - on the basis that if it was going to be wrong, it may as well be at least consistent. It's entirely possible that I made a mistake, as I verified by eyeballing a comparison between shading on the MD5 file vs shading on the origjnal MDL, checking that the angles all looked correct (using an all-white (actually all-grey) texture to do so). But if there are cases where it doesn't work right, or where my verification approach missed something, I wouldn't be surprised, and it could probably stand to be rewritten. 

Thanks for clarifying.

Yes, vanilla Quake shading is almost flat and makes shading issues harder to notice. I bet they did it this way because the direction of the shading was fixed.

Plus, the original software renderer can make it hard to distinguish between split edges and color banding.

I'll look into it and try to figure out how to recompute the vertex normals. 

I've looked over the code again and it's straightforward and standard enough - compute triangle normals then average them to get vertex normals.

There must be something in the data that I'm missing, such as the same vertex being reused for different facings, that must be causing this. It's probably worthwhile looking into other MD5 loaders and renderers to see how they handle it; from memory it was a fairly common format for samples and tutorials, so code tro rustle up normals should be easy to find. 

I just fixed the bad angles, it was my fuckup; sorry for that.

However, there still are seams in the shading.

The shading in Quake MDLs is seamless because each 3D vertex in the model can correspond to two 2D vertices in the skin; this allows the shading to be smoothed across the 3D mesh, while allowing the skin to be split in the 2D texture.

I'm going to examine the MD5 format to see how the skin is mapped, because it seems that the seams in the shading are appearing where the texture's polygons are split.
If the 3D vertices are mapped 1:1 to the 2D vertices, this means that some 3D vertices are duplicated for each skin vertex. In this case, the engine will have to compare the initial positions of all vertices to find the duplicated vertices and smooth the normals across them. 

Looking into the MD5 code now, that seems to be the case.

md5_vertex_t contains the 2D vertices in skin space.
md5_weight_t contains the 3D vertices in model space.

MD5_BuildBaseNormals reads the 3D weights from the 2D vertices, which makes the normals to be not smoothed across all neighboring 3D weights; this is what causes the seams in the lighting. Also, this problem seems to be rooted into the MD5 format itself.

I'll devise a way to fix this. 

I just fixed the bad angles, it was my fuckup; sorry for that.

No probs. I'd still like to get a better normals calculation than just ripping from modelgen anyway, but I guess that can be left as an exercise for each implementer. 

Keeping the normals consistent between MDL, MD5 and other formats is good because it allows the lighting functions to be shared between model formats.

When porting your MD5 implementation, I deleted the R_MD5SetupLighting function and used my custom lighting functions instead (R_AliasSetupLighting, R_AliasSetupLighting_new, R_AliasSetupLighting_emitter, R_AliasSetupLighting_fullbright). It was just a matter of using the same globals (e.g. alias_forward, alias_right, alias_up) for both formats. Being able to reuse the code like this makes it easier to maintain.

By the way, maybe id Software didn't care much about getting the normals perfectly smoothed in the MD5 format because they were going to use normal/bump maps anyway.

And in the MDL format, the only way to force the normals to be not smoothed is to duplicate the 3D vertices for each triangle that shouldn't be smoothed. Since the MD5 format supports multiple meshes, maybe that's how they planned to support non-smoothed normals — instead of duplicating individual 3D vertices, just move their entire triangles to other meshes. Non-smoothed normals are useful to give sharp edges to cubes, blades, etc.

Anyway, I'll keep studying this. 

I totally agree on consistency and ability to use the same functions.

What I haven't done is reviewed the MD2 normal generating code; my gut feeling is that it's not going to be wildly different in the end result to MDL, but since MD2s are properly uv-mapped (rather than MDLs which just have a front and back skin, and the facesfront/onseam stuff) it might be a better fit for MD5s. 

So I cross-checked with the Doom 3 and Doom 3 BFG Edition code, and both of their MD5 loaders have extra code for duplicating mirror seam vertices, which I guess is what's needed to resolve all of this.

The source I used didn't have this code.

We obviously don't have access to the KEX source code, but I'd say the likelihood is that they just grabbed the Doom 3 code and mostly reused that.

That's not an option for most Quake source ports as the Doom 3 code is C++ and used a lot of custom types.

The mirror seam code does seem straightforward enough to port though, but I do want to look over how it's going to interact with the rendering side of things before doing anything. 

Thanks. What's weird is that the MD5 format itself doesn't have enough data to simplify that process.

Let's say, they could have changed this:

typedef struct md5_vertex_s
{
vec2_t st;

int start; // start weight
int count; // weight count
} md5_vertex_t;

To this:

typedef struct md5_vertex_s
{
vec2_t st;

int start; // start weight
int count_2d_space; // weight count for skin (same as above).
int count_3d_space; // same as count_2d_space, plus the extra ones for 3D mesh that are non-adjacent in 2D space.
} md5_vertex_t;

And there would be no need for that extra code in the model loader. 

OK, I've implemented the mirror seam stuff in the Software MD5 code at https://github.com/mhQuake/MD5Stuff/commit/e2b86e5892e846f80969b564975fff81af502778 - there *may* be extra work required because there's one piece of data that's allocated but not used so far; let's see how we get on with this part of it first.

I've tried to keep the implementation as low impact as possible, so it's just a couple of extra struct members, a single function call in the loader, then the functions themselves to duplicate the verts. That means that there is one place where I leak some memory; it's hunk-allocated memory so it will be freed in Host_ClearMemory between maps, but a proper production-ready implementation would address this properly. 

Hmm, I've copied & pasted those changes, but they didn't make any difference.

After including a warning here, I've confirmed that none of the MD5 models in the Quake remaster have mirrored vertices:

static void R_DuplicateMirroredVertexes (md5_mesh_t *mesh, char *name)

...

// now create the new list
if (totalVerts == mesh->num_verts)
{
mesh->mirrored_vertices = NULL;
Con_Printf ("No mirrored vertices for %s.md5mesh\n", name);
return;
}

So, mirrored vertices are not the issue. 

Here's how it looks:
Scene 1, MD5 off
Scene 1, MD5 on
Scene 2, MD5 off
Scene 2, MD5 on

Our approaches are having zero effect on this. I implemented some code to weld all normals from vertices from identical 3D space positions, and it still looked the same. 

Alright, I've fixed it for good. My code to weld the md5header_t->vnorms->normal data works now. 

Good news - it would be great if you could submit a patch to the Github repo, or at least indicate what was required to do it, so that others can avail of the fix. 

I'm looking into it now, trying to figure out how this Github thing works. If I knew your e-mail, I'd have emailed you before. 

Thanks, that seems to have worked, I've accepted the change and it's in the Software Quake code now.

I'm also going to implement it in the Fitz/GL code but it has slightly different structures and types so needs a bit more work. 

Speaking of data structures, I wonder how different it would be to implement IQM model support in software.

Most people I've talked to said that the IQM format is much easier to work with than MD5, and some were disappointed to see the remaster using MD5 instead of IQM. 

maybe because IQM is community-made and MD5 is id Software created? They might think it's safer to use MD5 from a licensing perspective. 

That's certainly a factor.

Another possible factor is that the author of the remastered MD5 models is Capnbubs, who started doing those models in 2013 Link. The skin of the grunt in the remaster is exactly the same, with an added muzzleflash, and the dog skin is 100% the same.

Back then Capnbubs already mentioned working with an skeletal model format, which may be the MD5 format itself. I don't know when the IQM format was created. 

Hmm, capnbubs actually used Blender. Might be the licensing thing then. 

I've implemented IQM in GL before, and it seems like it should be straightforward to do in software as well.

The big pressure point of doing it in GL was limited constant register space for storing bone matrices, as I was constrained to SM2.0 hardware at the time (256 constant registers - ouch!) That wouldn't exist in software as you'd be doing the bone animation in software anyway (you could also do the bone animation in software with a GL implementation, but it's substantially slower).

Otherwise I'd advocate the same approach as I took with the MD5 software code - just produce the same data structs as are used by the stock MDL code so it will easily plug into the rasterizer.

Re: community vs ID: they used BSP2 and that's definitely community-made (I made it, so I can say that quite authoritatively). Obviously I haven't seen the remaster engine code, but I'd hazard a guess that they used MD5 because they could just lift over most of the code from Doom 3.

Otherwise nobody authors directly in either MD5 or IQM - they'll author in a proper modelling package and use an exporter.