Page MenuHome

Native STL importer
AbandonedPublic

Authored by Iyad Ahmed (iyadahmed2001) on Jan 6 2022, 10:03 AM.

Details

Summary

When importing large files, or many files in batch, the current STL importer is slow,
many file types are getting a native importer/exporter now, so a native importer for STL makes sense too.

An alternative solution is to write a new Python importer, but writing a native C or C++ importer brings better performance.

This is intended to replace the Python importer, an exporter is also planned, better to split into a different patch.

Diff Detail

Repository
rB Blender

Event Timeline

Iyad Ahmed (iyadahmed2001) requested review of this revision.Jan 6 2022, 10:03 AM
Iyad Ahmed (iyadahmed2001) planned changes to this revision.
Iyad Ahmed (iyadahmed2001) created this revision.
Iyad Ahmed (iyadahmed2001) planned changes to this revision.Jan 6 2022, 10:10 AM

I just realized the ASCII importer is not very robust with different whitespace types, I need to rewrite it

Updated ASCII import to read token by token (word), instead of line by line

Iyad Ahmed (iyadahmed2001) planned changes to this revision.Jan 6 2022, 10:18 PM

Fixed mesh not updating after import when the operator is called from Python console

Iyad Ahmed (iyadahmed2001) planned changes to this revision.Jan 7 2022, 1:53 PM
Iyad Ahmed (iyadahmed2001) retitled this revision from Fast STL IO to Fast STL IO [WIP].Jan 7 2022, 2:17 PM
Iyad Ahmed (iyadahmed2001) planned changes to this revision.Jan 8 2022, 1:59 PM
Iyad Ahmed (iyadahmed2001) updated this revision to Diff 46760.
Iyad Ahmed (iyadahmed2001) planned changes to this revision.Jan 8 2022, 5:16 PM
Iyad Ahmed (iyadahmed2001) updated this revision to Diff 46763.
Iyad Ahmed (iyadahmed2001) planned changes to this revision.Jan 8 2022, 5:28 PM
Iyad Ahmed (iyadahmed2001) updated this revision to Diff 46764.
Iyad Ahmed (iyadahmed2001) retitled this revision from Fast STL IO [WIP] to Native STL importer.
Iyad Ahmed (iyadahmed2001) edited the summary of this revision. (Show Details)
Jesse Yurkovich (deadpin) requested changes to this revision.Jan 11 2022, 8:17 AM

A few notes

  • The old importer allowed multiple files to be imported at once, this one does not. I think you need to look at the "files" property instead of just "filepath".
  • If this is to replace the old importer, when will that happen? Right now both will be listed in File->Import -- unsure of the policy but might have to remove the old one at the time of checkin OR state in your description that there will be 2 for a short time and why etc.
  • For binary, as much as possible, reduce the calls to fread. It will be more efficient to read in say 64kb of data in 1 call vs. what happens here where it's called 5 times for each triangle.
  • There's an issue with detecting ASCII files. It seems like detecting "solid" isn't entirely correct and I have a few files not loading. Ping me on chat for how to get them.
  • There's an issue with attempting to only create verts exactly once by using the ghash here. Some other files I have end up triggering asserts in bmesh code -- effectively v1 and v2 in this case are exactly the same, since it was only created once, and bmesh doesn't like that:
BLI_assert failed: F:\source\blender-git\blender\source\blender\bmesh\intern\bmesh_core.c:142, BM_edge_create(), at 'v1 != v2'

 	blender.exe!_BLI_assert_abort() Line 61	C
>	blender.exe!BM_edge_create(BMesh * bm, BMVert * v1, BMVert * v2, const BMEdge * e_example, const eBMCreateFlag create_flag) Line 142	C
 	blender.exe!BM_edges_from_verts_ensure(BMesh * bm, BMEdge * * edge_arr, BMVert * * vert_arr, const int len) Line 72	C
 	blender.exe!BM_face_create_verts(BMesh * bm, BMVert * * vert_arr, const int len, const BMFace * f_example, const eBMCreateFlag create_flag, const bool create_edges) Line 489	C
 	blender.exe!BM_face_create_quad_tri(BMesh * bm, BMVert * v1, BMVert * v2, BMVert * v3, BMVert * v4, const BMFace * f_example, const eBMCreateFlag create_flag) Line 91	C
 	blender.exe!read_stl_binary(BMesh * bm, _iobuf * file, float[3] * mat, bool use_facet_normal, GHash * gh_float3) Line 139	C
This revision now requires changes to proceed.Jan 11 2022, 8:17 AM

Many thnaks for review🙏 will look into these later today

Fix some binary STLs misidentified as ASCII
Refactor ASCII read to read line by line (same as old importer), made great gains in performance after measuring

  • reduce calls to fread for Binary import
  • skip degenerate faces (was triggering BMesh asserts)

Support importing multiple files, same as old importer

handle endianness

Generally looks ok to me functionally and feature wise compared to the old importer now, thank you for addressing those prior comments. I'll defer to other reviewers for a final signoff and review though.

Some further comments:

  • Hans suggested to use c++ for all this if I remember correctly. It would allow you to use some modern classes like Set instead of ghash. Let's see what the others say on this review.
  • I made 2 local changes to see if binary performance could be improved by reducing fread calls further. First I tried just 1 read instead of 2 for each triangle, removing the fseek skip of the uint16_t. This improved things slightly but not worth it alone (1.01x faster). Then I tried reading in 1000 triangles at once (a 50k temporary buffer is fine for an importer). This improved things greatly and made it 1.32x faster over a directory of 8 files totaling 2.3 million faces on a spinning disk (1.48 sec vs. 1.95 sec). Just something to consider.
source/blender/io/stl/stl.h
36

Padding is unnecessary in normal structs like this.

Awesome, will change the struct, many thanks for experimenting with binary importer, I guess I will make it buffered then

Remove unnecessary struct padding

Iyad Ahmed (iyadahmed2001) marked an inline comment as done.Jan 22 2022, 1:43 PM

I realized a mistake with code though will fix asap

I'm using endian_switch_float on a float array

What about Triangle structure alignment, I'm having issues getting the read to work with an array of triangles

typedef struct Triangle {
  float normal[3];
  float v1[3];
  float v2[3];
  float v3[3];
  uint16_t attribute_byte_count;
} Triangle;
  • Fix unused variable warning
  • Check fread and fgets results
  • Fix BLI_endian_switch_float used for float array

Waiting for reviewers to be set

Thank you for your submission. The approach in this patch should have been discussed before spending a lot of time on the code, though. There is a design task T68936 for a similar effort: to reimplement certain importers and exporters in C++.

Most importantly: talk before coding. If you want to make sure your efforts are not in vain, discuss your ideas and their design, before spending a considerable amount of time on them. Please read through Contributing Code for more info on how to best get your contributions into Blender.

Thanks for feedback, I asked on blender chat coders channels before, and got the illusion that STL was not being worked on yet

Thanks for feedback, I asked on blender chat coders channels before, and got the illusion that STL was not being worked on yet

I believe your implementation is only one so far, I believe there is no implementation for STL and PLY, just obj export and import are worked on.

Thanks for feedback, I asked on blender chat coders channels before, and got the illusion that STL was not being worked on yet

I believe your implementation is only one so far, I believe there is no implementation for STL and PLY, just obj export and import are worked on.

Yes but STL is planned as part of fast io gsoc project, and part of the obj importer design (it is designed in a way it can work with STL too)