How to copy separate tables into table vector?

[AndrewJDR]

Let's say I have:

table SomeTable {
  id:uint;
  someStrings:[string];
  name:string;
  name1:string;
  name2:string;
  name3:string;
  name5:string;
  
  // Etc... Pretend this is a very large table with many, many members.
}

I have a whole bunch of these SomeTable flatbuffers saved out into separate binary files somewhere on my filesystem as root objects.

Now let's say that in my code, I want to create a new table comprised of a vector of those SomeTables flatbuffers:

table SomeTables {
  tables:[Sometable];
}

How can I build this SomeTables buffer as concisely as possible? That is, I'd very much prefer to avoid code like this (pseudocode):

ns(SomeTables_start_as_root(B));
ns(SomeTables_tables_start(B));
ns(SomeTables_tables_push_start(B));
for(thisTableFromMyFilesystem from someTablesOnMyFilesystem)
{
  ns(SomeTable_table_t) thisTable = ns(SomeTable_as_root(thisTableFromMyFilesystem));
  ns(SomeTables_tables_push_create(&b, flatbuffers_string_create_str(B, ns(SomeTable_name1(thisTable), ns(SomeTable_name2(thisTable), ...)); // <-- I have to update this line when the SomeTable schema changes.
}
ns(SomeTables_tables_push_end(B));
ns(SomeTables_tables_end(B));
etc...

In such an arrangement, I have to update the SomeTables_tables_push_create line when the SomeTable schema changes. I'd prefer a "Take this SomeTable_table_t, copy its contents into the flatbuffer, and push it onto this table vector" function.
Does such a thing exist? e.g. SomeTables_tables_push_create_from_table_t in this example?

ns(SomeTables_start_as_root(B));
ns(SomeTables_tables_start(B));
ns(SomeTables_tables_push_start(B));
for(thisFlatBufferFromMyFilesystem from someTablesOnMyFilesystem)
{
  ns(SomeTable_table_t) thisTable = ns(SomeTable_as_root(thisTableFromMyFilesystem));
  ns(SomeTables_tables_push_create_from_table_t(B, thisTable)); // With this, I don't have to worry about a long push_create() line where I pass in all the values...
}
ns(SomeTables_tables_push_end(B));
ns(SomeTables_tables_end(B));
etc...

[mikkelfj]

Hi Andrew, thanks for asking.

Unfortunately there is no such function although it would be good to have.

It does not exist because it requires code for recursively iterating over tables - a copied table can have a vector of other tables etc. and there could be two references to the same table during copy - if this is not handled, data could explode in size.

So it is non-trival to implement, but definitely possible - it would approximately follow the same logic as the generated verifier code and I am open to contributions if anyone feels up to the task.

[AndrewJDR]

@mikkelfj Thanks for the quick reply! Taking on such a task is a bit more than I can manage at the moment, so I'll just rethink my overall approach and try avoid this problem altogether for now.

[mikkelfj]

I would suggest you implement your own type specific table clone function. It would return a reference to the table and you can push that reference onto a vector or store it as a member of another table. In monster_test you will find examples of how to juggle table creation with references in different ways.

You still need to adapt the function, but you only have to do it in one place per schema change. You can also avoid copying all fields, if some are irrelevant and not flagged as required.

If you then feel like you need a function as you suggest, you could easily create a non-changing push_clone on top of that.

This is what I had in mind when leaving out the table clone feature in the early design.

[AndrewJDR]

Thanks. As a side note, is this even something that the flatbuffers C++ project supports to your knowledge? Just curious.

[mikkelfj]

I'm not really sure - I believe the C++ version does have some advanced reflection logic that allows you to modify a buffer in temporary storage and rewrite it, and possibly also to copy from an external buffer - but it is probably not the fastest way to go about things.

[mikkelfj]

You could also write your own purpose specific clone function generator in C using reflection without supporting all sorts of special cases. It could be based on https://github.com/dvidelabs/flatcc/tree/master/samples/reflection

[AndrewJDR]

Another quick, slightly related question. It feels like it should be possible to clone an existing flatbuffers_string_vec_t from another buffer and add it to something, e.g. something like:

// let's say a flatbuffers_string_vec_t is in strVecT
ns(SomeTable_someStrings_add(flatbuffers_string_vec_clone(strVecT)));

But I've tried all different combinations of things and can't seem to get anything to work. Is this possible?

[mikkelfj]

It appears not - though it could be implemented with a simple loop since the string_vec has a push_clone method.

If you look at flatbuffers_common_builder.h you'll find there is a vec_clone but no string_vec_clone:

#define __flatbuffers_build_vector(NS, N, T, S, A)
...
static inline N ## _vec_ref_t N ## _vec_clone(NS ## builder_t *B, N ##_vec_t vec)
...

#define __flatbuffers_build_string_vector_ops(NS, N)
...

[mikkelfj]

Note: strictly speaking a string_vec_clone and string_vec_slice should handle multiple references to the same string, so a robust implementation is not that trivial to do. It would require a hash table, and probably an option to not check for duplicates for performance and memory concerns.

[AndrewJDR]

Thanks again for your pointers. I'd like to move back to the original question a bit...

Let's say I want to embed a bunch of separately saved flatbuffers each containing a SomeTable_table_t into a brand new flatbuffer that I'm building (literally, I just memcpy them in there, end-to-end), retrieve references to those tables each time I memcpy them in, then populate the SomeTables_tables list with references to each of the SomeTables that i memcpy'd in earlier.

Given each flatbuffer containing the SomeTable_table_t is self contained and will by definition contain everything else required by that SomeTable_table_t, why is what I describe not possible? Basically, I don't entirely understand the need for the table recursion as you described it, why can't I just blindly copy in these flatbuffers as "sub flatbuffers", then refer to them from later in the flatbuffer? Is this some limitation of the flatbuffer format that it doesn't allow references of this kind?

Thanks!

[AndrewJDR]

As a side note, I actually tried to do something like what I just described using flatcc_builder_embed_buffer, but I didn't get far with that. I believe I'm misunderstanding something fundamental about the flatbuffer format, here.

[mikkelfj]

This is complex question, but I'll try:

The short answer you probably seek is that if a table does not contain any offset types, you can copy the table into a new buffer iff you make sure to also copy the vtable. A string, a vector, and a table are all offset types meaning that these exists it different memory blocks in the existing buffer. You would then have to chase each offset and copy the data, then copy the table referencing those data and update the tables references to the copies. Note that these references are known as uoffset_t types and are NOT the reference type the builder works with. If the offset types do not contain further offset types, the recursion stops there, meaning a table does contain subtables or vectors of tables. But even here, you need to consider the possibility of the same string being referenced in two places and decide if you want to copy the string once, or just duplicate it naively. Updating offsets and vtable references and ensuring proper alignment in the resulting buffer is non-trivial - this is what the builder does for you.

If you want to get creative, you can use the untyped low-level flatcc_builder.h interface to build you own data with proper alignment iff you respect the schema the buffer is expected to have.

You also need to understand that the ref type in the builder is an relative offset to specific location in the new buffer being built. Even if the buffer does not (yet) physically exist as a single coherent block, the reference will be translated to a relative distance in the final resulting buffer. This final distance is an offset type. You (or the builder) cannot know the offset before you both know the location of the source location and destination location in the new buffer. References exists to work around this temporarily. When a vector is being built piecemeal, the references are pushed, and when the vector closes, all references are converted. Anything added to an open vector must already have been added to the buffer. (Edit: unlike C++ you can add things, such as a subtable or a string, while the vector is open, and then push the reference, but you can't have the reference to push before the object has been created. C++ requires you to create the object even before opening the vector, and then remember all references before creating the vector. The same applies to Go and other languages).

You cannot get cannot get a reference into another existing buffer, because it makes no sense as a concept. You also cannot easily access an offset from an existing buffer because it is not exposed - it is always converted into a pointer for easy access. Read only bavigation works internally by adding offsets to pointers (roughly).

So, you could theoretically copy a very simple table type, but then it is almost the same as struct type, except for schema evolution. So you might consider using structs instead, and structs can be cloned by the existing builder API.

None of this prevents copying tables from one buffer to another, there is just a lot of detail to manage.

[mikkelfj]

Note on shared strings - which also apply to vectors and tables:

If a string is referenced in two different places in the same buffer, it will be stored as two different uoffset_t values that happen to reach the same location.

When building a new buffer, a string will get a single string_ref_t value. This reference can be used in several places and will later by converted to offsets, where the offset value depends on the location from which the string is being referenced.