I am considering Airbyte as a replacement for singer-io because of its user-friendly UI and code reduction features. However, facing an issue where Airbyte is unable to detect certain columns in our source (Atlas hosted on AWS), which is a significant concern for me.
For instance,
Consider a table users with columns user_id, name, location, phone_num.
While the sync is performed, column location is missed, while it is present in mongo db.
In Airbyte UI also column location is not being shown.
I tried to figure out this issue via Octavia CLI, but could not find any way that columns can be defined.
Just wanted to mention that this column has ~70% nan values.
But going with the fact that even for 30% of data the column should not be missed.
I tried to re-sync data but no results. I tried to install Octavia CLI to try to define columns for the table, but no such option was available.
I've had issues with custom datatypes in Airbyte before where fields are silently left out of the schema discovery process.
Try sticking to the intrinsic types.