I have some data coming in avro format v1 and getting stored in HDFS under a partition dt=yyyymmdd.
Now the data is maintained with two versions, v1 and v2 under the same partition.
Is it feasible to maintain a single hive table for two different versions?
Avro Dynamic schema change on Hive
801 Views Asked by Apoorva Ballari At
1
There are 1 best solutions below
Related Questions in HADOOP
- Can anyoone help me with this problem while trying to install hadoop on ubuntu?
- Hadoop No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster)
- Top-N using Python, MapReduce
- Spark Driver vs MapReduce Driver on YARN
- ERROR: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "maprfs"
- can't write pyspark dataframe to parquet file on windows
- How to optimize writing to a large table in Hive/HDFS using Spark
- Can't replicate block xxx because the block file doesn't exist, or is not accessible
- HDFS too many bad blocks due to "Operation category WRITE is not supported in state standby" - Understanding why datanode can't find Active NameNode
- distcp throws java.io.IOException when copying files
- Hadoop MapReduce WordPairsCount produces inconsistent results
- If my data is not partitioned can that be why I’m getting maxResultSize error for my PySpark job?
- resource manager and nodemanager connectivity issues
- ERROR flume.SinkRunner: Unable to deliver event
- converting varchar(7) to decimal (7,5) in hive
Related Questions in HIVE
- Type Adapter for Offset in hive flutter
- HIVE Sql Date conversion
- How to set spark.executor.extraClassPath & spark.driver.extraClassPath in hive query without adding those in hive-site.xml
- Hive query on HUE shows different timestamp than programatically/on data
- descending order of data in hive using collect_set
- How to optimize writing to a large table in Hive/HDFS using Spark
- Spark SQL repartition before insert operation
- Alter datatype of complex type(array<struct>>) in hive
- SqlAlchemy connection to Hive using http thrift transport and basic auth
- Aggregate values into a new column while retaining the old column
- Is it possible to query MAPR hdfs/hive tables from Trino?
- Can we make a column having both partitioning and bucketing in hive?
- converting varchar(7) to decimal (7,5) in hive
- Extract all characters before numeric values in hive SQL
- Livy session to submit pyspark from HDFS
Related Questions in AVRO
- Incorrect Serialization and Deserialization of Union Types with dataclasses-avroschema
- Lambda function returning null parameters when receiving Kafka event
- Azure Data Factory: How to import a complex json object from Avro file
- Neo4j Source Connectors Failing to build the Schema where the source query returns null for some of the fields
- Kafka message not deserializable. How to debug
- Avro4k - Exception: Not a named type: "int"
- How to convert an avro schema into an asyncapi programatically?
- How I deserialize Avro from Kafka with spring boot 2.7.18
- What format does apache pinot use for storing segments in deep storage?
- Avro after upgrading to JDK 17
- Is there a console code formatter for Avro IDL?
- ReflectDatumWriter failing with error "Array data must be a Collection or Array"
- How to create an avro schema containing list of records for apache nifi?
- avro-tools-1.11.1.jar causes NoClassDefFoundError in my existing program
- How to figure out why Glue Schema Registry Avro Schema Evolution failed
Related Questions in HORTONWORKS-DATA-PLATFORM
- I want to ingest the csv data to the HDFS with Hortonworks Data Platform Sandbox
- Creating Data Lineage
- Is it possible to get data both hadoop cluster?
- Apache ranger hive plugin for HDP cluster is not working
- How can upload large files to the Horton work Sandbox HDP 2.6.5
- Hortonworks 2.6.5 yum install python-pip not working
- Partition the data frame using column X and writes the data without column X
- Why does ambari is showing this kerberos authentication error : AmbariAuthToLocalUserDetailsService
- How to get hortonworks data platform and cloudera distribution for hadoop latest version
- NiFi processors cannot connect to Zookeeper
- Postgresql stuck in recovery mode
- wget + download ambari tar ball
- How to copy a file from /user/maria_dev/tutorials/test.csv (HDP) to /sandbox/tutorial-files/640/nifi/input (HDF)?
- Do we need install all HDP's Services Client in all node?
- Hive - Convert epoch time (ms - 13 digit) to timestamp till milliseconds in hive sql
Related Questions in JACKSON-DATAFORMAT-AVRO
- How do I convert an Avro object to Json and back ? (when avro schema contains union)
- Deserialize an avro BigDecimal custom field and then serialize it to String
- Is there a way to define avro schema file ( .avsc file ) that generates a POJO with a 'Set' member variable?
- Does Avro Schema Allow Conditional Fields?
- How generate Avro Schema with Jackson without Java types
- Error register jackson avro schema with confluent cloud schema registry client
- How to prevent org.apache.avro.AvroTypeException: Unknown union branch after making a field nullable?
- Serialize a JSON String to Avro Object that has Union Fields
- Avro Schema Data Validations with Regular Expressions in Java
- Avro Conversions are not being called for extensions of Encoder
- Is it possible to assign default Value null to union{null, int} avro schema?
- Generate Avro schema for java <Object> type
- Unable to send Kafka Avro Message to Message Channel <Failed to convert Generic Message to Outbound Message>
- Not in union ["null","int"] Avro Format org.apache.avro.UnresolvedUnionException
- How to have a Json field in AVRO?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Avro defines a schema evolution protocol
If v2 has simply added a field with a default value, for example, then updating the table with that schema, it can read the entirety of the old data, as it'll simply return the default values where they are missing.
If you've broken compatibility, you must make a separate table, then union the two to get a consistent result set