[Variant] add variant_to_arrow union builder

[sdf-jkl]

Which issue does this PR close?

The last type to be supported by variant_to_arrow cast.

• Closes [Variant] variant_to_arrow types support #8477.

Rationale for this change

Check issue

What changes are included in this PR?

• Add variant_to_arrow union cast support + unit tests

Are these changes tested?

• Yes, unit tests

Are there any user-facing changes?

• Users can now cast Variant to Union type

[Jefffrey]

disclaimer: this is kinda my first time looking at variant code 😅

[Jefffrey]

do we need to consider nullability of the field?

[sdf-jkl]

I checked the existing Arrow union policy. UnionBuilder::append_null intentionally writes nulls into child arrays, while UnionBuilder::build still creates those child Fields with nullable = false. UnionArray::try_new does not validate field nullability, and union nullness is derived from the selected child array. Therefore the fallback does not need to select a nullable-declared field.

arrow-rs/arrow-array/src/builder/union_builder.rs

Lines 285 to 317 in e28fd0d

	let mut children = Vec::with_capacity(self.fields.len());
	let union_fields = self
	.fields
	.into_iter()
	.map(
	|(
	name,
	FieldData {
	type_id,
	data_type,
	mut values_buffer,
	slots,
	mut null_buffer_builder,
	},
	)| {
	let array_ref = make_array(unsafe {
	ArrayDataBuilder::new(data_type.clone())
	.add_buffer(values_buffer.finish())
	.len(slots)
	.nulls(null_buffer_builder.finish())
	.build_unchecked()
	});
	children.push(array_ref);
	(type_id, Arc::new(Field::new(name, data_type, false)))
	},
	)
	.collect();
	UnionArray::try_new(
	union_fields,
	self.type_id_builder.into(),
	self.value_offset_builder.map(Into::into),
	children,
	)

[Jefffrey]

the builder thing is a known issue:

• UnionBuilder produces incorrect Union DataType #1637

i guess its one of those things which is a bit messy in the codebase as is 🤔

[sdf-jkl]

Thanks @Jefffrey 🙏

This is not the most Varianty code, mostly about understanding the Union type

[sdf-jkl]

@klion26 please you take a look 🙏

[Jefffrey]

thanks @sdf-jkl