Scio 0.5.0 released

[Rafal Wojdyla]

Hi all,

We have just released Scio 0.5.0. This is a major/breaking release - make sure to read the breaking changes section below.

- rav

https://github.com/spotify/scio/releases/tag/v0.5.0

"In ictu"

Breaking changes

• BigQueryIO in JobTest#output now requires a type parameter. Explicit .map(T.toTableRow)of test data is no longer needed.
• Typed AvroIO now accepts case classes instead of Avro records in JobTest. Explicit .map(T.toGenericRecord) of test data is no longer needed. See this change for more.
• Package com.spotify.scio.extra.transforms is moved from scio-extra to scio-core, under com.spotify.scio.transforms.

See this section for more details.

Features

• Support reading BigQuery as Avro #964, #992
• BigQuery client now supports load and export #1060
• Add TFRecordSpec support for Featran #1002
• Add AsyncLookupDoFn #1012
• Bump sparkey to 2.2.1, protobuf-generic to 0.2.4 #1028
• Added ser/der support for joda DateTime #1038
• Password is now optional for jdbc connection #1040
• Add job cancellation option to ScioResult#waitUntilDone #1056 #1058 #1062 #1066

Bug fixes

• Fix transform name in joins #1034 #1035
• Add applyKvTransform #1020 #1032
• Add helpers to initialize counters #1026 #1027
• Fix SCollectionMatchers serialization #1001
• Check runner version #1008 #1009
• Log exception in AsyncLookupDoFn only if cache put fails #1039
• ProjectId nonEmpty string check in BigQueryClient #1045
• Fix SCollection#withSlidingWindows #1054

[Eugene Kirpichov]

Hi!

On Fri, Mar 9, 2018 at 1:22 PM Rafal Wojdyla <r...@spotify.com> wrote:

Hi all,

We have just released Scio 0.5.0. This is a major/breaking release - make sure to read the breaking changes section below.

- rav

https://github.com/spotify/scio/releases/tag/v0.5.0

"In ictu"

Breaking changes

• BigQueryIO in JobTest#output now requires a type parameter. Explicit .map(T.toTableRow)of test data is no longer needed.
• Typed AvroIO now accepts case classes instead of Avro records in JobTest. Explicit .map(T.toGenericRecord) of test data is no longer needed. See this change for more.
• Package com.spotify.scio.extra.transforms is moved from scio-extra to scio-core, under com.spotify.scio.transforms.

See this section for more details.

Features

• Support reading BigQuery as Avro #964, #992
• BigQuery client now supports load and export #1060
• Add TFRecordSpec support for Featran #1002
• Add AsyncLookupDoFn #1012

This looks like a very useful general-purpose tool, Beam users have asked for something like this many times; and the implementation looks very high-quality as well. Any interest in backporting it into Beam?

[Neville Li]

On Fri, Mar 9, 2018 at 4:31 PM 'Eugene Kirpichov' via Scio Users <scio-...@googlegroups.com> wrote:

Hi!

On Fri, Mar 9, 2018 at 1:22 PM Rafal Wojdyla <r...@spotify.com> wrote:

Hi all,

We have just released Scio 0.5.0. This is a major/breaking release - make sure to read the breaking changes section below.

- rav

https://github.com/spotify/scio/releases/tag/v0.5.0

"In ictu"

Breaking changes

• BigQueryIO in JobTest#output now requires a type parameter. Explicit .map(T.toTableRow)of test data is no longer needed.
• Typed AvroIO now accepts case classes instead of Avro records in JobTest. Explicit .map(T.toGenericRecord) of test data is no longer needed. See this change for more.
• Package com.spotify.scio.extra.transforms is moved from scio-extra to scio-core, under com.spotify.scio.transforms.

See this section for more details.

Features

• Support reading BigQuery as Avro #964, #992
• BigQuery client now supports load and export #1060
• Add TFRecordSpec support for Featran #1002
• Add AsyncLookupDoFn #1012

This looks like a very useful general-purpose tool, Beam users have asked for something like this many times; and the implementation looks very high-quality as well. Any interest in backporting it into Beam?

Sounds like a good idea. We'll look into it. Might take a while though since our tests are in scala.

https://github.com/spotify/scio/issues/1071 

 

• Bump sparkey to 2.2.1, protobuf-generic to 0.2.4 #1028
• Added ser/der support for joda DateTime #1038
• Password is now optional for jdbc connection #1040
• Add job cancellation option to ScioResult#waitUntilDone #1056 #1058 #1062 #1066

Bug fixes

• Fix transform name in joins #1034 #1035
• Add applyKvTransform #1020 #1032
• Add helpers to initialize counters #1026 #1027
• Fix SCollectionMatchers serialization #1001
• Check runner version #1008 #1009
• Log exception in AsyncLookupDoFn only if cache put fails #1039
• ProjectId nonEmpty string check in BigQueryClient #1045
• Fix SCollection#withSlidingWindows #1054

--
You received this message because you are subscribed to the Google Groups "Scio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scio-users+...@googlegroups.com.
To post to this group, send email to scio-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scio-users/CAFmTo4-7m9X9zH7pE1G0Mi-X3FuPCGFurkCh23ghdSaH48c-sA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.