Scio 0.8.0 released

7 views
Skip to first unread message

Neville Li

unread,
Jan 8, 2020, 1:03:42 PM1/8/20
to Scio Users, us...@beam.apache.org
Hi all,

We just released Scio 0.8.0. This is based on the most recent Beam 2.17.0 release and includes a lot of new features & bug fixes over the past 10 months.

Cheers,
Neville


"Amato Animo Animato Animagus"

Breaking changes & deprecations

  • See v0.8.0 Migration Guide for detailed instructions
  • Remove @experimental from transform #2537
  • Deprecate scio-elasticsearch2 and scio-cassandra2 #2414 #2421
  • Deprecate hashFilter #2442
  • Deprecate legacy components in scio-extras #2533

Features

  • Bump Beam to 2.17.0 #2577
  • Add sharded Sparkey support. #2336
  • Rework side input cache #2363
  • Cleanup Side Inputs API, introduce Singleton Set SideInputs #2424
  • Add schema support for GenericRecord #2514

Bug fixes & improvements

  • Add file:linenum only to outer transform #2405
  • Fix join transform names #2444
  • Remove Coder context bound for partitionByKey #2451
  • Rename that method argument in join functions to rhs #2466
  • Replace custom ClosureCleaner with chill's #2423Use chill's Externalizer to serialize predicates in SCollectionMatchers #2410
  • Add errmsg when beamOpts == null in JobTest, fix #2430 #2545
  • Add bigQuerySelect() method with default flattenResults value #2500
  • Better consistency around BigQuery API #2412Fail early on malformed BigQuery spec #2345
  • Rewrite typedBigQueryStorage #2434
  • Add DML query support to bigquery client #2418
  • Treat Avro array as java List in BigQuery read, fix #2068 #2415
  • Fix NPE in scio-bigtable's ChannelPoolCreator when credentials aren't set #2317Fix bigtable scollection ops return type #2486
  • Refactor PubsubIO for more typesafety #2457
  • Avoid Mutation coder fallback for Spanner #2478
  • Fix Parquet sink suffix #2367Improve iterable equality #2483
  • Improve back compat with Scio 0.7 #2401
  • Improve coder gen by checking companion implicits #2522
  • Make recursive coders serializable #2404
  • Remove kryo coder override in intermediate steps #2422Fix fallback warning when implicit is in scope #2511
  • Improve the schema compatibility error message #2366
  • Remove schema fallback #2489
  • Add Schemas support for more types #2364
  • Assert FileStorage.isDone in MaterializeTap #2518
  • Add support for cleaning up TF models on shutdown #2549
  • Rework TensorFlow predict ops #2343
  • Remove unused/deprecated TensorFlow graph DoFn #2339
  • Mark some APIs in scio-extras as experimental #2517 #2572
 

Robert Bradshaw

unread,
Jan 8, 2020, 2:28:27 PM1/8/20
to user, Scio Users
Nice!

Kenneth Knowles

unread,
Jan 9, 2020, 9:42:54 PM1/9/20
to user, Scio Users
So fast! Excellent.
Reply all
Reply to author
Forward
0 new messages