Is there a limit on column count for schemas?

[Adam Kjems]

I have a inferred schema that comes to field_651 in the column count. On trying to create a dataset based on this I get 

Unknown error: org.codehaus.jackson.JsonParseException: Unexpected end-of-input: Was expecting closing quote for a string value. 

I checked the json file using http://jsonlint.com/ and it is valid. 

If I cut the schema file down to only field_10 as in 10 columns it works fine.

Is there a limit on column count for schemas when create a dataset?

[Ryan Blue]

Adam,

There shouldn't be a limit. Could you re-run that command with -v to
give the full stack trace? If you can share your schema file, I can take
a look at that as well.

kite-dataset -v create ...

rb

--
Ryan Blue
Software Engineer
Cloudera, Inc.

[Ryan Blue]

Okay, it looks like you're hitting a Hive MetaStore issue. Notice that
the error message doesn't seem to match: "StringReader...: line 1,
column 6001" but the attached file isn't one line JSON blob on a single
line. The error is happening when Kite tries to load the schema back
from the metastore.

In older Kite versions (like the one included in the QuickStart VM you
are using) Kite stored the schema literal in a table property in the
Hive MetaStore. Unfortunately, that had a size limitation and would cut
off the schema and cause this problem.

There are 2 solutions:
1. Use a newer version of Kite that maintains a schema folder in HDFS
2. Create a schema folder in HFDS and point kite-dataset at that when
creating your table.

rb

On 10/13/2015 12:26 PM, Adam Kjems wrote:
> Ryan,
>
> Thanks for the quick reply. Attached is the schema the error with -v
>
> Cheers,
> Adam

> --
> You received this message because you are subscribed to the Google
> Groups "CDK Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdk-dev+u...@cloudera.org
> <mailto:cdk-dev+u...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.