2d Shape Dataset

[Nathen Paisley]

Ifeel like I am following this Keras training guide almost verbatim. I'm trying to use a tf.data.Dataset to train my model. Doing so raises an error involving the shape of the dataset; this error is not raised if I pass numpy arrays or pd.DataFrames.

Now, normally, I would be like "Well, duh, the shapes don't align." But my question is: (1) why and (2) how do I construct a tf.data.Dataset that I can pass for fitting? I am following the guide (linked above) almost verbatim and don't see any discrepancies in creating the Dataset object. It seems like the .fit call isn't correctly dividing the data from the labels (The data should be of shape 784, not (784,1), and the 1 seems like it's the label column.

SHAPES is a dataset of synthetic images designed to benchmark systems for understanding of spatial and logical relations among multiple objects. The dataset consists of complex questions about arrangements of colored shapes. The questions are built around compositions of concepts and relations, e.g. Is there a red shape above a circle? or Is a red shape blue?. Questions contain between two and four attributes, object types, or relationships. There are 244 questions and 15,616 images in total, with all questions having a yes and no answer (and corresponding supporting image). This eliminates the risk of learning biases.

I'm trying to create a batch dataset from a tensor dataset and having trouble with the shape.I saw some references to enqueue but a couple years out of date and the tf.dataset.batch doesn't have any options. My dataset looks like:

Datasets are very similar to NumPy arrays. They are homogeneous collections ofdata elements, with an immutable datatype and (hyper)rectangular shape.Unlike NumPy arrays, they support a variety of transparent storage featuressuch as compression, error-detection, and chunked I/O.

As with NumPy arrays, the len() of a dataset is the length of the firstaxis, and iterating over a dataset iterates over the first axis. However,modifications to the yielded data are not recorded in the file. Resizing adataset while iterating has undefined results.

Chunked data may be transformed by the HDF5 filter pipeline. The mostcommon use is applying transparent compression. Data is compressed on theway to disk, and automatically decompressed when read. Once the datasetis created with a particular compression filter applied, data may be readand written as normal with no special steps required.

In addition to the compression filters listed above, compression filters can bedynamically loaded by the underlying HDF5 library. This is done by passing afilter number to Group.create_dataset() as the compression parameter.The compression_opts parameter will then be passed to this filter.

For integer data, this specifies the number of bits to retain. Set to 0 to haveHDF5 automatically compute the number of bits required for lossless compressionof the chunk. For floating-point data, indicates the number of digits afterthe decimal point to retain.

Block-oriented compressors like GZIP or LZF work better when presented withruns of similar values. Enabling the shuffle filter rearranges the bytes inthe chunk and may improve compression ratio. No significant speed penalty,lossless.

The full H5Sselect_hyperslab API is exposed via the MultiBlockSlice object.This takes four elements to define the selection (start, count, stride andblock) in contrast to the built-in slice object, which takes three elements.A MultiBlockSlice can be used in place of a slice to select a number of (count)blocks of multiple elements separated by a stride, rather than a set of singleelements separated by a step.

HDF5 has the concept of Empty or Null datasets and attributes. These are notthe same as an array with a shape of (), or a scalar dataspace in HDF5 terms.Instead, it is a dataset with an associated type, no data, and no shape. Inh5py, we represent this as either a dataset with shape None, or aninstance of h5py.Empty. Empty datasets and attributes cannot be sliced.

The dtype of the dataset can be accessed via .dtype as per normal.As empty datasets cannot be sliced, some methods of datasets such asread_direct will raise a TypeError exception if used on a empty dataset.

Read from an HDF5 dataset directly into a NumPy array, which canavoid making an intermediate copy as happens with slicing. Thedestination array must be C-contiguous and writable, and must havea datatype to which the source data may be cast. Data type conversionwill be carried out on the fly by HDF5.

For each chunk within the given region, the iterator yields a tuple ofslices that gives the intersection of the given chunk with theselection area. This can be used to read or write data in thatchunk.

Integer giving the total number of bytes required to load the full dataset into RAM (i.e. dset[()]).This may not be the amount of disk space occupied by the dataset,as datasets may be compressed when written or only partly filled with data.This value also does not include the array overhead, as it only describes the size of the data itself.Thus the real amount of RAM occupied by this dataset may be slightly greater.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Early and accurate diagnosis of ear deformities in newborns is crucial for an effective non-surgical correction treatment, since this commonly seen ear anomalies would affect aesthetics and cause mental problems if untreated. It is not easy even for experienced physicians to diagnose the auricular deformities of newborns and the classification of the sub-types, because of the rich bio-metric features embedded in the ear shape. Machine learning has already been introduced to analyze the auricular shape. However, there is little publicly available datasets of ear images from newborns. We released a dataset that contains quality-controlled photos of 3,852 ears from 1,926 newborns. The dataset also contains medical diagnosis of the ear shape, and the health data of each newborn and its mother. Our aim is to provide a freely accessible dataset, which would facilitate researches related with ear anatomies, such as the AI-aided detection and classification of auricular deformities and medical risk analysis.

The substructures of the auricle and different types of ear deformities. (a) presents the morphology of a normal ear from one newborn. The ear contains multiple sub-structures. (b) shows examples of some sub-types of auricular deformities. The abnormal structures are marked with arrows. All the ears in this figure are from the dataset.

Analyzing the auricular shape with AI is not new to the community7. Many studies achieved to recognize the ear location from an image of human head8,9. It is possible to deduce the gender and age from the shape of the ears10,11. Actually, the bio-metric features of the auricle are so individually specific, that make the ear shape available as another fingerprint for human recognition12,13. Researches have also been conducted to identify the auricular deformities and evaluate the corrective effects using convolutional neural networks14,15.

To address the gap, this paper describes the first release of the BabyEar4k dataset, which contains the left and right ear photos from 1,926 newborns. These photos were obtained in a controlled clinical environment, all taken with the same hardware. The image size and the proportion of the ear were controlled in order to give sufficient resolution for the ear shape. Image quality assessment were conducted for each ear image. Moreover, the diagnosis (identification and classification of deformities) from two experienced physicians were provided. Some health data of the babies and their mothers were also included. We hope that the availability of this dataset will accelerate research in image-based, computer-aided analyzing of ear morphologies, especially the identification and classification of auricular deformities in newborns.

The study was approved by Ethics Committee of School of Basic Medical Sciences, Fudan University. The data publication was approved by the committee, and the parents were acknowledged and provided informed consent to the open publication of the anonymized data (including the ear images and the health data). Initially, a total of 2,000 newborns between November 2018 and April 2019 were included in this study. All data were collected with the permission of the parents, being aware of the publicly open of the ear images and medical data. Inclusion criterion was viable fetus. Exclusion criterion was defined as newborns who require intensive care due to severe premature birth or other reasons. All the babies were born and taken care at the Obstetrics & Gynecology Hospital, Fudan University.

Procedure of image processing and labelling in BabyEar4k dataset. (a) gives a sketch of the image processing procedure. The raw ear photographs were rotated, cropped, and labelled to obtain the final dataset. (b) demonstrates the three regions of interest of the pinna, and (c) shows the diagnosis procedure according to the division of the three regions.

The two experts blindly diagnosed the auricles following the above process, merely depending on the images. If the diagnostic results from the two experts for one ear are different, a further evaluation and discussion by these two experts were conducted to determine the final consensus.

The BabyEar4k dataset is open for public use, which is available at figshare19.The final released dataset includes 3,852 ear images of 1,926 newborns from 1903 mothers (including 23 pairs of twins), together with the diagnostic results from two experts, and the healthy data. The images and diagnostic results facilitate the training of neural networks for diagnosis of newborn auricular deformities. The health data may be beneficial to study the non-hereditary risk factors of the deformities. This dataset does not distinguish between the train set and the test set.

3a8082e126