Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.

Dismiss

[PATCH 00/20] i.MX Media Driver

479 views

Skip to first unread message

Steve Longerbeam

unread,

Dec 29, 2016, 5:30:06 PM12/29/16

This is a media driver for video capture on i.MX.

Refer to Documentation/media/v4l-drivers/imx.rst for example capture
pipelines on SabreSD, SabreAuto, and SabreLite reference platforms.

This patchset includes the OF graph layout as proposed by Philipp Zabel,
with only minor changes which are enumerated in the patch header.

Philipp Zabel (2):
ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
connections
media: imx: Add video switch subdev driver

Steve Longerbeam (18):
ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
ARM: dts: imx6qdl: add media device
ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabreauto: create i2cmux for i2c3
ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
gpio: pca953x: Add optional reset gpio control
media: Add i.MX media core driver
media: imx: Add CSI subdev driver
media: imx: Add SMFC subdev driver
media: imx: Add IC subdev drivers
media: imx: Add Camera Interface subdev driver
media: imx: Add MIPI CSI-2 Receiver subdev driver
media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
media: imx: Add Parallel OV5642 sensor subdev driver
ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

.../devicetree/bindings/gpio/gpio-pca953x.txt | 4 +
Documentation/devicetree/bindings/media/imx.txt | 205 +
Documentation/media/v4l-drivers/imx.rst | 429 ++
arch/arm/boot/dts/imx6dl-sabrelite.dts | 5 +
arch/arm/boot/dts/imx6dl-sabresd.dts | 5 +
arch/arm/boot/dts/imx6dl.dtsi | 183 +
arch/arm/boot/dts/imx6q-sabrelite.dts | 6 +
arch/arm/boot/dts/imx6q-sabresd.dts | 5 +
arch/arm/boot/dts/imx6q.dtsi | 123 +
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 142 +-
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 122 +-
arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 +-
arch/arm/boot/dts/imx6qdl.dtsi | 25 +-
arch/arm/configs/imx_v6_v7_defconfig | 10 +-
drivers/gpio/gpio-pca953x.c | 17 +
drivers/staging/media/Kconfig | 2 +
drivers/staging/media/Makefile | 1 +
drivers/staging/media/imx/Kconfig | 36 +
drivers/staging/media/imx/Makefile | 16 +
drivers/staging/media/imx/TODO | 18 +
drivers/staging/media/imx/imx-camif.c | 1011 +++++
drivers/staging/media/imx/imx-csi.c | 638 +++
drivers/staging/media/imx/imx-ic-common.c | 113 +
drivers/staging/media/imx/imx-ic-pp.c | 636 +++
drivers/staging/media/imx/imx-ic-prpenc.c | 1037 +++++
drivers/staging/media/imx/imx-ic-prpvf.c | 1181 ++++++
drivers/staging/media/imx/imx-ic.h | 36 +
drivers/staging/media/imx/imx-media-common.c | 981 +++++
drivers/staging/media/imx/imx-media-dev.c | 479 +++
drivers/staging/media/imx/imx-media-fim.c | 508 +++
drivers/staging/media/imx/imx-media-internal-sd.c | 456 ++
drivers/staging/media/imx/imx-media-of.c | 291 ++
drivers/staging/media/imx/imx-media-of.h | 25 +
drivers/staging/media/imx/imx-media.h | 290 ++
drivers/staging/media/imx/imx-mipi-csi2.c | 509 +++
drivers/staging/media/imx/imx-smfc.c | 739 ++++
drivers/staging/media/imx/imx-video-switch.c | 349 ++
drivers/staging/media/imx/ov5640-mipi.c | 2349 +++++++++++
drivers/staging/media/imx/ov5642.c | 4364 ++++++++++++++++++++
include/media/imx.h | 15 +
include/uapi/Kbuild | 1 +
include/uapi/linux/v4l2-controls.h | 4 +
include/uapi/media/Kbuild | 2 +
include/uapi/media/imx.h | 30 +
44 files changed, 17484 insertions(+), 28 deletions(-)
create mode 100644 Documentation/devicetree/bindings/media/imx.txt
create mode 100644 Documentation/media/v4l-drivers/imx.rst
create mode 100644 drivers/staging/media/imx/Kconfig
create mode 100644 drivers/staging/media/imx/Makefile
create mode 100644 drivers/staging/media/imx/TODO
create mode 100644 drivers/staging/media/imx/imx-camif.c
create mode 100644 drivers/staging/media/imx/imx-csi.c
create mode 100644 drivers/staging/media/imx/imx-ic-common.c
create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
create mode 100644 drivers/staging/media/imx/imx-ic.h
create mode 100644 drivers/staging/media/imx/imx-media-common.c
create mode 100644 drivers/staging/media/imx/imx-media-dev.c
create mode 100644 drivers/staging/media/imx/imx-media-fim.c
create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
create mode 100644 drivers/staging/media/imx/imx-media-of.c
create mode 100644 drivers/staging/media/imx/imx-media-of.h
create mode 100644 drivers/staging/media/imx/imx-media.h
create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
create mode 100644 drivers/staging/media/imx/imx-smfc.c
create mode 100644 drivers/staging/media/imx/imx-video-switch.c
create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
create mode 100644 drivers/staging/media/imx/ov5642.c
create mode 100644 include/media/imx.h
create mode 100644 include/uapi/media/Kbuild
create mode 100644 include/uapi/media/imx.h

--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:30:06 PM12/29/16

Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
for sensors with a MIPI CSI2 interface.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---
drivers/staging/media/imx/Makefile | 1 +
drivers/staging/media/imx/imx-mipi-csi2.c | 509 ++++++++++++++++++++++++++++++
2 files changed, 510 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index fe9e992..0decef7 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
new file mode 100644
index 0000000..84df16e
--- /dev/null
+++ b/drivers/staging/media/imx/imx-mipi-csi2.c
@@ -0,0 +1,509 @@
+/*
+ * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
+ *
+ * Copyright (c) 2012-2014 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/export.h>
+#include <linux/types.h>
+#include <linux/init.h>
+#include <linux/platform_device.h>
+#include <linux/err.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/clk.h>
+#include <linux/list.h>
+#include <linux/irq.h>
+#include <linux/of_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <asm/mach/irq.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+/*
+ * there must be 5 pads: 1 input pad from sensor, and
+ * the 4 virtual channel output pads
+ */
+#define CSI2_NUM_SINK_PADS 1
+#define CSI2_NUM_SRC_PADS 4
+#define CSI2_NUM_PADS 5
+
+struct imxcsi2_dev {
+ struct device *dev;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;
+ struct media_pad pad[CSI2_NUM_PADS];
+ struct v4l2_mbus_framefmt format_mbus;
+ struct v4l2_subdev *src_sd;
+ struct v4l2_subdev *sink_sd[CSI2_NUM_SRC_PADS];
+ int input_pad;
+ struct clk *dphy_clk;
+ struct clk *cfg_clk;
+ struct clk *pix_clk; /* what is this? */
+ void __iomem *base;
+ int intr1;
+ int intr2;
+ struct v4l2_of_bus_mipi_csi2 bus;
+ bool on;
+ bool stream_on;
+};
+
+#define DEVICE_NAME "imx-mipi-csi2"
+
+/* Register offsets */
+#define CSI2_VERSION 0x000
+#define CSI2_N_LANES 0x004
+#define CSI2_PHY_SHUTDOWNZ 0x008
+#define CSI2_DPHY_RSTZ 0x00c
+#define CSI2_RESETN 0x010
+#define CSI2_PHY_STATE 0x014
+#define CSI2_DATA_IDS_1 0x018
+#define CSI2_DATA_IDS_2 0x01c
+#define CSI2_ERR1 0x020
+#define CSI2_ERR2 0x024
+#define CSI2_MSK1 0x028
+#define CSI2_MSK2 0x02c
+#define CSI2_PHY_TST_CTRL0 0x030
+#define CSI2_PHY_TST_CTRL1 0x034
+#define CSI2_SFT_RESET 0xf00
+
+static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
+{
+ return container_of(sdev, struct imxcsi2_dev, sd);
+}
+
+static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
+{
+ return readl(csi2->base + regoff);
+}
+
+static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
+ unsigned int regoff)
+{
+ writel(val, csi2->base + regoff);
+}
+
+static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
+{
+ int lanes = csi2->bus.num_data_lanes;
+
+ imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
+}
+
+static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
+{
+ if (enable) {
+ imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
+ imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
+ imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
+ } else {
+ imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
+ imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
+ imxcsi2_write(csi2, 0x0, CSI2_RESETN);
+ }
+}
+
+static void imxcsi2_reset(struct imxcsi2_dev *csi2)
+{
+ imxcsi2_enable(csi2, false);
+
+ imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
+ imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+
+ imxcsi2_enable(csi2, true);
+}
+
+static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
+{
+ u32 reg;
+ int i;
+
+ /* wait for mipi sensor ready */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+ if (reg != 0x200)
+ break;
+ usleep_range(10000, 20000);
+ }
+
+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for clock lane timeout, phy_state = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ /* wait for mipi stable */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_ERR1);
+ if (reg == 0x0)
+ break;
+ usleep_range(10000, 20000);
+ }
+
+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for controller timeout, err1 = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ /* finally let's wait for active clock on the clock lane */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+ if (reg & (1 << 8))
+ break;
+ usleep_range(10000, 20000);
+ }
+
+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for active clock timeout, phy_state = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
+ imxcsi2_read(csi2, CSI2_VERSION));
+
+ return 0;
+}
+
+/*
+ * V4L2 subdev operations
+ */
+
+static int imxcsi2_link_setup(struct media_entity *entity,
+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+ struct v4l2_subdev *remote_sd;
+
+ dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,
+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (csi2->sink_sd[local->index])
+ return -EBUSY;
+ csi2->sink_sd[local->index] = remote_sd;
+ } else {
+ csi2->sink_sd[local->index] = NULL;
+ }
+ } else {
+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (csi2->src_sd)
+ return -EBUSY;
+ csi2->src_sd = remote_sd;
+ } else {
+ csi2->src_sd = NULL;
+ }
+ }
+
+ return 0;
+}
+
+static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ if (on && !csi2->on) {
+ v4l2_info(&csi2->sd, "power ON\n");
+ clk_prepare_enable(csi2->cfg_clk);
+ clk_prepare_enable(csi2->dphy_clk);
+ imxcsi2_set_lanes(csi2);
+ imxcsi2_reset(csi2);
+ } else if (!on && csi2->on) {
+ v4l2_info(&csi2->sd, "power OFF\n");
+ imxcsi2_enable(csi2, false);
+ clk_disable_unprepare(csi2->dphy_clk);
+ clk_disable_unprepare(csi2->cfg_clk);
+ }
+
+ csi2->on = on;
+ return 0;
+}
+
+static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+ int i, ret = 0;
+
+ if (!csi2->src_sd)
+ return -EPIPE;
+ for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
+ if (csi2->sink_sd[i])
+ break;
+ }
+ if (i >= CSI2_NUM_SRC_PADS)
+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !csi2->stream_on) {
+ clk_prepare_enable(csi2->pix_clk);
+ ret = imxcsi2_dphy_wait(csi2);
+ if (ret)
+ clk_disable_unprepare(csi2->pix_clk);
+ } else if (!enable && csi2->stream_on) {
+ clk_disable_unprepare(csi2->pix_clk);
+ }
+
+ if (!ret)
+ csi2->stream_on = enable;
+ return ret;
+}
+
+static int imxcsi2_get_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ sdformat->format = csi2->format_mbus;
+
+ return 0;
+}
+
+static int imxcsi2_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ if (sdformat->pad >= CSI2_NUM_PADS)
+ return -EINVAL;
+
+ if (csi2->stream_on)
+ return -EBUSY;
+
+ /* Output pads mirror active input pad, no limits on input pads */
+ if (sdformat->pad != csi2->input_pad)
+ sdformat->format = csi2->format_mbus;
+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY)
+ cfg->try_fmt = sdformat->format;
+ else
+ csi2->format_mbus = sdformat->format;
+
+ return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int imxcsi2_registered(struct v4l2_subdev *sd)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+
+ /* get media device */
+ csi2->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(csi2->md, sd);
+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 4)
+ return -EINVAL;
+
+ for (i = 0; i < CSI2_NUM_PADS; i++) {
+ pad = &imxsd->pad[i];
+ csi2->pad[i] = pad->pad;
+ if (csi2->pad[i].flags & MEDIA_PAD_FL_SINK)
+ csi2->input_pad = i;
+ }
+
+ /* set a default mbus format */
+ ret = imx_media_init_mbus_fmt(&csi2->format_mbus,
+ 640, 480, 0, V4L2_FIELD_NONE, NULL);
+ if (ret)
+ return ret;
+
+ return media_entity_pads_init(&sd->entity, CSI2_NUM_PADS, csi2->pad);
+}
+
+static struct media_entity_operations imxcsi2_entity_ops = {
+ .link_setup = imxcsi2_link_setup,
+ .link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops imxcsi2_core_ops = {
+ .s_power = imxcsi2_s_power,
+};
+
+static struct v4l2_subdev_video_ops imxcsi2_video_ops = {
+ .s_stream = imxcsi2_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imxcsi2_pad_ops = {
+ .get_fmt = imxcsi2_get_fmt,
+ .set_fmt = imxcsi2_set_fmt,
+};
+
+static struct v4l2_subdev_ops imxcsi2_subdev_ops = {
+ .core = &imxcsi2_core_ops,
+ .video = &imxcsi2_video_ops,
+ .pad = &imxcsi2_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imxcsi2_internal_ops = {
+ .registered = imxcsi2_registered,
+};
+
+static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
+{
+ struct device_node *node = csi2->dev->of_node;
+ struct device_node *epnode;
+ struct v4l2_of_endpoint ep;
+ int ret = 0;
+
+ epnode = of_graph_get_next_endpoint(node, NULL);
+ if (!epnode) {
+ v4l2_err(&csi2->sd, "failed to get endpoint node\n");
+ return -EINVAL;
+ }
+
+ v4l2_of_parse_endpoint(epnode, &ep);
+ if (ep.bus_type != V4L2_MBUS_CSI2) {
+ v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");
+ ret = -EINVAL;
+ goto out;
+ }
+
+ csi2->bus = ep.bus.mipi_csi2;
+
+ v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
+ v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);
+out:
+ of_node_put(epnode);
+ return ret;
+}
+
+static int imxcsi2_probe(struct platform_device *pdev)
+{
+ struct imxcsi2_dev *csi2;
+ struct resource *res;
+ int ret;
+
+ csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+ if (!csi2)
+ return -ENOMEM;
+
+ csi2->dev = &pdev->dev;
+
+ v4l2_subdev_init(&csi2->sd, &imxcsi2_subdev_ops);
+ v4l2_set_subdevdata(&csi2->sd, &pdev->dev);
+ csi2->sd.internal_ops = &imxcsi2_internal_ops;
+ csi2->sd.entity.ops = &imxcsi2_entity_ops;
+ csi2->sd.dev = &pdev->dev;
+ csi2->sd.owner = THIS_MODULE;
+ csi2->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+ strcpy(csi2->sd.name, DEVICE_NAME);
+ /* FIXME: this is a mipi-csi2 receiver, function isn't right */
+ csi2->sd.entity.function = MEDIA_ENT_F_ATV_DECODER;
+ csi2->sd.grp_id = IMX_MEDIA_GRP_ID_CSI2;
+
+ ret = imxcsi2_parse_endpoints(csi2);
+ if (ret)
+ return ret;
+
+ csi2->cfg_clk = devm_clk_get(&pdev->dev, "cfg_clk");
+ if (IS_ERR(csi2->cfg_clk)) {
+ v4l2_err(&csi2->sd, "failed to get cfg clock\n");
+ ret = PTR_ERR(csi2->cfg_clk);
+ return ret;
+ }
+
+ csi2->dphy_clk = devm_clk_get(&pdev->dev, "dphy_clk");
+ if (IS_ERR(csi2->dphy_clk)) {
+ v4l2_err(&csi2->sd, "failed to get dphy clock\n");
+ ret = PTR_ERR(csi2->dphy_clk);
+ return ret;
+ }
+
+ csi2->pix_clk = devm_clk_get(&pdev->dev, "pix_clk");
+ if (IS_ERR(csi2->pix_clk)) {
+ v4l2_err(&csi2->sd, "failed to get pixel clock\n");
+ ret = PTR_ERR(csi2->pix_clk);
+ return ret;
+ }
+
+ csi2->intr1 = platform_get_irq(pdev, 0);
+ csi2->intr2 = platform_get_irq(pdev, 1);
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+
+ if (!res || csi2->intr1 < 0 || csi2->intr2 < 0) {
+ v4l2_err(&csi2->sd, "failed to get platform resources\n");
+ return -ENODEV;
+ }
+
+ csi2->base = devm_ioremap(&pdev->dev, res->start, PAGE_SIZE);
+ if (!csi2->base) {
+ v4l2_err(&csi2->sd, "failed to map CSI-2 registers\n");
+ return -ENOMEM;
+ }
+
+ platform_set_drvdata(pdev, &csi2->sd);
+
+ return v4l2_async_register_subdev(&csi2->sd);
+}
+
+static int imxcsi2_remove(struct platform_device *pdev)
+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ imxcsi2_s_power(sd, 0);
+
+ v4l2_async_unregister_subdev(&csi2->sd);
+ media_entity_cleanup(&csi2->sd.entity);
+ v4l2_device_unregister_subdev(sd);
+
+ return 0;
+}
+
+static const struct of_device_id imxcsi2_dt_ids[] = {
+ { .compatible = "fsl,imx-mipi-csi2", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
+
+static struct platform_driver imxcsi2_driver = {
+ .driver = {
+ .name = DEVICE_NAME,
+ .owner = THIS_MODULE,
+ .of_match_table = imxcsi2_dt_ids,
+ },
+ .probe = imxcsi2_probe,
+ .remove = imxcsi2_remove,
+};
+
+module_platform_driver(imxcsi2_driver);
+
+MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");
+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");
+
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:30:06 PM12/29/16

Enable imx v4l2 staging drivers. For video capture on
the SabreAuto, the ADV7180 video decoder also requires the
i2c-mux-gpio and the max7310 port expander. The Sabrelite
requires PWM clocks for the OV5640.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/configs/imx_v6_v7_defconfig | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig
index cbe7faf..5da4d8e 100644
--- a/arch/arm/configs/imx_v6_v7_defconfig
+++ b/arch/arm/configs/imx_v6_v7_defconfig
@@ -51,6 +51,7 @@ CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_AEABI=y
CONFIG_HIGHMEM=y
CONFIG_CMA=y
+CONFIG_FORCE_MAX_ZONEORDER=14
CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"
CONFIG_KEXEC=y
CONFIG_CPU_FREQ=y
@@ -181,6 +182,7 @@ CONFIG_SERIAL_FSL_LPUART=y
CONFIG_SERIAL_FSL_LPUART_CONSOLE=y
# CONFIG_I2C_COMPAT is not set
CONFIG_I2C_CHARDEV=y
+CONFIG_I2C_MUX=y
CONFIG_I2C_MUX_GPIO=y
# CONFIG_I2C_HELPER_AUTO is not set
CONFIG_I2C_ALGOPCF=m
@@ -194,11 +196,11 @@ CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_MC9S08DZ60=y
CONFIG_GPIO_PCA953X=y
CONFIG_GPIO_STMPE=y
-CONFIG_POWER_SUPPLY=y
CONFIG_POWER_RESET=y
CONFIG_POWER_RESET_IMX=y
CONFIG_POWER_RESET_SYSCON=y
CONFIG_POWER_RESET_SYSCON_POWEROFF=y
+CONFIG_POWER_SUPPLY=y
CONFIG_SENSORS_GPIO_FAN=y
CONFIG_SENSORS_IIO_HWMON=y
CONFIG_THERMAL=y
@@ -221,6 +223,8 @@ CONFIG_REGULATOR_PFUZE100=y
CONFIG_MEDIA_SUPPORT=y
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_RC_SUPPORT=y
+CONFIG_MEDIA_CONTROLLER=y
+CONFIG_VIDEO_V4L2_SUBDEV_API=y
CONFIG_RC_DEVICES=y
CONFIG_IR_GPIO_CIR=y
CONFIG_MEDIA_USB_SUPPORT=y
@@ -229,6 +233,8 @@ CONFIG_V4L_PLATFORM_DRIVERS=y
CONFIG_SOC_CAMERA=y
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_CODA=y
+# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
+CONFIG_VIDEO_ADV7180=m
CONFIG_SOC_CAMERA_OV2640=y
CONFIG_IMX_IPUV3_CORE=y
CONFIG_DRM=y
@@ -338,6 +344,8 @@ CONFIG_FSL_EDMA=y
CONFIG_IMX_SDMA=y
CONFIG_MXS_DMA=y
CONFIG_STAGING=y
+CONFIG_STAGING_MEDIA=y
+CONFIG_COMMON_CLK_PWM=y
CONFIG_IIO=y
CONFIG_VF610_ADC=y
CONFIG_MPL3115=y
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:30:07 PM12/29/16

This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Kconfig | 8 +
drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/ov5640-mipi.c | 2349 +++++++++++++++++++++++++++++++
3 files changed, 2359 insertions(+)
create mode 100644 drivers/staging/media/imx/ov5640-mipi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index ce2d2c8..09f373d 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
---help---
A video4linux camera capture driver for i.MX5/6.

+config IMX_OV5640_MIPI
+ tristate "OmniVision OV5640 MIPI CSI-2 camera support"
+ depends on GPIOLIB && VIDEO_IMX_CAMERA
+ select IMX_MIPI_CSI2
+ default y
+ ---help---
+ MIPI CSI-2 OV5640 Camera support.
+
endmenu
endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index e3d6d8d..f96e623 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -11,3 +11,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-video-switch.o
+
+obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
new file mode 100644
index 0000000..a83b581
--- /dev/null
+++ b/drivers/staging/media/imx/ov5640-mipi.c
@@ -0,0 +1,2349 @@
+/*
+ * Copyright (c) 2014 Mentor Graphics Inc.
+ * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.

+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/ctype.h>
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/of_device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/regulator/consumer.h>
+#include <linux/clk.h>
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+
+#define OV5640_VOLTAGE_ANALOG 2800000
+#define OV5640_VOLTAGE_DIGITAL_CORE 1500000
+#define OV5640_VOLTAGE_DIGITAL_IO 1800000
+
+#define MIN_FPS 15
+#define MAX_FPS 30
+#define DEFAULT_FPS 30
+
+/* min/typical/max system clock (xclk) frequencies */
+#define OV5640_XCLK_MIN 6000000
+#define OV5640_XCLK_TYP 24000000
+#define OV5640_XCLK_MAX 54000000
+
+/* min/typical/max pixel clock (mclk) frequencies */
+#define OV5640_MCLK_MIN 48000000
+#define OV5640_MCLK_TYP 48000000
+#define OV5640_MCLK_MAX 96000000
+
+#define OV5640_CHIP_ID 0x300A
+#define OV5640_SLAVE_ID 0x3100
+#define OV5640_DEFAULT_SLAVE_ID 0x3c
+
+#define OV5640_MAX_CONTROLS 64
+
+enum ov5640_mode {
+ ov5640_mode_MIN = 0,
+ ov5640_mode_QCIF_176_144 = 0,
+ ov5640_mode_QVGA_320_240,
+ ov5640_mode_VGA_640_480,
+ ov5640_mode_NTSC_720_480,
+ ov5640_mode_PAL_720_576,
+ ov5640_mode_XGA_1024_768,
+ ov5640_mode_720P_1280_720,
+ ov5640_mode_1080P_1920_1080,
+ ov5640_mode_QSXGA_2592_1944,
+ ov5640_num_modes,
+ ov5640_mode_INIT = 0xff, /*only for sensor init*/
+};
+
+enum ov5640_frame_rate {
+ ov5640_15_fps,
+ ov5640_30_fps
+};
+
+static int ov5640_framerates[] = {
+ [ov5640_15_fps] = 15,
+ [ov5640_30_fps] = 30,
+};
+#define ov5640_num_framerates ARRAY_SIZE(ov5640_framerates)
+
+/* image size under 1280 * 960 are SUBSAMPLING
+ * image size upper 1280 * 960 are SCALING
+ */
+enum ov5640_downsize_mode {
+ SUBSAMPLING,
+ SCALING,
+};
+
+struct reg_value {
+ u16 reg_addr;
+ u8 val;
+ u8 mask;
+ u32 delay_ms;
+};
+
+struct ov5640_mode_info {
+ enum ov5640_mode mode;
+ enum ov5640_downsize_mode dn_mode;
+ u32 width;
+ u32 height;
+ struct reg_value *init_data_ptr;
+ u32 init_data_size;
+};
+
+struct ov5640_dev {
+ struct i2c_client *i2c_client;
+ struct device *dev;
+ struct v4l2_subdev sd;
+ struct media_pad pad;
+ struct v4l2_ctrl_handler ctrl_hdl;
+ struct v4l2_of_endpoint ep; /* the parsed DT endpoint info */
+ struct v4l2_mbus_framefmt fmt;
+ struct v4l2_captureparm streamcap;
+ struct clk *xclk; /* system clock to OV5640 */
+ int xclk_freq; /* requested xclk freq from devicetree */
+
+ enum ov5640_mode current_mode;
+ enum ov5640_frame_rate current_fr;
+
+ bool on;
+ bool awb_on;
+ bool agc_on;
+
+ /* cached control settings */
+ int ctrl_cache[OV5640_MAX_CONTROLS];
+
+ struct gpio_desc *reset_gpio;
+ struct gpio_desc *pwdn_gpio;
+ struct gpio_desc *gp_gpio;
+
+ int prev_sysclk, prev_hts;
+ int ae_low, ae_high, ae_target;
+
+ struct regulator *io_regulator;
+ struct regulator *core_regulator;
+ struct regulator *analog_regulator;
+ struct regulator *gpo_regulator;
+};
+
+static inline struct ov5640_dev *to_ov5640_dev(struct v4l2_subdev *sd)
+{
+ return container_of(sd, struct ov5640_dev, sd);
+}
+
+static inline struct ov5640_dev *ctrl_to_ov5640_dev(struct v4l2_ctrl *ctrl)
+{
+ return container_of(ctrl->handler, struct ov5640_dev, ctrl_hdl);
+}
+
+struct ov5640_control {
+ struct v4l2_queryctrl ctrl;
+ int (*set)(struct ov5640_dev *sensor, int value);
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable);
+static void ov5640_reset(struct ov5640_dev *sensor);
+static int ov5640_restore_ctrls(struct ov5640_dev *sensor);
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value);
+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value);
+static int ov5640_get_exposure(struct ov5640_dev *sensor);
+static int ov5640_set_gain(struct ov5640_dev *sensor, int value);
+static int ov5640_get_gain(struct ov5640_dev *sensor);
+
+static struct reg_value ov5640_init_setting_30fps_VGA[] = {
+
+ {0x3103, 0x11, 0, 0}, {0x3008, 0x82, 0, 5}, {0x3008, 0x42, 0, 0},
+ {0x3103, 0x03, 0, 0}, {0x3017, 0x00, 0, 0}, {0x3018, 0x00, 0, 0},
+ {0x3034, 0x18, 0, 0}, {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0},
+ {0x3037, 0x13, 0, 0}, {0x3108, 0x01, 0, 0}, {0x3630, 0x36, 0, 0},
+ {0x3631, 0x0e, 0, 0}, {0x3632, 0xe2, 0, 0}, {0x3633, 0x12, 0, 0},
+ {0x3621, 0xe0, 0, 0}, {0x3704, 0xa0, 0, 0}, {0x3703, 0x5a, 0, 0},
+ {0x3715, 0x78, 0, 0}, {0x3717, 0x01, 0, 0}, {0x370b, 0x60, 0, 0},
+ {0x3705, 0x1a, 0, 0}, {0x3905, 0x02, 0, 0}, {0x3906, 0x10, 0, 0},
+ {0x3901, 0x0a, 0, 0}, {0x3731, 0x12, 0, 0}, {0x3600, 0x08, 0, 0},
+ {0x3601, 0x33, 0, 0}, {0x302d, 0x60, 0, 0}, {0x3620, 0x52, 0, 0},
+ {0x371b, 0x20, 0, 0}, {0x471c, 0x50, 0, 0}, {0x3a13, 0x43, 0, 0},
+ {0x3a18, 0x00, 0, 0}, {0x3a19, 0xf8, 0, 0}, {0x3635, 0x13, 0, 0},
+ {0x3636, 0x03, 0, 0}, {0x3634, 0x40, 0, 0}, {0x3622, 0x01, 0, 0},
+ {0x3c01, 0xa4, 0, 0}, {0x3c04, 0x28, 0, 0}, {0x3c05, 0x98, 0, 0},
+ {0x3c06, 0x00, 0, 0}, {0x3c07, 0x08, 0, 0}, {0x3c08, 0x00, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
+ {0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
+ {0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
+ {0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
+ {0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
+ {0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
+ {0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
+ {0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
+ {0x5187, 0x09, 0, 0}, {0x5188, 0x09, 0, 0}, {0x5189, 0x88, 0, 0},
+ {0x518a, 0x54, 0, 0}, {0x518b, 0xee, 0, 0}, {0x518c, 0xb2, 0, 0},
+ {0x518d, 0x50, 0, 0}, {0x518e, 0x34, 0, 0}, {0x518f, 0x6b, 0, 0},
+ {0x5190, 0x46, 0, 0}, {0x5191, 0xf8, 0, 0}, {0x5192, 0x04, 0, 0},
+ {0x5193, 0x70, 0, 0}, {0x5194, 0xf0, 0, 0}, {0x5195, 0xf0, 0, 0},
+ {0x5196, 0x03, 0, 0}, {0x5197, 0x01, 0, 0}, {0x5198, 0x04, 0, 0},
+ {0x5199, 0x6c, 0, 0}, {0x519a, 0x04, 0, 0}, {0x519b, 0x00, 0, 0},
+ {0x519c, 0x09, 0, 0}, {0x519d, 0x2b, 0, 0}, {0x519e, 0x38, 0, 0},
+ {0x5381, 0x1e, 0, 0}, {0x5382, 0x5b, 0, 0}, {0x5383, 0x08, 0, 0},
+ {0x5384, 0x0a, 0, 0}, {0x5385, 0x7e, 0, 0}, {0x5386, 0x88, 0, 0},
+ {0x5387, 0x7c, 0, 0}, {0x5388, 0x6c, 0, 0}, {0x5389, 0x10, 0, 0},
+ {0x538a, 0x01, 0, 0}, {0x538b, 0x98, 0, 0}, {0x5300, 0x08, 0, 0},
+ {0x5301, 0x30, 0, 0}, {0x5302, 0x10, 0, 0}, {0x5303, 0x00, 0, 0},
+ {0x5304, 0x08, 0, 0}, {0x5305, 0x30, 0, 0}, {0x5306, 0x08, 0, 0},
+ {0x5307, 0x16, 0, 0}, {0x5309, 0x08, 0, 0}, {0x530a, 0x30, 0, 0},
+ {0x530b, 0x04, 0, 0}, {0x530c, 0x06, 0, 0}, {0x5480, 0x01, 0, 0},
+ {0x5481, 0x08, 0, 0}, {0x5482, 0x14, 0, 0}, {0x5483, 0x28, 0, 0},
+ {0x5484, 0x51, 0, 0}, {0x5485, 0x65, 0, 0}, {0x5486, 0x71, 0, 0},
+ {0x5487, 0x7d, 0, 0}, {0x5488, 0x87, 0, 0}, {0x5489, 0x91, 0, 0},
+ {0x548a, 0x9a, 0, 0}, {0x548b, 0xaa, 0, 0}, {0x548c, 0xb8, 0, 0},
+ {0x548d, 0xcd, 0, 0}, {0x548e, 0xdd, 0, 0}, {0x548f, 0xea, 0, 0},
+ {0x5490, 0x1d, 0, 0}, {0x5580, 0x02, 0, 0}, {0x5583, 0x40, 0, 0},
+ {0x5584, 0x10, 0, 0}, {0x5589, 0x10, 0, 0}, {0x558a, 0x00, 0, 0},
+ {0x558b, 0xf8, 0, 0}, {0x5800, 0x23, 0, 0}, {0x5801, 0x14, 0, 0},
+ {0x5802, 0x0f, 0, 0}, {0x5803, 0x0f, 0, 0}, {0x5804, 0x12, 0, 0},
+ {0x5805, 0x26, 0, 0}, {0x5806, 0x0c, 0, 0}, {0x5807, 0x08, 0, 0},
+ {0x5808, 0x05, 0, 0}, {0x5809, 0x05, 0, 0}, {0x580a, 0x08, 0, 0},
+ {0x580b, 0x0d, 0, 0}, {0x580c, 0x08, 0, 0}, {0x580d, 0x03, 0, 0},
+ {0x580e, 0x00, 0, 0}, {0x580f, 0x00, 0, 0}, {0x5810, 0x03, 0, 0},
+ {0x5811, 0x09, 0, 0}, {0x5812, 0x07, 0, 0}, {0x5813, 0x03, 0, 0},
+ {0x5814, 0x00, 0, 0}, {0x5815, 0x01, 0, 0}, {0x5816, 0x03, 0, 0},
+ {0x5817, 0x08, 0, 0}, {0x5818, 0x0d, 0, 0}, {0x5819, 0x08, 0, 0},
+ {0x581a, 0x05, 0, 0}, {0x581b, 0x06, 0, 0}, {0x581c, 0x08, 0, 0},
+ {0x581d, 0x0e, 0, 0}, {0x581e, 0x29, 0, 0}, {0x581f, 0x17, 0, 0},
+ {0x5820, 0x11, 0, 0}, {0x5821, 0x11, 0, 0}, {0x5822, 0x15, 0, 0},
+ {0x5823, 0x28, 0, 0}, {0x5824, 0x46, 0, 0}, {0x5825, 0x26, 0, 0},
+ {0x5826, 0x08, 0, 0}, {0x5827, 0x26, 0, 0}, {0x5828, 0x64, 0, 0},
+ {0x5829, 0x26, 0, 0}, {0x582a, 0x24, 0, 0}, {0x582b, 0x22, 0, 0},
+ {0x582c, 0x24, 0, 0}, {0x582d, 0x24, 0, 0}, {0x582e, 0x06, 0, 0},
+ {0x582f, 0x22, 0, 0}, {0x5830, 0x40, 0, 0}, {0x5831, 0x42, 0, 0},
+ {0x5832, 0x24, 0, 0}, {0x5833, 0x26, 0, 0}, {0x5834, 0x24, 0, 0},
+ {0x5835, 0x22, 0, 0}, {0x5836, 0x22, 0, 0}, {0x5837, 0x26, 0, 0},
+ {0x5838, 0x44, 0, 0}, {0x5839, 0x24, 0, 0}, {0x583a, 0x26, 0, 0},
+ {0x583b, 0x28, 0, 0}, {0x583c, 0x42, 0, 0}, {0x583d, 0xce, 0, 0},
+ {0x5025, 0x00, 0, 0}, {0x3a0f, 0x30, 0, 0}, {0x3a10, 0x28, 0, 0},
+ {0x3a1b, 0x30, 0, 0}, {0x3a1e, 0x26, 0, 0}, {0x3a11, 0x60, 0, 0},
+ {0x3a1f, 0x14, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3c00, 0x04, 0, 300},
+};
+
+static struct reg_value ov5640_setting_30fps_VGA_640_480[] = {
+
+ {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_VGA_640_480[] = {
+ {0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_XGA_1024_768[] = {
+
+ {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x04, 0, 0}, {0x380f, 0x38, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x0e, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3503, 0x00, 0, 0},
+ {0x3808, 0x04, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0},
+ {0x380b, 0x00, 0, 0}, {0x3035, 0x12, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_XGA_1024_768[] = {
+ {0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0x80, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x3808, 0x04, 0, 0},
+ {0x3809, 0x00, 0, 0}, {0x380a, 0x03, 0, 0}, {0x380b, 0x00, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QVGA_320_240[] = {
+ {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+ {0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QVGA_320_240[] = {
+ {0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x01, 0, 0}, {0x3809, 0x40, 0, 0}, {0x380a, 0x00, 0, 0},
+ {0x380b, 0xf0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_QCIF_176_144[] = {
+ {0x3035, 0x14, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+ {0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+static struct reg_value ov5640_setting_15fps_QCIF_176_144[] = {
+ {0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x00, 0, 0}, {0x3809, 0xb0, 0, 0}, {0x380a, 0x00, 0, 0},
+ {0x380b, 0x90, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_NTSC_720_480[] = {
+ {0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_NTSC_720_480[] = {
+ {0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x01, 0, 0},
+ {0x380b, 0xe0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x3c, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_PAL_720_576[] = {
+ {0x3035, 0x12, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+ {0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_PAL_720_576[] = {
+ {0x3035, 0x22, 0, 0}, {0x3036, 0x38, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x04, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9b, 0, 0},
+ {0x3808, 0x02, 0, 0}, {0x3809, 0xd0, 0, 0}, {0x380a, 0x02, 0, 0},
+ {0x380b, 0x40, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x68, 0, 0},
+ {0x380e, 0x03, 0, 0}, {0x380f, 0xd8, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x38, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x06, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0xa3, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_720P_1280_720[] = {
+ {0x3008, 0x42, 0, 0},
+ {0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+ {0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+ {0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+ {0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+ {0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+ {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+ {0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+ {0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0}, {0x4005, 0x1a, 0, 0},
+ {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_720P_1280_720[] = {
+ {0x3035, 0x41, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x41, 0, 0}, {0x3821, 0x07, 0, 0}, {0x3814, 0x31, 0, 0},
+ {0x3815, 0x31, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0xfa, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x06, 0, 0}, {0x3807, 0xa9, 0, 0},
+ {0x3808, 0x05, 0, 0}, {0x3809, 0x00, 0, 0}, {0x380a, 0x02, 0, 0},
+ {0x380b, 0xd0, 0, 0}, {0x380c, 0x07, 0, 0}, {0x380d, 0x64, 0, 0},
+ {0x380e, 0x02, 0, 0}, {0x380f, 0xe4, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+ {0x3618, 0x00, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3709, 0x52, 0, 0}, {0x370c, 0x03, 0, 0}, {0x3a02, 0x02, 0, 0},
+ {0x3a03, 0xe4, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0xbc, 0, 0},
+ {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x72, 0, 0}, {0x3a0e, 0x01, 0, 0},
+ {0x3a0d, 0x02, 0, 0}, {0x3a14, 0x02, 0, 0}, {0x3a15, 0xe4, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x4713, 0x02, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0},
+ {0x3824, 0x04, 0, 0}, {0x5001, 0x83, 0, 0},
+};
+
+static struct reg_value ov5640_setting_30fps_1080P_1920_1080[] = {
+ {0x3008, 0x42, 0, 0},
+ {0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+ {0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+ {0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+ {0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+ {0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+ {0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+ {0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x11, 0, 0},
+ {0x3036, 0x54, 0, 0}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+ {0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+ {0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+ {0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+ {0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+ {0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+ {0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+ {0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+ {0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+ {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+ {0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0},
+ {0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_1080P_1920_1080[] = {
+ {0x3008, 0x42, 0, 0},
+ {0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+ {0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+ {0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+ {0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+ {0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+ {0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+ {0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 0}, {0x3035, 0x21, 0, 0},
+ {0x3036, 0x54, 0, 1}, {0x3c07, 0x07, 0, 0}, {0x3c08, 0x00, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3800, 0x01, 0, 0}, {0x3801, 0x50, 0, 0}, {0x3802, 0x01, 0, 0},
+ {0x3803, 0xb2, 0, 0}, {0x3804, 0x08, 0, 0}, {0x3805, 0xef, 0, 0},
+ {0x3806, 0x05, 0, 0}, {0x3807, 0xf1, 0, 0}, {0x3808, 0x07, 0, 0},
+ {0x3809, 0x80, 0, 0}, {0x380a, 0x04, 0, 0}, {0x380b, 0x38, 0, 0},
+ {0x380c, 0x09, 0, 0}, {0x380d, 0xc4, 0, 0}, {0x380e, 0x04, 0, 0},
+ {0x380f, 0x60, 0, 0}, {0x3612, 0x2b, 0, 0}, {0x3708, 0x64, 0, 0},
+ {0x3a02, 0x04, 0, 0}, {0x3a03, 0x60, 0, 0}, {0x3a08, 0x01, 0, 0},
+ {0x3a09, 0x50, 0, 0}, {0x3a0a, 0x01, 0, 0}, {0x3a0b, 0x18, 0, 0},
+ {0x3a0e, 0x03, 0, 0}, {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x04, 0, 0},
+ {0x3a15, 0x60, 0, 0}, {0x4713, 0x02, 0, 0}, {0x4407, 0x04, 0, 0},
+ {0x460b, 0x37, 0, 0}, {0x460c, 0x20, 0, 0}, {0x3824, 0x04, 0, 0},
+ {0x4005, 0x1a, 0, 0}, {0x3008, 0x02, 0, 0}, {0x3503, 0, 0, 0},
+};
+
+static struct reg_value ov5640_setting_15fps_QSXGA_2592_1944[] = {
+ {0x4202, 0x0f, 0, 0}, /* stream off the sensor */
+ {0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, /*disable flip*/
+ {0x3035, 0x21, 0, 0}, {0x3036, 0x54, 0, 0}, {0x3c07, 0x08, 0, 0},
+ {0x3c09, 0x1c, 0, 0}, {0x3c0a, 0x9c, 0, 0}, {0x3c0b, 0x40, 0, 0},
+ {0x3820, 0x40, 0, 0}, {0x3821, 0x06, 0, 0}, {0x3814, 0x11, 0, 0},
+ {0x3815, 0x11, 0, 0}, {0x3800, 0x00, 0, 0}, {0x3801, 0x00, 0, 0},
+ {0x3802, 0x00, 0, 0}, {0x3803, 0x00, 0, 0}, {0x3804, 0x0a, 0, 0},
+ {0x3805, 0x3f, 0, 0}, {0x3806, 0x07, 0, 0}, {0x3807, 0x9f, 0, 0},
+ {0x3808, 0x0a, 0, 0}, {0x3809, 0x20, 0, 0}, {0x380a, 0x07, 0, 0},
+ {0x380b, 0x98, 0, 0}, {0x380c, 0x0b, 0, 0}, {0x380d, 0x1c, 0, 0},
+ {0x380e, 0x07, 0, 0}, {0x380f, 0xb0, 0, 0}, {0x3810, 0x00, 0, 0},
+ {0x3811, 0x10, 0, 0}, {0x3812, 0x00, 0, 0}, {0x3813, 0x04, 0, 0},
+ {0x3618, 0x04, 0, 0}, {0x3612, 0x29, 0, 0}, {0x3708, 0x21, 0, 0},
+ {0x3709, 0x12, 0, 0}, {0x370c, 0x00, 0, 0}, {0x3a02, 0x03, 0, 0},
+ {0x3a03, 0xd8, 0, 0}, {0x3a08, 0x01, 0, 0}, {0x3a09, 0x27, 0, 0},
+ {0x3a0a, 0x00, 0, 0}, {0x3a0b, 0xf6, 0, 0}, {0x3a0e, 0x03, 0, 0},
+ {0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
+ {0x4001, 0x02, 0, 0}, {0x4004, 0x06, 0, 0}, {0x4713, 0x03, 0, 0},
+ {0x4407, 0x04, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
+ {0x3824, 0x02, 0, 0}, {0x5001, 0x83, 0, 70},
+ {0x4202, 0x00, 0, 0}, /* stream on the sensor */
+};
+
+static struct ov5640_mode_info
+ov5640_mode_info_data[ov5640_num_framerates][ov5640_num_modes] = {
+ {
+ {ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+ ov5640_setting_15fps_QCIF_176_144,
+ ARRAY_SIZE(ov5640_setting_15fps_QCIF_176_144)},
+ {ov5640_mode_QVGA_320_240, SUBSAMPLING, 320, 240,
+ ov5640_setting_15fps_QVGA_320_240,
+ ARRAY_SIZE(ov5640_setting_15fps_QVGA_320_240)},
+ {ov5640_mode_VGA_640_480, SUBSAMPLING, 640, 480,
+ ov5640_setting_15fps_VGA_640_480,
+ ARRAY_SIZE(ov5640_setting_15fps_VGA_640_480)},
+ {ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+ ov5640_setting_15fps_NTSC_720_480,
+ ARRAY_SIZE(ov5640_setting_15fps_NTSC_720_480)},
+ {ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+ ov5640_setting_15fps_PAL_720_576,
+ ARRAY_SIZE(ov5640_setting_15fps_PAL_720_576)},
+ {ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+ ov5640_setting_15fps_XGA_1024_768,
+ ARRAY_SIZE(ov5640_setting_15fps_XGA_1024_768)},
+ {ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+ ov5640_setting_15fps_720P_1280_720,
+ ARRAY_SIZE(ov5640_setting_15fps_720P_1280_720)},
+ {ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+ ov5640_setting_15fps_1080P_1920_1080,
+ ARRAY_SIZE(ov5640_setting_15fps_1080P_1920_1080)},
+ {ov5640_mode_QSXGA_2592_1944, SCALING, 2592, 1944,
+ ov5640_setting_15fps_QSXGA_2592_1944,
+ ARRAY_SIZE(ov5640_setting_15fps_QSXGA_2592_1944)},
+ }, {
+ {ov5640_mode_QCIF_176_144, SUBSAMPLING, 176, 144,
+ ov5640_setting_30fps_QCIF_176_144,
+ ARRAY_SIZE(ov5640_setting_30fps_QCIF_176_144)},
+ {ov5640_mode_QVGA_320_240, SUBSAMPLING, 320, 240,
+ ov5640_setting_30fps_QVGA_320_240,
+ ARRAY_SIZE(ov5640_setting_30fps_QVGA_320_240)},
+ {ov5640_mode_VGA_640_480, SUBSAMPLING, 640, 480,
+ ov5640_setting_30fps_VGA_640_480,
+ ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480)},
+ {ov5640_mode_NTSC_720_480, SUBSAMPLING, 720, 480,
+ ov5640_setting_30fps_NTSC_720_480,
+ ARRAY_SIZE(ov5640_setting_30fps_NTSC_720_480)},
+ {ov5640_mode_PAL_720_576, SUBSAMPLING, 720, 576,
+ ov5640_setting_30fps_PAL_720_576,
+ ARRAY_SIZE(ov5640_setting_30fps_PAL_720_576)},
+ {ov5640_mode_XGA_1024_768, SUBSAMPLING, 1024, 768,
+ ov5640_setting_30fps_XGA_1024_768,
+ ARRAY_SIZE(ov5640_setting_30fps_XGA_1024_768)},
+ {ov5640_mode_720P_1280_720, SUBSAMPLING, 1280, 720,
+ ov5640_setting_30fps_720P_1280_720,
+ ARRAY_SIZE(ov5640_setting_30fps_720P_1280_720)},
+ {ov5640_mode_1080P_1920_1080, SCALING, 1920, 1080,
+ ov5640_setting_30fps_1080P_1920_1080,
+ ARRAY_SIZE(ov5640_setting_30fps_1080P_1920_1080)},
+ {ov5640_mode_QSXGA_2592_1944, -1, 0, 0, NULL, 0},
+ },
+};
+
+static int ov5640_probe(struct i2c_client *adapter,
+ const struct i2c_device_id *device_id);
+static int ov5640_remove(struct i2c_client *client);
+
+static int ov5640_init_slave_id(struct ov5640_dev *sensor)
+{
+ struct i2c_msg msg;
+ u8 buf[4];
+ int ret;
+
+ if (sensor->i2c_client->addr == OV5640_DEFAULT_SLAVE_ID)
+ return 0;
+
+ buf[0] = OV5640_SLAVE_ID >> 8;
+ buf[1] = OV5640_SLAVE_ID & 0xff;
+ buf[2] = sensor->i2c_client->addr << 1;
+ msg.addr = OV5640_DEFAULT_SLAVE_ID;
+ msg.flags = 0;
+ msg.len = 3;
+ msg.buf = buf;
+
+ ret = i2c_transfer(sensor->i2c_client->adapter, &msg, 1);
+ if (ret < 0) {
+ dev_err(sensor->dev, "%s: failed with %d\n", __func__, ret);

+ return ret;
+ }
+

+ return 0;
+}
+

+static int ov5640_write_reg(struct ov5640_dev *sensor, u16 reg, u8 val)
+{
+ u8 buf[3] = {0};
+ int ret;
+
+ buf[0] = reg >> 8;
+ buf[1] = reg & 0xff;
+ buf[2] = val;
+
+ ret = i2c_master_send(sensor->i2c_client, buf, 3);
+ if (ret < 0) {
+ v4l2_err(&sensor->sd, "%s: error: reg=%x, val=%x\n",
+ __func__, reg, val);

+ return ret;
+ }
+

+ return 0;
+}
+

+static int ov5640_read_reg(struct ov5640_dev *sensor, u16 reg, u8 *val)
+{
+ u8 reg_buf[2] = {0};
+ u8 read_val = 0;
+
+ reg_buf[0] = reg >> 8;
+ reg_buf[1] = reg & 0xff;
+
+ if (2 != i2c_master_send(sensor->i2c_client, reg_buf, 2)) {
+ v4l2_err(&sensor->sd, "%s: write reg error: reg=%x\n",
+ __func__, reg);
+ return -EIO;
+ }
+
+ if (1 != i2c_master_recv(sensor->i2c_client, &read_val, 1)) {
+ v4l2_err(&sensor->sd, "%s: read reg error: reg=%x, val=%x\n",
+ __func__, reg, read_val);
+ return -EIO;
+ }
+
+ *val = read_val;

+ return 0;
+}
+

+#define OV5640_READ_REG(s, r, v) { \
+ ret = ov5640_read_reg((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }

+#define OV5640_WRITE_REG(s, r, v) { \
+ ret = ov5640_write_reg((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+static int ov5640_read_reg16(struct ov5640_dev *sensor, u16 reg, u16 *val)
+{
+ u8 hi, lo;
+ int ret;
+
+ OV5640_READ_REG(sensor, reg, &hi);
+ OV5640_READ_REG(sensor, reg+1, &lo);
+
+ *val = ((u16)hi << 8) | (u16)lo;
+ return 0;
+}
+#define OV5640_READ_REG16(s, r, v) { \
+ ret = ov5640_read_reg16((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)
+{
+ int ret;
+
+ OV5640_WRITE_REG(sensor, reg, val >> 8);
+ OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
+ return 0;
+}
+#define OV5640_WRITE_REG16(s, r, v) { \
+ ret = ov5640_write_reg16((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 reg,
+ u8 mask, u8 val)
+{
+ u8 readval;
+ int ret;
+
+ OV5640_READ_REG(sensor, reg, &readval);
+
+ readval &= ~mask;
+ val &= mask;
+ val |= readval;
+
+ OV5640_WRITE_REG(sensor, reg, val);
+ return 0;
+}
+#define OV5640_MOD_REG(s, r, m, v) { \
+ ret = ov5640_mod_reg((s), (r), (m), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+ int i, ret;
+

+ for (i = 0; i < size; ++i, ++regs) {
+ delay_ms = regs->delay_ms;
+ reg_addr = regs->reg_addr;
+ val = regs->val;
+ mask = regs->mask;
+
+ if (mask) {
+ OV5640_MOD_REG(sensor, reg_addr, mask, val);
+ } else {
+ OV5640_WRITE_REG(sensor, reg_addr, val);
+ }
+ if (delay_ms)
+ usleep_range(1000*delay_ms, 1000*delay_ms+100);

+ }
+
+ return 0;
+}
+

+static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)
+{
+ int ret;
+
+ OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);

+ return 0;
+}
+

+static int ov5640_get_sysclk(struct ov5640_dev *sensor)
+{
+ /* calculate sysclk */
+ int xvclk = sensor->xclk_freq / 10000;
+ int multiplier, prediv, VCO, sysdiv, pll_rdiv;
+ int sclk_rdiv_map[] = {1, 2, 4, 8};
+ int bit_div2x = 1, sclk_rdiv, sysclk;
+ u8 temp1, temp2;
+ int ret;
+
+ OV5640_READ_REG(sensor, 0x3034, &temp1);
+ temp2 = temp1 & 0x0f;
+ if (temp2 == 8 || temp2 == 10)
+ bit_div2x = temp2 / 2;
+
+ OV5640_READ_REG(sensor, 0x3035, &temp1);
+ sysdiv = temp1>>4;
+ if (sysdiv == 0)
+ sysdiv = 16;
+
+ OV5640_READ_REG(sensor, 0x3036, &temp1);
+ multiplier = temp1;
+
+ OV5640_READ_REG(sensor, 0x3037, &temp1);
+ prediv = temp1 & 0x0f;
+ pll_rdiv = ((temp1 >> 4) & 0x01) + 1;
+
+ OV5640_READ_REG(sensor, 0x3108, &temp1);
+ temp2 = temp1 & 0x03;
+ sclk_rdiv = sclk_rdiv_map[temp2];
+
+ VCO = xvclk * multiplier / prediv;
+
+ sysclk = VCO / sysdiv / pll_rdiv * 2 / bit_div2x / sclk_rdiv;
+
+ return sysclk;
+}
+
+static int ov5640_set_night_mode(struct ov5640_dev *sensor)
+{
+ /* read HTS from register settings */
+ u8 mode;
+ int ret;
+
+ OV5640_READ_REG(sensor, 0x3a00, &mode);
+ mode &= 0xfb;
+ OV5640_WRITE_REG(sensor, 0x3a00, mode);

+ return 0;
+}
+

+static int ov5640_get_HTS(struct ov5640_dev *sensor)
+{
+ /* read HTS from register settings */
+ u16 HTS;
+ int ret;
+
+ OV5640_READ_REG16(sensor, 0x380c, &HTS);
+ return HTS;
+}
+
+static int ov5640_get_VTS(struct ov5640_dev *sensor)
+{
+ u16 VTS;
+ int ret;
+
+ OV5640_READ_REG16(sensor, 0x380e, &VTS);
+ return VTS;
+}
+
+static int ov5640_set_VTS(struct ov5640_dev *sensor, int VTS)
+{
+ int ret;
+
+ OV5640_WRITE_REG16(sensor, 0x380e, VTS);

+ return 0;
+}
+

+static int ov5640_get_light_freq(struct ov5640_dev *sensor)
+{
+ /* get banding filter value */
+ u8 temp, temp1;
+ int light_freq = 0;
+ int ret;
+
+ OV5640_READ_REG(sensor, 0x3c01, &temp);
+
+ if (temp & 0x80) {
+ /* manual */
+ OV5640_READ_REG(sensor, 0x3c00, &temp1);
+ if (temp1 & 0x04) {
+ /* 50Hz */
+ light_freq = 50;
+ } else {
+ /* 60Hz */
+ light_freq = 60;
+ }
+ } else {
+ /* auto */
+ OV5640_READ_REG(sensor, 0x3c0c, &temp1);
+ if (temp1 & 0x01) {
+ /* 50Hz */
+ light_freq = 50;
+ } else {
+ /* 60Hz */
+ }
+ }
+
+ return light_freq;
+}
+
+static int ov5640_set_bandingfilter(struct ov5640_dev *sensor)
+{
+ int prev_vts;
+ int band_step60, max_band60, band_step50, max_band50;
+ int ret;
+
+ /* read preview PCLK */
+ ret = ov5640_get_sysclk(sensor);
+ if (ret < 0)
+ return ret;
+ sensor->prev_sysclk = ret;
+ /* read preview HTS */
+ ret = ov5640_get_HTS(sensor);
+ if (ret < 0)
+ return ret;
+ sensor->prev_hts = ret;
+
+ /* read preview VTS */
+ ret = ov5640_get_VTS(sensor);
+ if (ret < 0)
+ return ret;
+ prev_vts = ret;
+
+ /* calculate banding filter */
+ /* 60Hz */
+ band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
+ OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
+
+ max_band60 = (int)((prev_vts-4)/band_step60);
+ OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
+
+ /* 50Hz */
+ band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
+ OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
+
+ max_band50 = (int)((prev_vts-4)/band_step50);
+ OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);

+
+ return 0;
+}
+

+static int ov5640_set_AE_target(struct ov5640_dev *sensor, int target)
+{
+ /* stable in high */
+ int fast_high, fast_low;
+ int ret;
+
+ sensor->ae_low = target * 23 / 25; /* 0.92 */
+ sensor->ae_high = target * 27 / 25; /* 1.08 */
+
+ fast_high = sensor->ae_high<<1;
+ if (fast_high > 255)
+ fast_high = 255;
+
+ fast_low = sensor->ae_low >> 1;
+
+ OV5640_WRITE_REG(sensor, 0x3a0f, sensor->ae_high);
+ OV5640_WRITE_REG(sensor, 0x3a10, sensor->ae_low);
+ OV5640_WRITE_REG(sensor, 0x3a1b, sensor->ae_high);
+ OV5640_WRITE_REG(sensor, 0x3a1e, sensor->ae_low);
+ OV5640_WRITE_REG(sensor, 0x3a11, fast_high);
+ OV5640_WRITE_REG(sensor, 0x3a1f, fast_low);

+
+ return 0;
+}
+

+static int ov5640_binning_on(struct ov5640_dev *sensor)
+{
+ u8 temp;
+ int ret;
+
+ OV5640_READ_REG(sensor, 0x3821, &temp);
+ temp &= 0xfe;
+
+ return temp ? 1 : 0;
+}
+
+static int ov5640_set_virtual_channel(struct ov5640_dev *sensor)
+{
+ u8 temp, channel = sensor->ep.base.id;
+ int ret;
+
+ OV5640_READ_REG(sensor, 0x4814, &temp);
+ temp &= ~(3 << 6);
+ temp |= (channel << 6);
+ OV5640_WRITE_REG(sensor, 0x4814, temp);

+
+ return 0;
+}
+

+static enum ov5640_mode
+ov5640_find_nearest_mode(struct ov5640_dev *sensor,
+ int width, int height)
+{
+ int i;
+
+ for (i = ov5640_num_modes - 1; i >= 0; i--) {
+ if (ov5640_mode_info_data[0][i].width <= width &&
+ ov5640_mode_info_data[0][i].height <= height)
+ break;
+ }
+
+ if (i < 0)
+ i = 0;
+
+ return (enum ov5640_mode)i;
+}
+
+/*
+ * sensor changes between scaling and subsampling, go through
+ * exposure calculation
+ */
+static int ov5640_change_mode_exposure_calc(struct ov5640_dev *sensor,
+ enum ov5640_frame_rate frame_rate,
+ enum ov5640_mode mode)
+{
+ struct reg_value *mode_data = NULL;
+ int mode_size = 0;
+ u8 average;
+ int prev_shutter, prev_gain16;
+ int cap_shutter, cap_gain16;
+ int cap_sysclk, cap_hts, cap_vts;
+ int light_freq, cap_bandfilt, cap_maxband;
+ long cap_gain16_shutter;

+ int ret = 0;
+

+ /* check if the input mode and frame rate is valid */
+ mode_data = ov5640_mode_info_data[frame_rate][mode].init_data_ptr;
+ mode_size = ov5640_mode_info_data[frame_rate][mode].init_data_size;
+
+ sensor->fmt.width = ov5640_mode_info_data[frame_rate][mode].width;
+ sensor->fmt.height = ov5640_mode_info_data[frame_rate][mode].height;
+
+ if (sensor->fmt.width == 0 || sensor->fmt.height == 0 ||
+ mode_data == NULL || mode_size == 0)
+ return -EINVAL;
+
+ /* auto focus */
+ /* ov5640_auto_focus();//if no af function, just skip it */
+
+ /* turn off AE/AG */
+ ret = ov5640_set_agc(sensor, false);
+ if (ret < 0)
+ return ret;
+
+ /* read preview shutter */
+ ret = ov5640_get_exposure(sensor);
+ if (ret < 0)
+ return ret;
+ prev_shutter = ret;
+ ret = ov5640_binning_on(sensor);
+ if (ret < 0)
+ return ret;
+ if (ret && mode != ov5640_mode_720P_1280_720 &&
+ mode != ov5640_mode_1080P_1920_1080)
+ prev_shutter *= 2;
+
+ /* read preview gain */
+ ret = ov5640_get_gain(sensor);
+ if (ret < 0)
+ return ret;
+ prev_gain16 = ret;
+
+ /* get average */
+ OV5640_READ_REG(sensor, 0x56a1, &average);
+
+ /* turn off night mode for capture */
+ ret = ov5640_set_night_mode(sensor);
+ if (ret < 0)
+ return ret;
+
+ /* turn off overlay */
+ /* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
+ just skip it */
+
+ ret = ov5640_set_stream(sensor, false);
+ if (ret < 0)
+ return ret;
+
+ /* Write capture setting */
+ ret = ov5640_load_regs(sensor, mode_data, mode_size);
+ if (ret < 0)
+ return ret;
+
+ /* read capture VTS */
+ ret = ov5640_get_VTS(sensor);
+ if (ret < 0)
+ return ret;
+ cap_vts = ret;
+ ret = ov5640_get_HTS(sensor);
+ if (ret < 0)
+ return ret;
+ cap_hts = ret;
+ ret = ov5640_get_sysclk(sensor);
+ if (ret < 0)
+ return ret;
+ cap_sysclk = ret;
+
+ /* calculate capture banding filter */
+ ret = ov5640_get_light_freq(sensor);
+ if (ret < 0)
+ return ret;
+ light_freq = ret;
+
+ if (light_freq == 60) {
+ /* 60Hz */
+ cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
+ } else {
+ /* 50Hz */
+ cap_bandfilt = cap_sysclk * 100 / cap_hts;
+ }
+ cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
+
+ /* calculate capture shutter/gain16 */
+ if (average > sensor->ae_low && average < sensor->ae_high) {
+ /* in stable range */
+ cap_gain16_shutter =
+ prev_gain16 * prev_shutter *
+ cap_sysclk / sensor->prev_sysclk *
+ sensor->prev_hts / cap_hts *
+ sensor->ae_target / average;
+ } else {
+ cap_gain16_shutter =
+ prev_gain16 * prev_shutter *
+ cap_sysclk / sensor->prev_sysclk *
+ sensor->prev_hts / cap_hts;
+ }
+
+ /* gain to shutter */
+ if (cap_gain16_shutter < (cap_bandfilt * 16)) {
+ /* shutter < 1/100 */
+ cap_shutter = cap_gain16_shutter / 16;
+ if (cap_shutter < 1)
+ cap_shutter = 1;
+
+ cap_gain16 = cap_gain16_shutter / cap_shutter;
+ if (cap_gain16 < 16)
+ cap_gain16 = 16;
+ } else {
+ if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
+ /* exposure reach max */
+ cap_shutter = cap_bandfilt * cap_maxband;
+ cap_gain16 = cap_gain16_shutter / cap_shutter;
+ } else {
+ /* 1/100 < (cap_shutter = n/100) =< max */
+ cap_shutter =
+ ((int)(cap_gain16_shutter / 16 / cap_bandfilt))
+ * cap_bandfilt;
+ cap_gain16 = cap_gain16_shutter / cap_shutter;
+ }
+ }
+
+ /* write capture gain */
+ ret = ov5640_set_gain(sensor, cap_gain16);
+ if (ret < 0)
+ return ret;
+
+ /* write capture shutter */
+ if (cap_shutter > (cap_vts - 4)) {
+ cap_vts = cap_shutter + 4;
+ ret = ov5640_set_VTS(sensor, cap_vts);
+ if (ret < 0)

+ return ret;
+ }
+

+ ret = ov5640_set_exposure(sensor, cap_shutter);
+ if (ret < 0)
+ return ret;
+
+ return ov5640_set_stream(sensor, true);
+}
+
+/*
+ * if sensor changes inside scaling or subsampling
+ * change mode directly
+ */
+static int ov5640_change_mode_direct(struct ov5640_dev *sensor,
+ enum ov5640_frame_rate frame_rate,
+ enum ov5640_mode mode)
+{
+ struct reg_value *mode_data = NULL;
+ int mode_size = 0;

+ int ret = 0;
+

+ if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
+ mode != ov5640_mode_INIT) {
+ v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");

+ return -EINVAL;
+ }
+

+ dn_mode = ov5640_mode_info_data[frame_rate][mode].dn_mode;
+ orig_dn_mode = ov5640_mode_info_data[frame_rate][orig_mode].dn_mode;
+ if (mode == ov5640_mode_INIT) {
+ mode_data = ov5640_init_setting_30fps_VGA;
+ mode_size = ARRAY_SIZE(ov5640_init_setting_30fps_VGA);
+
+ sensor->fmt.width = 640;
+ sensor->fmt.height = 480;
+ ret = ov5640_load_regs(sensor, mode_data, mode_size);
+ if (ret < 0)
+ return ret;
+
+ mode_data = ov5640_setting_30fps_VGA_640_480;
+ mode_size = ARRAY_SIZE(ov5640_setting_30fps_VGA_640_480);
+ ret = ov5640_load_regs(sensor, mode_data, mode_size);
+ } else if ((dn_mode == SUBSAMPLING && orig_dn_mode == SCALING) ||
+ (dn_mode == SCALING && orig_dn_mode == SUBSAMPLING)) {
+ /* change between subsampling and scaling
+ * go through exposure calucation */
+ ret = ov5640_change_mode_exposure_calc(sensor, frame_rate,
+ mode);
+ } else {
+ /* change inside subsampling or scaling
+ * download firmware directly */
+ ret = ov5640_change_mode_direct(sensor, frame_rate, mode);
+ }
+
+ if (ret < 0)
+ return ret;
+
+ ret = ov5640_set_AE_target(sensor, sensor->ae_target);
+ if (ret < 0)
+ return ret;
+ ret = ov5640_get_light_freq(sensor);
+ if (ret < 0)
+ return ret;
+ ret = ov5640_set_bandingfilter(sensor);
+ if (ret < 0)
+ return ret;
+ ret = ov5640_set_virtual_channel(sensor);
+ if (ret < 0)
+ return ret;
+
+ /* restore controls */
+ ov5640_restore_ctrls(sensor);
+
+ if (ret >= 0 && mode != ov5640_mode_INIT) {
+ sensor->current_mode = mode;
+ sensor->current_fr = frame_rate;

+ }
+
+ return 0;
+}
+

+/* restore the last set video mode after chip power-on */
+static int ov5640_restore_mode(struct ov5640_dev *sensor)
+{

+ int ret = 0;
+

+ /* first we need to set some initial register values */
+ ret = ov5640_change_mode(sensor, sensor->current_fr,
+ ov5640_mode_INIT, ov5640_mode_INIT);
+ if (ret < 0)
+ return ret;
+
+ /* now restore the last capture mode */
+ return ov5640_change_mode(sensor,
+ sensor->current_fr,
+ sensor->current_mode,
+ ov5640_mode_VGA_640_480);
+}
+
+static int ov5640_regulators_on(struct ov5640_dev *sensor)
+{
+ int ret;
+
+ if (sensor->io_regulator) {
+ ret = regulator_enable(sensor->io_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "io reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->core_regulator) {
+ ret = regulator_enable(sensor->core_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "core reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->gpo_regulator) {
+ ret = regulator_enable(sensor->gpo_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "gpo reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->analog_regulator) {
+ ret = regulator_enable(sensor->analog_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "analog reg enable failed\n");

+ return ret;
+ }
+ }
+

+ return 0;
+}
+

+static void ov5640_regulators_off(struct ov5640_dev *sensor)
+{
+ if (sensor->analog_regulator)
+ regulator_disable(sensor->analog_regulator);
+ if (sensor->core_regulator)
+ regulator_disable(sensor->core_regulator);
+ if (sensor->io_regulator)
+ regulator_disable(sensor->io_regulator);
+ if (sensor->gpo_regulator)
+ regulator_disable(sensor->gpo_regulator);
+}
+
+/* --------------- Subdev Operations --------------- */
+
+static int ov5640_s_power(struct v4l2_subdev *sd, int on)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ int ret;
+
+ v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+ if (on && !sensor->on) {
+ if (sensor->xclk)
+ clk_prepare_enable(sensor->xclk);
+
+ ret = ov5640_regulators_on(sensor);

+ if (ret)
+ return ret;
+

+ ov5640_reset(sensor);
+ ov5640_power(sensor, true);
+
+ ret = ov5640_init_slave_id(sensor);

+ if (ret)
+ return ret;
+

+ ret = ov5640_restore_mode(sensor);

+ if (ret)
+ return ret;
+

+ /*
+ * NOTE: Freescale adds a long delay (600 msec) after
+ * powering up and programming a mode on the ov5640-mipi
+ * camera (search for "msec_wait4stable" in FSL's
+ * ov5640_mipi.c), which equivalently would need to go
+ * right here. If we run into MIPI CSI-2 receiver dphy
+ * ready timeouts, it might be a clue to add that delay
+ * here.
+ */
+ } else if (!on && sensor->on) {
+ ov5640_power(sensor, false);
+
+ ov5640_regulators_off(sensor);
+
+ if (sensor->xclk)
+ clk_disable_unprepare(sensor->xclk);
+ }
+
+ sensor->on = on;

+
+ return 0;
+}
+

+static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ struct v4l2_captureparm *cparm = &a->parm.capture;
+
+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ /* This is the only case currently handled. */
+ memset(a, 0, sizeof(*a));
+ a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ cparm->capability = sensor->streamcap.capability;
+ cparm->timeperframe = sensor->streamcap.timeperframe;
+ cparm->capturemode = sensor->streamcap.capturemode;

+
+ return 0;
+}
+

+static int ov5640_s_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ struct v4l2_fract *timeperframe = &a->parm.capture.timeperframe;
+ enum ov5640_frame_rate frame_rate;
+ u32 tgt_fps; /* target frames per secound */

+ int ret = 0;
+

+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ /* Check that the new frame rate is allowed. */
+ if ((timeperframe->numerator == 0) ||
+ (timeperframe->denominator == 0)) {
+ timeperframe->denominator = DEFAULT_FPS;
+ timeperframe->numerator = 1;
+ }
+
+ tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+ if (tgt_fps > MAX_FPS) {
+ timeperframe->denominator = MAX_FPS;
+ timeperframe->numerator = 1;
+ } else if (tgt_fps < MIN_FPS) {
+ timeperframe->denominator = MIN_FPS;
+ timeperframe->numerator = 1;
+ }
+
+ /* Actual frame rate we use */
+ tgt_fps = timeperframe->denominator / timeperframe->numerator;
+
+ if (tgt_fps == 15)
+ frame_rate = ov5640_15_fps;
+ else if (tgt_fps == 30)
+ frame_rate = ov5640_30_fps;
+ else {
+ v4l2_err(&sensor->sd, "frame rate %u not supported!\n",
+ tgt_fps);

+ return -EINVAL;
+ }
+

+ ret = ov5640_change_mode(sensor, frame_rate,
+ sensor->current_mode,
+ sensor->current_mode);
+ if (ret < 0)
+ return ret;
+
+ sensor->streamcap.timeperframe = *timeperframe;

+
+ return 0;
+}
+

+static int ov5640_get_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *format)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ if (format->pad != 0)
+ return -EINVAL;
+
+ format->format = sensor->fmt;

+
+ return 0;
+}
+

+static int ov5640_try_fmt_internal(struct v4l2_subdev *sd,
+ struct v4l2_mbus_framefmt *fmt,
+ enum ov5640_mode *new_mode)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ enum ov5640_mode mode;
+
+ mode = ov5640_find_nearest_mode(sensor, fmt->width, fmt->height);
+
+ fmt->width = ov5640_mode_info_data[0][mode].width;
+ fmt->height = ov5640_mode_info_data[0][mode].height;
+ fmt->code = sensor->fmt.code;
+
+ if (new_mode)
+ *new_mode = mode;

+ return 0;
+}
+

+static int ov5640_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *format)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ enum ov5640_mode new_mode;
+ int ret;
+
+ if (format->pad != 0)
+ return -EINVAL;
+
+ if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
+ ret = ov5640_try_fmt_internal(sd, &format->format, NULL);

+ if (ret)
+ return ret;

+ cfg->try_fmt = format->format;

+ return 0;
+ }
+

+ ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);

+ if (ret)
+ return ret;
+

+ ret = ov5640_change_mode(sensor, sensor->current_fr,
+ new_mode, sensor->current_mode);
+ if (ret >= 0)
+ sensor->fmt = format->format;
+

+ return ret;
+}
+
+

+/*
+ * Sensor Controls.
+ */
+
+static int ov5640_set_hue(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+ OV5640_WRITE_REG16(sensor, 0x5581, value);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);

+
+ return 0;
+}
+

+static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+ OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);

+
+ return 0;
+}
+

+static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
+ OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);

+
+ return 0;
+}
+

+static int ov5640_set_awb(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ sensor->awb_on = value ? true : false;
+ OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);

+ return 0;
+}
+

+static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);

+ return 0;
+}
+

+#if 0
+static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
+ return 0;
+}
+#endif
+
+static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);

+ return 0;
+}
+

+static int ov5640_set_exposure(struct ov5640_dev *sensor, int value)
+{
+ u16 max_exp = 0;
+ int ret;
+
+ if (sensor->agc_on)
+ return -EINVAL;
+
+ OV5640_READ_REG16(sensor, 0x350c, &max_exp);
+ if (value < max_exp) {
+ u32 exp = value << 4;
+
+ OV5640_WRITE_REG(sensor, 0x3502, exp & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3501, (exp >> 8) & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3500, (exp >> 16) & 0x0f);

+ }
+
+ return 0;
+}
+

+/* read exposure, in number of line periods */
+static int ov5640_get_exposure(struct ov5640_dev *sensor)
+{
+ u8 temp;
+ int exp, ret;
+
+ if (sensor->agc_on)
+ return -EINVAL;
+
+ OV5640_READ_REG(sensor, 0x3500, &temp);
+ exp = ((int)temp & 0x0f) << 16;
+ OV5640_READ_REG(sensor, 0x3501, &temp);
+ exp |= ((int)temp << 8);
+ OV5640_READ_REG(sensor, 0x3502, &temp);
+ exp |= (int)temp;
+
+ return exp >> 4;
+}
+
+static int ov5640_set_agc(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ /* this enables/disables both AEC and AGC */
+ sensor->agc_on = value ? true : false;
+ OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);

+
+ return 0;
+}
+

+static int ov5640_set_gain(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ if (sensor->agc_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);

+ return 0;
+}
+

+static int ov5640_get_gain(struct ov5640_dev *sensor)
+{
+ u16 gain;
+ int ret;
+
+ if (sensor->agc_on)
+ return -EINVAL;
+
+ OV5640_READ_REG16(sensor, 0x350a, &gain);
+
+ return gain & 0x3ff;
+}
+
+#if 0
+static int ov5640_set_test_pattern(struct ov5640_dev *sensor, int value)
+{
+ int ret;
+
+ OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
+ return 0;
+}
+#endif
+
+static struct ov5640_control ov5640_ctrls[] = {
+ {
+ .set = ov5640_set_agc,
+ .ctrl = {
+ .id = V4L2_CID_AUTOGAIN,
+ .name = "Auto Gain/Exposure Control",
+ .minimum = 0,
+ .maximum = 1,
+ .step = 1,
+ .default_value = 1,
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ },
+ }, {
+ .set = ov5640_set_exposure,
+ .ctrl = {
+ .id = V4L2_CID_EXPOSURE,
+ .name = "Exposure",
+ .minimum = 0,
+ .maximum = 65535,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_gain,
+ .ctrl = {
+ .id = V4L2_CID_GAIN,
+ .name = "Gain",
+ .minimum = 0,
+ .maximum = 1023,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_hue,
+ .ctrl = {
+ .id = V4L2_CID_HUE,
+ .name = "Hue",
+ .minimum = 0,
+ .maximum = 359,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_contrast,
+ .ctrl = {
+ .id = V4L2_CID_CONTRAST,
+ .name = "Contrast",
+ .minimum = 0,
+ .maximum = 255,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_saturation,
+ .ctrl = {
+ .id = V4L2_CID_SATURATION,
+ .name = "Saturation",
+ .minimum = 0,
+ .maximum = 255,
+ .step = 1,
+ .default_value = 64,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_awb,
+ .ctrl = {
+ .id = V4L2_CID_AUTO_WHITE_BALANCE,
+ .name = "Auto White Balance",
+ .minimum = 0,
+ .maximum = 1,
+ .step = 1,
+ .default_value = 1,
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ },
+ }, {
+ .set = ov5640_set_red_balance,
+ .ctrl = {
+ .id = V4L2_CID_RED_BALANCE,
+ .name = "Red Balance",
+ .minimum = 0,
+ .maximum = 4095,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_blue_balance,
+ .ctrl = {
+ .id = V4L2_CID_BLUE_BALANCE,
+ .name = "Blue Balance",
+ .minimum = 0,
+ .maximum = 4095,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ },
+};
+#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
+
+static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
+{
+ struct ov5640_control *ret = NULL;
+ int i;
+
+ for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+ if (id == ov5640_ctrls[i].ctrl.id) {
+ ret = &ov5640_ctrls[i];
+ break;
+ }
+ }
+
+ if (ret && index)
+ *index = i;

+ return ret;
+}
+

+static int ov5640_restore_ctrls(struct ov5640_dev *sensor)
+{
+ struct ov5640_control *c;
+ int i;
+
+ for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+ c = &ov5640_ctrls[i];
+ c->set(sensor, sensor->ctrl_cache[i]);

+ }
+
+ return 0;
+}
+

+static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
+ struct ov5640_control *c;

+ int ret = 0;

+ int i;
+
+ c = ov5640_get_ctrl(ctrl->id, &i);
+ if (!c)
+ return -EINVAL;
+
+ ret = c->set(sensor, ctrl->val);
+ /* update cached value if no error */
+ if (!ret)
+ sensor->ctrl_cache[i] = ctrl->val;
+

+ return ret;
+}
+

+static const struct v4l2_ctrl_ops ov5640_ctrl_ops = {
+ .s_ctrl = ov5640_s_ctrl,
+};
+
+static int ov5640_init_controls(struct ov5640_dev *sensor)
+{
+ struct ov5640_control *c;
+ int i;
+
+ v4l2_ctrl_handler_init(&sensor->ctrl_hdl, OV5640_NUM_CONTROLS);
+
+ for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+ c = &ov5640_ctrls[i];
+
+ v4l2_ctrl_new_std(&sensor->ctrl_hdl, &ov5640_ctrl_ops,
+ c->ctrl.id, c->ctrl.minimum, c->ctrl.maximum,
+ c->ctrl.step, c->ctrl.default_value);
+ }
+
+ sensor->sd.ctrl_handler = &sensor->ctrl_hdl;
+ if (sensor->ctrl_hdl.error) {
+ int err = sensor->ctrl_hdl.error;
+
+ v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+
+ v4l2_err(&sensor->sd, "%s: error %d\n", __func__, err);
+ return err;
+ }
+ v4l2_ctrl_handler_setup(&sensor->ctrl_hdl);

+
+ return 0;
+}
+

+static int ov5640_enum_frame_size(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_frame_size_enum *fse)
+{
+ if (fse->pad != 0)
+ return -EINVAL;
+ if (fse->index >= ov5640_num_modes)
+ return -EINVAL;
+
+ fse->min_width = fse->max_width =
+ ov5640_mode_info_data[0][fse->index].width;
+ fse->min_height = fse->max_height =
+ ov5640_mode_info_data[0][fse->index].height;

+
+ return 0;
+}
+

+static int ov5640_enum_frame_interval(
+ struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_frame_interval_enum *fie)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ enum ov5640_mode mode;
+
+ if (fie->pad != 0)
+ return -EINVAL;
+ if (fie->index < 0 || fie->index >= ov5640_num_framerates)
+ return -EINVAL;
+
+ if (fie->width == 0 || fie->height == 0)
+ return -EINVAL;
+
+ mode = ov5640_find_nearest_mode(sensor, fie->width, fie->height);
+
+ if (ov5640_mode_info_data[fie->index][mode].init_data_ptr == NULL)
+ return -EINVAL;
+
+ fie->interval.numerator = 1;
+ fie->interval.denominator = ov5640_framerates[fie->index];
+
+ dev_dbg(sensor->dev, "%dx%d: [%d] = %d fps\n",
+ fie->width, fie->height, fie->index, fie->interval.denominator);

+ return 0;
+}
+

+static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ *status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;

+
+ return 0;
+}
+

+static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
+ u32 output, u32 config)
+{
+ return (input != 0) ? -EINVAL : 0;
+}
+
+static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ if (code->pad != 0)
+ return -EINVAL;
+ if (code->index != 0)
+ return -EINVAL;
+
+ code->code = sensor->fmt.code;

+
+ return 0;
+}
+

+static int ov5640_g_mbus_config(struct v4l2_subdev *sd,
+ struct v4l2_mbus_config *cfg)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ cfg->type = V4L2_MBUS_CSI2;
+ cfg->flags = sensor->ep.bus.mipi_csi2.flags;
+ cfg->flags |= (1 << (sensor->ep.bus.mipi_csi2.num_data_lanes - 1));
+ cfg->flags |= V4L2_MBUS_CSI2_CHANNEL_0;

+
+ return 0;
+}
+

+static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);

+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ return ov5640_set_stream(sensor, enable);
+}
+
+static struct v4l2_subdev_core_ops ov5640_core_ops = {
+ .s_power = ov5640_s_power,
+};
+
+static struct v4l2_subdev_video_ops ov5640_video_ops = {
+ .s_parm = ov5640_s_parm,
+ .g_parm = ov5640_g_parm,
+ .g_input_status = ov5640_g_input_status,
+ .s_routing = ov5640_s_routing,
+ .g_mbus_config = ov5640_g_mbus_config,
+ .s_stream = ov5640_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops ov5640_pad_ops = {
+ .enum_mbus_code = ov5640_enum_mbus_code,
+ .get_fmt = ov5640_get_fmt,
+ .set_fmt = ov5640_set_fmt,
+ .enum_frame_size = ov5640_enum_frame_size,
+ .enum_frame_interval = ov5640_enum_frame_interval,
+};
+
+static struct v4l2_subdev_ops ov5640_subdev_ops = {
+ .core = &ov5640_core_ops,
+ .video = &ov5640_video_ops,
+ .pad = &ov5640_pad_ops,
+};
+
+static void ov5640_power(struct ov5640_dev *sensor, bool enable)
+{
+ gpiod_set_value(sensor->pwdn_gpio, enable ? 0 : 1);
+}
+
+static void ov5640_reset(struct ov5640_dev *sensor)
+{
+ gpiod_set_value(sensor->reset_gpio, 0);
+
+ /* camera power cycle */
+ ov5640_power(sensor, false);
+ usleep_range(5000, 10000);
+ ov5640_power(sensor, true);
+ usleep_range(5000, 10000);
+
+ gpiod_set_value(sensor->reset_gpio, 1);
+ usleep_range(1000, 2000);
+
+ gpiod_set_value(sensor->reset_gpio, 0);
+ usleep_range(5000, 10000);
+}
+
+static void ov5640_get_regulators(struct ov5640_dev *sensor)
+{
+ sensor->io_regulator = devm_regulator_get(sensor->dev, "DOVDD");
+ if (!IS_ERR(sensor->io_regulator)) {
+ regulator_set_voltage(sensor->io_regulator,
+ OV5640_VOLTAGE_DIGITAL_IO,
+ OV5640_VOLTAGE_DIGITAL_IO);
+ } else {
+ dev_dbg(sensor->dev, "%s: no io voltage reg found\n",
+ __func__);
+ sensor->io_regulator = NULL;
+ }
+
+ sensor->core_regulator = devm_regulator_get(sensor->dev, "DVDD");
+ if (!IS_ERR(sensor->core_regulator)) {
+ regulator_set_voltage(sensor->core_regulator,
+ OV5640_VOLTAGE_DIGITAL_CORE,
+ OV5640_VOLTAGE_DIGITAL_CORE);
+ } else {
+ sensor->core_regulator = NULL;
+ dev_dbg(sensor->dev, "%s: no core voltage reg found\n",
+ __func__);
+ }
+
+ sensor->analog_regulator = devm_regulator_get(sensor->dev, "AVDD");
+ if (!IS_ERR(sensor->analog_regulator)) {
+ regulator_set_voltage(sensor->analog_regulator,
+ OV5640_VOLTAGE_ANALOG,
+ OV5640_VOLTAGE_ANALOG);
+ } else {
+ sensor->analog_regulator = NULL;
+ dev_dbg(sensor->dev, "%s: no analog voltage reg found\n",
+ __func__);
+ }
+}
+
+static int ov5640_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct device *dev = &client->dev;
+ struct device_node *endpoint;
+ struct ov5640_dev *sensor;
+ int i, xclk, ret;
+
+ sensor = devm_kzalloc(dev, sizeof(struct ov5640_dev), GFP_KERNEL);
+ if (!sensor)
+ return -ENOMEM;
+
+ sensor->i2c_client = client;
+ sensor->dev = dev;
+ sensor->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
+ sensor->fmt.width = 640;
+ sensor->fmt.height = 480;
+ sensor->fmt.field = V4L2_FIELD_NONE;
+ sensor->streamcap.capability = V4L2_MODE_HIGHQUALITY |
+ V4L2_CAP_TIMEPERFRAME;
+ sensor->streamcap.capturemode = 0;
+ sensor->streamcap.timeperframe.denominator = DEFAULT_FPS;
+ sensor->streamcap.timeperframe.numerator = 1;
+
+ sensor->current_mode = ov5640_mode_VGA_640_480;
+ sensor->current_fr = ov5640_30_fps;
+
+ sensor->ae_target = 52;
+
+ endpoint = of_graph_get_next_endpoint(client->dev.of_node, NULL);
+ if (!endpoint) {
+ dev_err(dev, "endpoint node not found\n");

+ return -EINVAL;
+ }
+

+ v4l2_of_parse_endpoint(endpoint, &sensor->ep);
+ if (sensor->ep.bus_type != V4L2_MBUS_CSI2) {
+ dev_err(dev, "invalid bus type, must be MIPI CSI2\n");
+ return -EINVAL;
+ }
+ of_node_put(endpoint);
+
+ /* get system clock (xclk) frequency */
+ ret = of_property_read_u32(dev->of_node, "xclk", &xclk);
+ if (!ret) {
+ if (xclk < OV5640_XCLK_MIN || xclk > OV5640_XCLK_MAX) {
+ dev_err(dev, "invalid xclk frequency\n");
+ return -EINVAL;
+ }
+ sensor->xclk_freq = xclk;
+ }
+
+ /* get system clock (xclk) */
+ sensor->xclk = devm_clk_get(dev, "xclk");
+ if (!IS_ERR(sensor->xclk)) {
+ if (!sensor->xclk_freq) {
+ dev_err(dev, "xclk requires xclk frequency!\n");
+ return -EINVAL;
+ }
+ clk_set_rate(sensor->xclk, sensor->xclk_freq);
+ } else {
+ /* assume system clock enabled by default */
+ sensor->xclk = NULL;
+ }
+
+ /* request power down pin */
+ sensor->pwdn_gpio = devm_gpiod_get(dev, "pwdn", GPIOD_OUT_HIGH);
+ if (IS_ERR(sensor->pwdn_gpio)) {
+ dev_err(dev, "request for power down gpio failed\n");
+ return PTR_ERR(sensor->pwdn_gpio);
+ }
+
+ /* request reset pin */
+ sensor->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
+ if (IS_ERR(sensor->reset_gpio)) {
+ dev_err(dev, "request for reset gpio failed\n");
+ return PTR_ERR(sensor->reset_gpio);
+ }
+
+ /* initialize the cached controls to their defaults */
+ for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+ struct ov5640_control *c = &ov5640_ctrls[i];
+
+ sensor->ctrl_cache[i] = c->ctrl.default_value;
+ }
+ sensor->awb_on = sensor->agc_on = true;
+
+ v4l2_i2c_subdev_init(&sensor->sd, client, &ov5640_subdev_ops);
+
+ sensor->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+ sensor->pad.flags = MEDIA_PAD_FL_SOURCE;
+ sensor->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
+ ret = media_entity_pads_init(&sensor->sd.entity, 1, &sensor->pad);

+ if (ret)
+ return ret;
+

+ ov5640_get_regulators(sensor);
+
+ ret = ov5640_s_power(&sensor->sd, 1);
+ if (ret)
+ goto entity_cleanup;
+ ret = ov5640_init_controls(sensor);
+ if (ret)
+ goto power_off;
+
+ ret = ov5640_s_power(&sensor->sd, 0);
+ if (ret)
+ goto free_ctrls;
+
+ ret = v4l2_async_register_subdev(&sensor->sd);
+ if (ret)
+ goto free_ctrls;

+
+ return 0;
+

+free_ctrls:
+ v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+ ov5640_s_power(&sensor->sd, 0);
+entity_cleanup:
+ media_entity_cleanup(&sensor->sd.entity);
+ ov5640_regulators_off(sensor);

+ return ret;
+}
+

+/*!
+ * ov5640 I2C detach function
+ *
+ * @param client struct i2c_client *
+ * @return Error code indicating success or failure
+ */
+static int ov5640_remove(struct i2c_client *client)
+{
+ struct v4l2_subdev *sd = i2c_get_clientdata(client);
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ ov5640_regulators_off(sensor);
+
+ v4l2_async_unregister_subdev(&sensor->sd);
+ media_entity_cleanup(&sensor->sd.entity);
+ v4l2_device_unregister_subdev(sd);
+ v4l2_ctrl_handler_free(&sensor->ctrl_hdl);

+
+ return 0;
+}
+

+static const struct i2c_device_id ov5640_id[] = {
+ {"ov5640_mipi", 0},
+ {},
+};
+MODULE_DEVICE_TABLE(i2c, ov5640_id);
+
+static const struct of_device_id ov5640_dt_ids[] = {
+ { .compatible = "ovti,ov5640_mipi" },

+ { /* sentinel */ }
+};

+MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
+
+static struct i2c_driver ov5640_i2c_driver = {
+ .driver = {
+ .owner = THIS_MODULE,
+ .name = "ov5640_mipi",
+ .of_match_table = ov5640_dt_ids,
+ },
+ .id_table = ov5640_id,
+ .probe = ov5640_probe,
+ .remove = ov5640_remove,
+};
+
+module_i2c_driver(ov5640_i2c_driver);
+
+MODULE_AUTHOR("Freescale Semiconductor, Inc.");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");

+MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:30:07 PM12/29/16

Enables the ADV7180 decoder sensor. The ADV7180 connects to the
parallel-bus mux input on ipu1_csi0_mux.

On the sabreauto, two analog video inputs are routed to the ADV7180,
composite on Ain1, and composite on Ain3. Those inputs are defined
via inputs and input-names under the ADV7180 node. The ADV7180 power
pin is via max7310_b port expander.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 56 ++++++++++++++++++++++++++++++++
1 file changed, 56 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 83ac2ff..30ee378 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -147,10 +147,42 @@
gpio-controller;
#gpio-cells = <2>;
};
+
+ camera: adv7180@21 {
+ compatible = "adi,adv7180";
+ reg = <0x21>;
+ powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+ interrupt-parent = <&gpio1>;
+ interrupts = <27 0x8>;
+ inputs = <0x00 0x02>;
+ input-names = "ADV7180 Composite on Ain1",
+ "ADV7180 Composite on Ain3";
+
+ port {
+ adv7180_to_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;
+ };
+ };
+ };
};
};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+ bus-width = <8>;
+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
&clks {
assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
<&clks IMX6QDL_PLL4_BYPASS>,
@@ -451,6 +483,30 @@
>;
};

+ pinctrl_ipu1_csi0: ipu1grp-csi0 {
+ fsl,pins = <
+ MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04 0x80000000
+ MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05 0x80000000
+ MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06 0x80000000
+ MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07 0x80000000
+ MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08 0x80000000
+ MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09 0x80000000
+ MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10 0x80000000
+ MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11 0x80000000
+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x80000000
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x80000000
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x80000000
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x80000000
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x80000000
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x80000000
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x80000000
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x80000000
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x80000000
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x80000000
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x80000000
+ >;
+ };
+
pinctrl_pwm3: pwm1grp {
fsl,pins = <
MX6QDL_PAD_SD4_DAT1__PWM3_OUT 0x1b0b1
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:05 PM12/29/16

From: Philipp Zabel <p.z...@pengutronix.de>

This patch adds the device tree graph connecting the input multiplexers
to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU
multiplexers are added as children of the iomuxc-gpr syscon device node.
On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2
CSI1 allow to select between CSI0/1 parallel input pads and the MIPI
CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1
CSI1 allow to select between CSI0/1 parallel input pads and any of the
four MIPI CSI-2 virtual channels.

Signed-off-by: Philipp Zabel <p.z...@pengutronix.de>

- Removed some dangling/unused endpoints (ipu2_csi0_from_csi2ipu)
- Renamed the mipi virtual channel endpoint labels, from "mipi_csiX_..."
to "mipi_vcX...".
- Added input endpoints to the video muxes for the connections from parallel
sensors.
- Added input endpoints to the mipi_csi for the connections from mipi csi-2
sensors.
- The video multiplexer node has compatible string "imx-video-mux" instead
of "video-multiplexer".
- The video multiplexer node indicates GPR register via reg propert only,
(register offset and bitmask), instead of specifying with "bit-mask" and
"bit-shift" properties.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6dl.dtsi | 183 +++++++++++++++++++++++++++++++++++++++++
arch/arm/boot/dts/imx6q.dtsi | 119 +++++++++++++++++++++++++++
arch/arm/boot/dts/imx6qdl.dtsi | 10 ++-
3 files changed, 311 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 1ade195..0a1718c 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -181,6 +181,189 @@
"di0", "di1";
};

+&gpr {
+ ipu1_csi0_mux: ipu1_csi0_mux@34 {
+ compatible = "imx-video-mux";
+ reg = <0x34 0x07>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "okay";
+
+ port@0 {
+ reg = <0>;
+
+ ipu1_csi0_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ ipu1_csi0_mux_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi0_mux_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+
+ ipu1_csi0_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+
+ ipu1_csi0_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@5 {
+ reg = <5>;
+
+ ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+ remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+ };
+ };
+ };
+
+ ipu1_csi1_mux: ipu1_csi1_mux@34 {
+ compatible = "imx-video-mux";
+ reg = <0x34 0x38>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "okay";
+
+ port@0 {
+ reg = <0>;
+
+ ipu1_csi1_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ ipu1_csi1_mux_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi1_mux_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+
+ ipu1_csi1_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+
+ ipu1_csi1_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@5 {
+ reg = <5>;
+
+ ipu1_csi1_mux_to_ipu1_csi1: endpoint {
+ remote-endpoint = <&ipu1_csi1_from_ipu1_csi1_mux>;
+ };
+ };
+ };
+};
+
+&ipu1_csi1 {
+ ipu1_csi1_from_ipu1_csi1_mux: endpoint {
+ remote-endpoint = <&ipu1_csi1_mux_to_ipu1_csi1>;
+ };
+};
+
+&mipi_csi {
+ port@0 {
+ reg = <0>;
+
+ mipi_csi_from_mipi_sensor: endpoint {
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ mipi_vc0_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+ };
+
+ mipi_vc0_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc0>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ mipi_vc1_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc1>;
+ };
+
+ mipi_vc1_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc1>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ mipi_vc2_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc2>;
+ };
+
+ mipi_vc2_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc2>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ mipi_vc3_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc3>;
+ };
+
+ mipi_vc3_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc3>;
+ };
+ };
+};
+
&vpu {
compatible = "fsl,imx6dl-vpu", "cnm,coda960";
};
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..56a314f 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -143,10 +143,18 @@

ipu2_csi0: port@0 {
reg = <0>;
+
+ ipu2_csi0_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu2_csi0>;
+ };
};

ipu2_csi1: port@1 {
reg = <1>;
+
+ ipu2_csi1_from_ipu2_csi1_mux: endpoint {
+ remote-endpoint = <&ipu2_csi1_mux_to_ipu2_csi1>;
+ };
};

ipu2_di0: port@2 {
@@ -266,6 +274,76 @@
};
};

+&gpr {
+ ipu1_csi0_mux: ipu1_csi0_mux@4 {
+ compatible = "imx-video-mux";
+ reg = <0x04 0x80000>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "okay";
+
+ port@0 {
+ reg = <0>;
+
+ ipu1_csi0_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ ipu1_csi0_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+ remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+ };
+ };
+ };
+
+ ipu2_csi1_mux: ipu2_csi1_mux@4 {
+ compatible = "imx-video-mux";
+ reg = <0x04 0x100000>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "okay";
+
+ port@0 {
+ reg = <0>;
+
+ ipu2_csi1_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu2_csi1_mux>;
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ ipu2_csi1_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu2_csi1_mux_to_ipu2_csi1: endpoint {
+ remote-endpoint = <&ipu2_csi1_from_ipu2_csi1_mux>;
+ };
+ };
+ };
+};
+
+&ipu1_csi1 {
+ ipu1_csi1_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi1>;
+ };
+};
+
&ldb {
clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
<&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -312,6 +390,47 @@
};
};

+&mipi_csi {
+ port@0 {
+ reg = <0>;
+
+ mipi_csi_from_mipi_sensor: endpoint {
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ mipi_vc0_to_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ mipi_vc1_to_ipu1_csi1: endpoint {
+ remote-endpoint = <&ipu1_csi1_from_mipi_vc1>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+
+ mipi_vc2_to_ipu2_csi0: endpoint {
+ remote-endpoint = <&ipu2_csi0_from_mipi_vc2>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+
+ mipi_vc3_to_ipu2_csi1_mux: endpoint {
+ remote-endpoint = <&ipu2_csi1_mux_from_mipi_vc3>;
+ };
+ };
+};
+
&mipi_dsi {
ports {
port@2 {
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 7b546e3..89218a4 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -799,8 +799,10 @@
};

gpr: iomuxc-gpr@020e0000 {
- compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+ compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
reg = <0x020e0000 0x38>;
+ #address-cells = <1>;
+ #size-cells = <0>;
};

iomuxc: iomuxc@020e0000 {
@@ -1127,6 +1129,8 @@
mipi_csi: mipi@021dc000 {
compatible = "fsl,imx-mipi-csi2";
reg = <0x021dc000 0x4000>;
+ #address-cells = <1>;
+ #size-cells = <0>;
interrupts = <0 100 0x04>, <0 101 0x04>;
clocks = <&clks IMX6QDL_CLK_HSI_TX>,
<&clks IMX6QDL_CLK_VIDEO_27M>,
@@ -1232,6 +1236,10 @@

ipu1_csi0: port@0 {
reg = <0>;
+
+ ipu1_csi0_from_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_to_ipu1_csi0>;
+ };
};

ipu1_csi1: port@1 {
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:05 PM12/29/16

This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO
Controller module. Video frames are received from the CSI and can
be routed to various sinks including the i.MX Image Converter for
scaling, color-space conversion, motion compensated deinterlacing,
and image rotation.
---
drivers/staging/media/imx/Makefile | 1 +
drivers/staging/media/imx/imx-smfc.c | 739 +++++++++++++++++++++++++++++++++++
2 files changed, 740 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-smfc.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 133672a..3559d7b 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

diff --git a/drivers/staging/media/imx/imx-smfc.c b/drivers/staging/media/imx/imx-smfc.c
new file mode 100644
index 0000000..75a3875
--- /dev/null
+++ b/drivers/staging/media/imx/imx-smfc.c
@@ -0,0 +1,739 @@
+/*
+ * V4L2 Capture SMFC Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of raw/unconverted video frames
+ * from the CSI, directly to memory via the Sensor Multi-FIFO Controller.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *

+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */

+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/timer.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/interrupt.h>
+#include <linux/spinlock.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W 176
+#define MIN_H 144
+#define MAX_W 8192
+#define MAX_H 4096
+#define W_ALIGN 4 /* multiple of 16 pixels */
+#define H_ALIGN 1 /* multiple of 2 lines */
+#define S_ALIGN 1 /* multiple of 2 */
+
+#define SMFC_NUM_PADS 2
+
+struct imx_smfc_priv {
+ struct device *dev;
+ struct ipu_soc *ipu;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;
+ struct media_pad pad[SMFC_NUM_PADS];
+ int ipu_id;
+ int smfc_id;
+ int input_pad;
+ int output_pad;
+
+ struct ipuv3_channel *smfc_ch;
+ struct ipu_smfc *smfc;
+
+ struct v4l2_mbus_framefmt format_mbus[SMFC_NUM_PADS];
+ const struct imx_media_pixfmt *cc[SMFC_NUM_PADS];
+
+ struct v4l2_mbus_config sensor_mbus_cfg;
+
+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct imx_media_dma_buf *next;
+
+ int ipu_buf_num; /* ipu double buffer index: 0-1 */
+
+ /* the sink that will receive the dma buffers */
+ struct v4l2_subdev *sink_sd;
+ struct v4l2_subdev *src_sd;
+
+ /*
+ * the CSI id and mipi virtual channel number at
+ * link validate
+ */
+ int csi_id;
+ int vc_num;
+
+ /* the attached sensor at stream on */
+ struct imx_media_subdev *sensor;
+
+ spinlock_t irqlock;
+ struct timer_list eof_timeout_timer;
+ int eof_irq;
+ int nfb4eof_irq;
+
+ bool stream_on; /* streaming is on */
+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;
+};
+
+static void imx_smfc_put_ipu_resources(struct imx_smfc_priv *priv)
+{
+ if (!IS_ERR_OR_NULL(priv->smfc_ch))
+ ipu_idmac_put(priv->smfc_ch);
+ priv->smfc_ch = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->smfc))
+ ipu_smfc_put(priv->smfc);
+ priv->smfc = NULL;
+}
+
+static int imx_smfc_get_ipu_resources(struct imx_smfc_priv *priv)
+{
+ int ch_num, ret;
+
+ priv->ipu = priv->md->ipu[priv->ipu_id];
+
+ ch_num = IPUV3_CHANNEL_CSI0 + priv->smfc_id;
+
+ priv->smfc = ipu_smfc_get(priv->ipu, ch_num);
+ if (IS_ERR(priv->smfc)) {
+ v4l2_err(&priv->sd, "failed to get SMFC\n");
+ ret = PTR_ERR(priv->smfc);

+ goto out;
+ }
+

+ priv->smfc_ch = ipu_idmac_get(priv->ipu, ch_num);
+ if (IS_ERR(priv->smfc_ch)) {
+ v4l2_err(&priv->sd, "could not get IDMAC channel %u\n", ch_num);
+ ret = PTR_ERR(priv->smfc_ch);
+ goto out;

+ }
+
+ return 0;

+out:
+ imx_smfc_put_ipu_resources(priv);

+ return ret;
+}
+

+static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
+{
+ struct imx_smfc_priv *priv = dev_id;
+ struct imx_media_dma_buf *done, *next;
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->irqlock, flags);
+
+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;
+ goto unlock;
+ }
+
+ /* inform CSI of this EOF so it can monitor frame intervals */
+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+
+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))
+ imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+ /* priv->next buffer is now the active one */
+ imx_media_dma_buf_set_active(priv->next);
+
+ /* bump the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+ if (ipu_idmac_buffer_is_ready(priv->smfc_ch, priv->ipu_buf_num))
+ ipu_idmac_clear_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+ /* get next queued buffer */
+ next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+ ipu_cpmem_set_buffer(priv->smfc_ch, priv->ipu_buf_num, next->phys);
+ ipu_idmac_select_buffer(priv->smfc_ch, priv->ipu_buf_num);
+
+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ priv->next = next;
+
+unlock:
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t imx_smfc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct imx_smfc_priv *priv = dev_id;
+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+
+ v4l2_err(&priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void imx_smfc_eof_timeout(unsigned long data)
+{
+ struct imx_smfc_priv *priv = (struct imx_smfc_priv *)data;
+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+
+ v4l2_err(&priv->sd, "EOF timeout\n");
+
+ v4l2_subdev_notify_event(&priv->sd, &ev);
+}
+
+/* init the SMFC IDMAC channel */
+static void imx_smfc_setup_channel(struct imx_smfc_priv *priv)
+{
+ struct v4l2_mbus_framefmt *infmt, *outfmt;
+ struct imx_media_dma_buf *buf0, *buf1;
+ unsigned int burst_size;
+ struct ipu_image image;
+ bool passthrough;
+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+
+ ipu_cpmem_zero(priv->smfc_ch);
+
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+
+ image.phys0 = buf0->phys;
+ image.phys1 = buf1->phys;
+ ipu_cpmem_set_image(priv->smfc_ch, &image);
+
+ burst_size = (outfmt->width & 0xf) ? 8 : 16;
+
+ ipu_cpmem_set_burstsize(priv->smfc_ch, burst_size);
+
+ /*
+ * If the sensor uses 16-bit parallel CSI bus, we must handle
+ * the data internally in the IPU as 16-bit generic, aka
+ * passthrough mode.
+ */
+ passthrough = (priv->sensor_mbus_cfg.type != V4L2_MBUS_CSI2 &&
+ priv->sensor->sensor_ep.bus.parallel.bus_width >= 16);
+
+ if (passthrough)
+ ipu_cpmem_set_format_passthrough(priv->smfc_ch, 16);
+
+ /*
+ * Set the channel for the direct CSI-->memory via SMFC
+ * use-case to very high priority, by enabling the watermark
+ * signal in the SMFC, enabling WM in the channel, and setting
+ * the channel priority to high.
+ *
+ * Refer to the i.mx6 rev. D TRM Table 36-8: Calculated priority
+ * value.
+ *
+ * The WM's are set very low by intention here to ensure that
+ * the SMFC FIFOs do not overflow.
+ */
+ ipu_smfc_set_watermark(priv->smfc, 0x02, 0x01);
+ ipu_cpmem_set_high_priority(priv->smfc_ch);
+ ipu_idmac_enable_watermark(priv->smfc_ch, true);
+ ipu_cpmem_set_axi_id(priv->smfc_ch, 0);
+ ipu_idmac_lock_enable(priv->smfc_ch, 8);
+
+ burst_size = ipu_cpmem_get_burstsize(priv->smfc_ch);
+ burst_size = passthrough ?
+ (burst_size >> 3) - 1 : (burst_size >> 2) - 1;
+
+ ipu_smfc_set_burstsize(priv->smfc, burst_size);
+
+ if (outfmt->field == V4L2_FIELD_NONE &&
+ (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+ infmt->field == V4L2_FIELD_ALTERNATE))
+ ipu_cpmem_interlaced_scan(priv->smfc_ch,
+ image.pix.bytesperline);
+
+ ipu_idmac_set_double_buffer(priv->smfc_ch, true);
+}
+
+static void imx_smfc_unsetup(struct imx_smfc_priv *priv)
+{
+ ipu_idmac_disable_channel(priv->smfc_ch);
+ ipu_smfc_disable(priv->smfc);
+}
+
+static void imx_smfc_setup(struct imx_smfc_priv *priv)
+{
+ imx_smfc_setup_channel(priv);
+
+ ipu_cpmem_dump(priv->smfc_ch);
+ ipu_dump(priv->ipu);
+
+ ipu_smfc_enable(priv->smfc);
+
+ /* set buffers ready */
+ ipu_idmac_select_buffer(priv->smfc_ch, 0);
+ ipu_idmac_select_buffer(priv->smfc_ch, 1);
+
+ /* enable the channels */
+ ipu_idmac_enable_channel(priv->smfc_ch);
+}
+
+static int imx_smfc_start(struct imx_smfc_priv *priv)

+{
+ int ret;
+

+ if (!priv->sensor) {
+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return -EINVAL;
+ }
+

+ ret = imx_smfc_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ ipu_smfc_map_channel(priv->smfc, priv->csi_id, priv->vc_num);
+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);
+ if (ret)
+ goto out_put_ipu;
+
+ priv->ipu_buf_num = 0;
+
+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+
+ imx_smfc_setup(priv);
+
+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+ priv->smfc_ch,
+ IPU_IRQ_NFB4EOF);
+ ret = devm_request_irq(priv->dev, priv->nfb4eof_irq,
+ imx_smfc_nfb4eof_interrupt, 0,
+ "imx-smfc-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&priv->sd,
+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_unsetup;
+ }
+
+ priv->eof_irq = ipu_idmac_channel_irq(priv->ipu, priv->smfc_ch,
+ IPU_IRQ_EOF);
+
+ ret = devm_request_irq(priv->dev, priv->eof_irq,
+ imx_smfc_eof_interrupt, 0,
+ "imx-smfc-eof", priv);
+ if (ret) {
+ v4l2_err(&priv->sd,
+ "Error registering eof irq: %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ /* start the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+out_free_nfb4eof_irq:
+ devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+ imx_smfc_unsetup(priv);
+out_put_ipu:
+ imx_smfc_put_ipu_resources(priv);

+ return ret;
+}
+

+static void imx_smfc_stop(struct imx_smfc_priv *priv)
+{
+ unsigned long flags;
+ int ret;
+
+ /* mark next EOF interrupt as the last before stream off */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ priv->last_eof = true;
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+
+ /*
+ * and then wait for interrupt handler to mark completion.
+ */
+ ret = wait_for_completion_timeout(
+ &priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+ if (ret == 0)
+ v4l2_warn(&priv->sd, "wait last EOF timeout\n");
+
+ devm_free_irq(priv->dev, priv->eof_irq, priv);
+ devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+
+ imx_smfc_unsetup(priv);
+
+ /* cancel the EOF timeout timer */
+ del_timer_sync(&priv->eof_timeout_timer);
+
+ priv->out_ring = NULL;
+
+ /* inform sink that the buffer ring can now be freed */
+ v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+
+ imx_smfc_put_ipu_resources(priv);
+}
+
+static int imx_smfc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)
+ return -EPIPE;

+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !priv->stream_on)
+ ret = imx_smfc_start(priv);
+ else if (!enable && priv->stream_on)
+ imx_smfc_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int imx_smfc_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (code->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+
+ return imx_media_enum_format(&code->code, code->index,
+ true, code->pad == priv->output_pad);
+}
+
+static int imx_smfc_get_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int imx_smfc_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+ struct v4l2_mbus_framefmt *infmt, *outfmt;
+ const struct imx_media_pixfmt *cc, *incc;
+ bool allow_planar;
+ u32 code;
+
+ if (sdformat->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+ allow_planar = (sdformat->pad == priv->output_pad);
+
+ cc = imx_media_find_format(0, sdformat->format.code,
+ true, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, false);
+ cc = imx_media_find_format(0, code, true, false);
+ sdformat->format.code = cc->codes[0];
+ }
+
+ v4l_bound_align_image(&sdformat->format.width, MIN_W, MAX_W,
+ W_ALIGN, &sdformat->format.height,
+ MIN_H, MAX_H, H_ALIGN, S_ALIGN);
+
+ if (sdformat->pad == priv->output_pad) {
+ incc = priv->cc[priv->input_pad];
+ sdformat->format.width = infmt->width;
+ sdformat->format.height = infmt->height;
+ if (sdformat->format.field != V4L2_FIELD_NONE)
+ sdformat->format.field = infmt->field;
+ if (cc->cs != incc->cs) {
+ sdformat->format.code = infmt->code;
+ cc = imx_media_find_format(0, sdformat->format.code,
+ true, false);
+ }
+ }
+

+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int imx_smfc_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)

+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ }
+

+ return 0;
+ }
+

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;

+ return 0;
+ }
+

+ /* must attach to CSI source */
+ if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_CSI))
+ return -EINVAL;
+

+ return 0;
+}
+

+static int imx_smfc_link_validate(struct v4l2_subdev *sd,
+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ switch (priv->src_sd->grp_id) {
+ case IMX_MEDIA_GRP_ID_CSI0:
+ priv->csi_id = 0;
+ break;
+ case IMX_MEDIA_GRP_ID_CSI1:
+ priv->csi_id = 1;
+ break;
+ default:

+ return -EINVAL;
+ }
+

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ priv->vc_num = 0;
+ if (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+ /* see NOTE in imx-csi.c */
+#if 0
+ priv->vc_num = imx_media_find_mipi_csi2_channel(
+ priv->md, &priv->sd.entity);
+ if (priv->vc_num < 0)
+ return vc_num;
+#endif

+ }
+
+ return 0;
+}
+
+/*

+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int imx_smfc_registered(struct v4l2_subdev *sd)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i;
+

+ /* get media device */

+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+ return -EINVAL;
+
+ for (i = 0; i < SMFC_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;

+
+ /* set a default mbus format */

+ imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);
+ }
+
+ return media_entity_pads_init(&sd->entity, SMFC_NUM_PADS, priv->pad);
+}
+
+static struct media_entity_operations imx_smfc_entity_ops = {
+ .link_setup = imx_smfc_link_setup,
+ .link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_video_ops imx_smfc_video_ops = {
+ .s_stream = imx_smfc_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imx_smfc_pad_ops = {
+ .enum_mbus_code = imx_smfc_enum_mbus_code,
+ .get_fmt = imx_smfc_get_fmt,
+ .set_fmt = imx_smfc_set_fmt,
+ .link_validate = imx_smfc_link_validate,
+};
+
+static struct v4l2_subdev_ops imx_smfc_subdev_ops = {
+ .video = &imx_smfc_video_ops,
+ .pad = &imx_smfc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imx_smfc_internal_ops = {
+ .registered = imx_smfc_registered,
+};
+
+static int imx_smfc_probe(struct platform_device *pdev)
+{
+ struct imx_media_internal_sd_platformdata *pdata;
+ struct imx_smfc_priv *priv;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, &priv->sd);
+ priv->dev = &pdev->dev;
+
+ pdata = priv->dev->platform_data;
+ priv->ipu_id = pdata->ipu_id;
+
+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;
+ priv->eof_timeout_timer.function = imx_smfc_eof_timeout;
+ spin_lock_init(&priv->irqlock);
+
+ v4l2_subdev_init(&priv->sd, &imx_smfc_subdev_ops);
+ v4l2_set_subdevdata(&priv->sd, priv);
+ priv->sd.internal_ops = &imx_smfc_internal_ops;
+ priv->sd.entity.ops = &imx_smfc_entity_ops;
+ /* FIXME: this the right function? */
+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;
+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+ /* get our group id and SMFC id */
+ priv->sd.grp_id = pdata->grp_id;
+ priv->smfc_id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;
+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+ return v4l2_async_register_subdev(&priv->sd);
+}
+
+static int imx_smfc_remove(struct platform_device *pdev)
+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+ struct imx_smfc_priv *priv = container_of(sd, struct imx_smfc_priv, sd);
+
+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);
+ v4l2_device_unregister_subdev(sd);

+
+ return 0;
+}
+

+static const struct platform_device_id imx_smfc_ids[] = {
+ { .name = "imx-ipuv3-smfc" },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
+
+static struct platform_driver imx_smfc_driver = {
+ .probe = imx_smfc_probe,
+ .remove = imx_smfc_remove,
+ .id_table = imx_smfc_ids,
+ .driver = {
+ .name = "imx-ipuv3-smfc",
+ .owner = THIS_MODULE,
+ },
+};
+module_platform_driver(imx_smfc_driver);
+
+MODULE_DESCRIPTION("i.MX SMFC subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");

+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-smfc");
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:05 PM12/29/16

The sabreauto uses a steering pin to select between the SDA signal on
i2c3 bus, and a data-in pin for an SPI NOR chip. Use i2cmux to control
this steering pin. Idle state of the i2cmux selects SPI NOR. This is not
a classic way to use i2cmux, since one side of the mux selects something
other than an i2c bus, but it works and is probably the cleanest
solution. Note that if one thread is attempting to access SPI NOR while
another thread is accessing i2c3, the SPI NOR access will fail since the
i2cmux has selected the SDA pin rather than SPI NOR data-in. This couldn't
be avoided in any case, the board is not designed to allow concurrent
i2c3 and SPI NOR functions (and the default device-tree does not enable
SPI NOR anyway).

Devices hanging off i2c3 should now be defined under i2cmux, so
that the steering pin can be properly controlled to access those
devices. The port expanders (MAX7310) are thus moved into i2cmux.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 65 +++++++++++++++++++++-----------
1 file changed, 44 insertions(+), 21 deletions(-)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 52390ba..4a6d038 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -108,6 +108,44 @@
default-brightness-level = <7>;
status = "okay";
};
+
+ i2cmux {
+ compatible = "i2c-mux-gpio";

+ #address-cells = <1>;
+ #size-cells = <0>;

+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_i2c3mux>;
+ mux-gpios = <&gpio5 4 0>;
+ i2c-parent = <&i2c3>;
+ idle-state = <0>;
+
+ i2c@1 {

+ #address-cells = <1>;
+ #size-cells = <0>;

+ reg = <1>;
+

+ max7310_a: gpio@30 {
+ compatible = "maxim,max7310";
+ reg = <0x30>;
+ gpio-controller;
+ #gpio-cells = <2>;
+ };
+
+ max7310_b: gpio@32 {
+ compatible = "maxim,max7310";
+ reg = <0x32>;
+ gpio-controller;
+ #gpio-cells = <2>;
+ };
+
+ max7310_c: gpio@34 {
+ compatible = "maxim,max7310";
+ reg = <0x34>;
+ gpio-controller;
+ #gpio-cells = <2>;
+ };
+ };
+ };
};

&clks {
@@ -291,27 +329,6 @@
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_i2c3>;
status = "okay";
-
- max7310_a: gpio@30 {
- compatible = "maxim,max7310";
- reg = <0x30>;
- gpio-controller;
- #gpio-cells = <2>;
- };
-
- max7310_b: gpio@32 {
- compatible = "maxim,max7310";
- reg = <0x32>;
- gpio-controller;
- #gpio-cells = <2>;
- };
-
- max7310_c: gpio@34 {
- compatible = "maxim,max7310";
- reg = <0x34>;
- gpio-controller;
- #gpio-cells = <2>;
- };
};

&iomuxc {
@@ -419,6 +436,12 @@
>;
};

+ pinctrl_i2c3mux: i2c3muxgrp {
+ fsl,pins = <
+ MX6QDL_PAD_EIM_A24__GPIO5_IO04 0x80000000

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

The reset pin to the port expander chip (MAX7310) is controlled by a gpio,
so define a reset-gpios property to control it. There are three MAX7310's
on the SabreAuto CPU card (max7310_[abc]), but all use the same pin for
their reset. Since all can't acquire the same pin, assign it to max7310_b,
that chip is needed by more functions (usb and adv7180).

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 4a6d038..516bac6 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -136,6 +136,9 @@
reg = <0x32>;
gpio-controller;
#gpio-cells = <2>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_max7310>;
+ reset-gpios = <&gpio1 15 GPIO_ACTIVE_LOW>;
};

max7310_c: gpio@34 {
@@ -442,6 +445,12 @@
>;
};

+ pinctrl_max7310: max7310grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x80000000

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
clocks.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 53e6e63..7b546e3 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1125,7 +1125,14 @@
};

mipi_csi: mipi@021dc000 {
+ compatible = "fsl,imx-mipi-csi2";
reg = <0x021dc000 0x4000>;
+ interrupts = <0 100 0x04>, <0 101 0x04>;
+ clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+ <&clks IMX6QDL_CLK_VIDEO_27M>,
+ <&clks IMX6QDL_CLK_EIM_SEL>;
+ clock-names = "dphy_clk", "cfg_clk", "pix_clk";
+ status = "disabled";
};

mipi_dsi: mipi@021e0000 {
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6q.dtsi | 4 ++++
arch/arm/boot/dts/imx6qdl.dtsi | 8 ++++++++
2 files changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index 56a314f..2fbe0b3 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -454,3 +454,7 @@
&vpu {
compatible = "fsl,imx6q-vpu", "cnm,coda960";
};
+
+&media0 {
+ ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 89218a4..12ae045 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1296,5 +1296,13 @@
};
};
};
+
+ media0: media@0 {
+ compatible = "fsl,imx-media";
+ ports = <&ipu1_csi0>, <&ipu1_csi1>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+ };
};
};
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

This is a media entity subdevice for the i.MX Camera
Serial Interface module.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Kconfig | 13 +
drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/imx-csi.c | 638 ++++++++++++++++++++++++++++++++++++
3 files changed, 653 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-csi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index bfde58d..ce2d2c8 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
Say yes here to enable support for video4linux media controller
driver for the i.MX5/6 SOC.

+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA
+ tristate "i.MX5/6 Camera driver"
+ depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+ select VIDEOBUF2_DMA_CONTIG

+ default y
+ ---help---

+ A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index ef9f11b..133672a 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \

obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
new file mode 100644
index 0000000..975eafb
--- /dev/null
+++ b/drivers/staging/media/imx/imx-csi.c
@@ -0,0 +1,638 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+#define CSI_NUM_PADS 2
+
+struct csi_priv {

+ struct device *dev;
+ struct ipu_soc *ipu;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[CSI_NUM_PADS];
+ struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
+ struct v4l2_mbus_config sensor_mbus_cfg;
+ struct v4l2_rect crop;
+ struct ipu_csi *csi;
+ int csi_id;

+ int input_pad;
+ int output_pad;

+ bool power_on; /* power is on */

+ bool stream_on; /* streaming is on */
+

+ /* the sink for the captured frames */
+ struct v4l2_subdev *sink_sd;
+ enum ipu_csi_dest dest;

+ struct v4l2_subdev *src_sd;
+

+ struct v4l2_ctrl_handler ctrl_hdlr;
+ struct imx_media_fim *fim;

+
+ /* the attached sensor at stream on */
+ struct imx_media_subdev *sensor;
+};
+

+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{
+ return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */
+static int csi_setup(struct csi_priv *priv)
+{
+ struct v4l2_mbus_framefmt infmt;
+
+ ipu_csi_set_window(priv->csi, &priv->crop);
+
+ /*
+ * the ipu-csi doesn't understand ALTERNATE, but it only
+ * needs to know whether the stream is interlaced, so set
+ * to INTERLACED if infmt field is ALTERNATE.
+ */
+ infmt = priv->format_mbus[priv->input_pad];
+ if (infmt.field == V4L2_FIELD_ALTERNATE)
+ infmt.field = V4L2_FIELD_INTERLACED;
+
+ ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
+
+ ipu_csi_set_dest(priv->csi, priv->dest);
+
+ ipu_csi_dump(priv->csi);

+
+ return 0;
+}
+

+static int csi_start(struct csi_priv *priv)

+{
+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return -EINVAL;
+ }
+

+ ret = csi_setup(priv);

+ if (ret)
+ return ret;
+

+ /* start the frame interval monitor */
+ ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);

+ if (ret)
+ return ret;
+

+ ret = ipu_csi_enable(priv->csi);
+ if (ret) {
+ v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);

+ return ret;
+ }
+

+ return 0;
+}
+

+static void csi_stop(struct csi_priv *priv)
+{
+ /* stop the frame interval monitor */
+ imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+ ipu_csi_disable(priv->csi);
+}
+
+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+
+ if (!priv->src_sd || !priv->sink_sd)
+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = csi_start(priv);

+ else if (!enable && priv->stream_on)

+ csi_stop(priv);

+
+ if (!ret)
+ priv->stream_on = enable;
+ return ret;
+}
+

+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+

+ v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+ if (on != priv->power_on)
+ ret = imx_media_fim_set_power(priv->fim, on);
+
+ if (!ret)
+ priv->power_on = on;

+ return ret;
+}
+

+static int csi_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_subdev *remote_sd;
+
+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SINK) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;
+ return 0;
+ }
+
+ return 0;
+ }
+
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;

+ return 0;
+ }
+

+ /* set CSI destination */
+ switch (remote_sd->grp_id) {
+ case IMX_MEDIA_GRP_ID_SMFC0:
+ case IMX_MEDIA_GRP_ID_SMFC1:
+ case IMX_MEDIA_GRP_ID_SMFC2:
+ case IMX_MEDIA_GRP_ID_SMFC3:
+ priv->dest = IPU_CSI_DEST_IDMAC;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPVF:
+ priv->dest = IPU_CSI_DEST_VDIC;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPENC:
+ priv->dest = IPU_CSI_DEST_IC;

+ break;
+ default:
+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int csi_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ bool is_csi2;

+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ ret = PTR_ERR(priv->sensor);
+ priv->sensor = NULL;
+ return ret;
+ }
+
+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+ &priv->sensor_mbus_cfg);
+ if (ret)
+ return ret;
+

+ is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+ if (is_csi2) {
+ int vc_num = 0;
+ /*
+ * NOTE! It seems the virtual channels from the mipi csi-2
+ * receiver are used only for routing by the video mux's,
+ * or for hard-wired routing to the CSI's. Once the stream
+ * enters the CSI's however, they are treated internally
+ * in the IPU as virtual channel 0.
+ */
+#if 0
+ vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+ &priv->sd.entity);
+ if (vc_num < 0)
+ return vc_num;
+#endif
+ ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+ &priv->format_mbus[priv->input_pad]);
+ }
+
+ /* select either parallel or MIPI-CSI2 as input to CSI */
+ ipu_set_csi_src_mux(priv->ipu, priv->csi_id, is_csi2);

+
+ return 0;
+}
+

+static int csi_eof_isr(struct v4l2_subdev *sd, u32 status, bool *handled)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ struct timespec cur_ts;
+
+ ktime_get_ts(&cur_ts);
+
+ /* call frame interval monitor */
+ imx_media_fim_eof_monitor(priv->fim, &cur_ts);

+
+ return 0;
+}
+

+static int csi_try_crop(struct csi_priv *priv, struct v4l2_rect *crop)
+{
+ struct v4l2_mbus_framefmt *infmt;
+ struct imx_media_subdev *sensor;
+ v4l2_std_id std;
+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ ret = v4l2_subdev_call(sensor->sd, video, g_mbus_config,

+ &priv->sensor_mbus_cfg);
+ if (ret)
+ return ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+

+ crop->width = min_t(__u32, infmt->width, crop->width);
+ if (crop->left + crop->width > infmt->width)
+ crop->left = infmt->width - crop->width;
+ /* adjust crop left/width to h/w alignment restrictions */
+ crop->left &= ~0x3;
+ crop->width &= ~0x7;
+
+ /*
+ * FIXME: not sure why yet, but on interlaced bt.656,
+ * changing the vertical cropping causes loss of vertical
+ * sync, so fix it to NTSC/PAL active lines. NTSC contains
+ * 2 extra lines of active video that need to be cropped.
+ */
+ if (priv->sensor_mbus_cfg.type == V4L2_MBUS_BT656) {
+ ret = v4l2_subdev_call(sensor->sd, video, g_std, &std);

+ if (ret)
+ return ret;

+ if (std & V4L2_STD_525_60) {
+ crop->top = 2;
+ crop->height = 480;
+ } else {
+ crop->top = 0;
+ crop->height = 576;
+ }
+ } else {
+ crop->height = min_t(__u32, infmt->height, crop->height);
+ if (crop->top + crop->height > infmt->height)
+ crop->top = infmt->height - crop->height;

+ }
+
+ return 0;
+}
+

+static int csi_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= CSI_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int csi_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ struct v4l2_rect crop;
+ int ret;
+
+ if (sdformat->pad >= CSI_NUM_PADS)

+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (sdformat->pad == priv->output_pad) {

+ sdformat->format.code = infmt->code;

+ sdformat->format.field = infmt->field;

+ crop.left = priv->crop.left;
+ crop.top = priv->crop.top;
+ crop.width = sdformat->format.width;
+ crop.height = sdformat->format.height;
+ ret = csi_try_crop(priv, &crop);

+ if (ret)
+ return ret;

+ sdformat->format.width = crop.width;
+ sdformat->format.height = crop.height;

+ }
+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;

+ /* Update the crop window if this is output pad */

+ if (sdformat->pad == priv->output_pad)

+ priv->crop = crop;

+ }
+
+ return 0;
+}
+

+static int csi_get_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_selection *sel)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ struct v4l2_mbus_framefmt *infmt;
+
+ if (sel->pad != priv->output_pad)
+ return -EINVAL;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+

+ switch (sel->target) {
+ case V4L2_SEL_TGT_CROP_BOUNDS:
+ sel->r.left = 0;
+ sel->r.top = 0;
+ sel->r.width = infmt->width;
+ sel->r.height = infmt->height;
+ break;
+ case V4L2_SEL_TGT_CROP:
+ sel->r = priv->crop;

+ break;
+ default:
+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int csi_set_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_selection *sel)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ struct v4l2_mbus_framefmt *outfmt;
+ int ret;
+
+ if (sel->pad != priv->output_pad ||
+ sel->target != V4L2_SEL_TGT_CROP)

+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ /*
+ * Modifying the crop rectangle always changes the format on the source
+ * pad. If the KEEP_CONFIG flag is set, just return the current crop
+ * rectangle.
+ */
+ if (sel->flags & V4L2_SEL_FLAG_KEEP_CONFIG) {
+ sel->r = priv->crop;
+ if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+ cfg->try_crop = sel->r;

+ return 0;
+ }
+

+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ ret = csi_try_crop(priv, &sel->r);

+ if (ret)
+ return ret;
+

+ if (sel->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_crop = sel->r;
+ } else {
+ priv->crop = sel->r;
+ /* Update the source format */
+ outfmt->width = sel->r.width;
+ outfmt->height = sel->r.height;

+ }
+
+ return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int csi_registered(struct v4l2_subdev *sd)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i, ret;

+
+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ /* get handle to IPU CSI */
+ priv->csi = ipu_csi_get(priv->ipu, priv->csi_id);
+ if (IS_ERR(priv->csi)) {
+ v4l2_err(&priv->sd, "failed to get CSI %d\n", priv->csi_id);
+ return PTR_ERR(priv->csi);
+ }

+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+ if (IS_ERR(imxsd)) {

+ ret = PTR_ERR(imxsd);
+ goto put_csi;
+ }

+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {

+ ret = -EINVAL;
+ goto put_csi;
+ }
+
+ for (i = 0; i < CSI_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];
+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;
+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ NULL);
+ if (ret)
+ goto put_csi;
+ }
+
+ priv->fim = imx_media_fim_init(&priv->sd);
+ if (IS_ERR(priv->fim)) {
+ ret = PTR_ERR(priv->fim);
+ goto put_csi;
+ }
+
+ ret = media_entity_pads_init(&sd->entity, CSI_NUM_PADS, priv->pad);
+ if (ret)
+ goto free_fim;
+
+ return 0;
+free_fim:
+ imx_media_fim_free(priv->fim);
+put_csi:
+ ipu_csi_put(priv->csi);

+ return ret;
+}
+

+static struct media_entity_operations csi_entity_ops = {
+ .link_setup = csi_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_core_ops csi_core_ops = {
+ .s_power = csi_s_power,
+ .interrupt_service_routine = csi_eof_isr,
+};
+
+static struct v4l2_subdev_video_ops csi_video_ops = {
+ .s_stream = csi_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops csi_pad_ops = {
+ .get_fmt = csi_get_fmt,
+ .set_fmt = csi_set_fmt,
+ .get_selection = csi_get_selection,
+ .set_selection = csi_set_selection,
+ .link_validate = csi_link_validate,
+};
+
+static struct v4l2_subdev_ops csi_subdev_ops = {
+ .core = &csi_core_ops,
+ .video = &csi_video_ops,
+ .pad = &csi_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops csi_internal_ops = {
+ .registered = csi_registered,
+};
+
+static int imx_csi_probe(struct platform_device *pdev)
+{
+ struct ipu_client_platformdata *pdata;
+ struct csi_priv *priv;
+ int ret;
+

+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, &priv->sd);
+ priv->dev = &pdev->dev;
+

+ /* get parent IPU */
+ priv->ipu = dev_get_drvdata(priv->dev->parent);
+
+ /* get our CSI id */

+ pdata = priv->dev->platform_data;

+ priv->csi_id = pdata->csi;
+
+ v4l2_subdev_init(&priv->sd, &csi_subdev_ops);
+ v4l2_set_subdevdata(&priv->sd, priv);
+ priv->sd.internal_ops = &csi_internal_ops;
+ priv->sd.entity.ops = &csi_entity_ops;

+ /* FIXME: this the right function? */
+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;

+ priv->sd.grp_id = priv->csi_id ?
+ IMX_MEDIA_GRP_ID_CSI1 : IMX_MEDIA_GRP_ID_CSI0;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;
+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ imx_media_grp_id_to_sd_name(priv->sd.name, sizeof(priv->sd.name),
+ priv->sd.grp_id, ipu_get_num(priv->ipu));
+
+ v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+ priv->sd.ctrl_handler = &priv->ctrl_hdlr;
+
+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ goto free_ctrls;
+
+ return 0;

+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ return ret;
+}
+

+static int imx_csi_remove(struct platform_device *pdev)

+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct csi_priv *priv = sd_to_dev(sd);
+
+ imx_media_fim_free(priv->fim);

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);
+ v4l2_device_unregister_subdev(sd);
+

+ if (!IS_ERR_OR_NULL(priv->csi))
+ ipu_csi_put(priv->csi);

+
+ return 0;
+}
+

+static const struct platform_device_id imx_csi_ids[] = {
+ { .name = "imx-ipuv3-csi" },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, imx_csi_ids);
+
+static struct platform_driver imx_csi_driver = {
+ .probe = imx_csi_probe,
+ .remove = imx_csi_remove,
+ .id_table = imx_csi_ids,
+ .driver = {
+ .name = "imx-ipuv3-csi",

+ .owner = THIS_MODULE,
+ },
+};

+module_platform_driver(imx_csi_driver);
+
+MODULE_DESCRIPTION("i.MX CSI subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-csi");
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

Add optional reset-gpios pin control. If present, de-assert the
specified reset gpio pin to bring the chip out of reset.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
Cc: Linus Walleij <linus....@linaro.org>
Cc: Alexandre Courbot <gnu...@gmail.com>
Cc: linux...@vger.kernel.org
Cc: linux-...@vger.kernel.org

---

v2:
- documented optional reset-gpios property in
Documentation/devicetree/bindings/gpio/gpio-pca953x.txt.
---
Documentation/devicetree/bindings/gpio/gpio-pca953x.txt | 4 ++++
drivers/gpio/gpio-pca953x.c | 17 +++++++++++++++++
2 files changed, 21 insertions(+)

diff --git a/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt b/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt
index 08dd15f..da54f4c 100644
--- a/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt
+++ b/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt
@@ -29,6 +29,10 @@ Required properties:
onsemi,pca9654
exar,xra1202

+Optional properties:
+ - reset-gpios: GPIO specification for the RESET input
+
+
Example:

diff --git a/drivers/gpio/gpio-pca953x.c b/drivers/gpio/gpio-pca953x.c
index d5d72d8..d1c0bd5 100644
--- a/drivers/gpio/gpio-pca953x.c
+++ b/drivers/gpio/gpio-pca953x.c
@@ -22,6 +22,7 @@
#include <linux/of_platform.h>
#include <linux/acpi.h>
#include <linux/regulator/consumer.h>
+#include <linux/gpio/consumer.h>

#define PCA953X_INPUT 0
#define PCA953X_OUTPUT 1
@@ -133,6 +134,7 @@ struct pca953x_chip {
const char *const *names;
unsigned long driver_data;
struct regulator *regulator;
+ struct gpio_desc *reset_gpio;

const struct pca953x_reg_config *regs;

@@ -756,6 +758,21 @@ static int pca953x_probe(struct i2c_client *client,
} else {
chip->gpio_start = -1;
irq_base = 0;
+
+ /* see if we need to de-assert a reset pin */
+ chip->reset_gpio = devm_gpiod_get_optional(&client->dev,
+ "reset",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(chip->reset_gpio)) {
+ dev_err(&client->dev, "request for reset pin failed\n");
+ return PTR_ERR(chip->reset_gpio);
+ }
+
+ if (chip->reset_gpio) {
+ /* bring chip out of reset */
+ dev_info(&client->dev, "releasing reset\n");
+ gpiod_set_value(chip->reset_gpio, 0);
+ }
}

chip->client = client;
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

This is the camera interface driver that provides the v4l2
user interface. Frames can be received from various sources:

- directly from SMFC for capturing unconverted images directly from
camera sensors.

- from the IC pre-process encode task.

- from the IC pre-process viewfinder task.

- from the IC post-process task.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 2 +-
drivers/staging/media/imx/imx-camif.c | 1011 +++++++++++++++++++++++++++++++++
2 files changed, 1012 insertions(+), 1 deletion(-)
create mode 100644 drivers/staging/media/imx/imx-camif.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index d2a962c..fe9e992 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
new file mode 100644
index 0000000..034a1ff
--- /dev/null
+++ b/drivers/staging/media/imx/imx-camif.c
@@ -0,0 +1,1011 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.

+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/timer.h>
+#include <linux/sched.h>
+#include <linux/slab.h>

+#include <linux/spinlock.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <linux/of_platform.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>

+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-event.h>
+#include <video/imx-ipu-v3.h>

+#include <media/imx.h>
+#include "imx-media.h"
+

+#define DEVICE_NAME "imx-media-camif"
+
+#define CAMIF_NUM_PADS 2
+
+#define CAMIF_DQ_TIMEOUT 5000
+
+struct camif_priv;
+
+struct camif_priv {
+ struct device *dev;
+ struct video_device vfd;
+ struct media_pipeline mp;

+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[CAMIF_NUM_PADS];
+ struct media_pad vd_pad;
+ int id;

+ int input_pad;
+ int output_pad;
+

+ struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+ const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+
+ /* dma buffer ring */
+ struct imx_media_dma_buf_ring *in_ring;

+ struct v4l2_subdev *src_sd;
+

+ struct mutex mutex; /* capture device mutex */
+ spinlock_t q_lock; /* protect ready_q */
+
+ /* buffer queue used in videobuf2 */
+ struct vb2_queue buffer_queue;
+
+ /* streaming buffer queue */
+ struct list_head ready_q;
+
+ /* controls inherited from subdevs */

+ struct v4l2_ctrl_handler ctrl_hdlr;
+

+ /* misc status */
+ int current_input; /* the current input */
+ v4l2_std_id current_std; /* current standard */
+ bool stop; /* streaming is stopping */
+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT SZ_64M
+
+static struct vb2_ops camif_qops;
+
+/*
+ * Video ioctls follow
+ */
+
+static int vidioc_querycap(struct file *file, void *fh,
+ struct v4l2_capability *cap)
+{
+ strncpy(cap->driver, DEVICE_NAME, sizeof(cap->driver) - 1);
+ strncpy(cap->card, DEVICE_NAME, sizeof(cap->card) - 1);
+ cap->bus_info[0] = 0;
+ cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+ cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

+
+ return 0;
+}
+

+static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_fmtdesc *f)
+{
+ const struct imx_media_pixfmt *cc;
+ u32 code;
+ int ret;
+
+ ret = imx_media_enum_format(&code, f->index, true, true);

+ if (ret)
+ return ret;

+ cc = imx_media_find_format(0, code, true, true);
+ if (!cc)
+ return -EINVAL;
+
+ f->pixelformat = cc->fourcc;

+
+ return 0;
+}
+

+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct v4l2_mbus_framefmt *outfmt;
+

+ /* user format is the same as the format from output pad */

+ outfmt = &priv->format_mbus[priv->output_pad];

+ return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
+}
+
+static int camif_try_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ return camif_g_fmt_vid_cap(file, fh, f);
+}
+
+static int camif_s_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct camif_priv *priv = video_drvdata(file);
+
+ if (vb2_is_busy(&priv->buffer_queue)) {
+ v4l2_err(&priv->sd, "%s queue busy\n", __func__);

+ return -EBUSY;
+ }
+

+ return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+

+ return v4l2_subdev_call(sensor->sd, video, querystd, std);
+}
+
+static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
+{
+ struct camif_priv *priv = video_drvdata(file);
+
+ *std = priv->current_std;

+ return 0;
+}
+

+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct imx_media_subdev *sensor;

+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+

+ if (vb2_is_busy(&priv->buffer_queue))
+ return -EBUSY;
+
+ ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
+ if (ret < 0)
+ return ret;
+
+ priv->current_std = std;

+ return 0;
+}
+

+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+

+ if (index >= sensor->input.num)
+ return -EINVAL;
+
+ input->type = V4L2_INPUT_TYPE_CAMERA;
+ strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+ if (index == priv->current_input) {
+ v4l2_subdev_call(sensor->sd, video, g_input_status,
+ &input->status);
+ v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
+ } else {
+ input->status = V4L2_IN_ST_NO_SIGNAL;
+ input->std = V4L2_STD_UNKNOWN;

+ }
+
+ return 0;
+}
+

+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+ struct camif_priv *priv = video_drvdata(file);
+
+ *index = priv->current_input;

+ return 0;
+}
+

+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct imx_media_subdev *sensor;

+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+

+ if (index >= sensor->input.num)
+ return -EINVAL;
+
+ if (index == priv->current_input)
+ return 0;
+
+ /* select the sensor's input */
+ ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+ sensor->input.value[index], 0, 0);
+ if (!ret)
+ priv->current_input = index;
+

+ return ret;
+}
+

+static int camif_g_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *a)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+

+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)

+ return -EINVAL;
+
+ return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *a)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+

+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)

+ return -EINVAL;
+
+ return v4l2_subdev_call(sensor->sd, video, s_parm, a);
+}
+
+
+static const struct v4l2_ioctl_ops camif_ioctl_ops = {
+ .vidioc_querycap = vidioc_querycap,
+
+ .vidioc_enum_fmt_vid_cap = camif_enum_fmt_vid_cap,
+ .vidioc_g_fmt_vid_cap = camif_g_fmt_vid_cap,
+ .vidioc_try_fmt_vid_cap = camif_try_fmt_vid_cap,
+ .vidioc_s_fmt_vid_cap = camif_s_fmt_vid_cap,
+
+ .vidioc_querystd = camif_querystd,
+ .vidioc_g_std = camif_g_std,
+ .vidioc_s_std = camif_s_std,
+
+ .vidioc_enum_input = camif_enum_input,
+ .vidioc_g_input = camif_g_input,
+ .vidioc_s_input = camif_s_input,
+
+ .vidioc_g_parm = camif_g_parm,
+ .vidioc_s_parm = camif_s_parm,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+ struct v4l2_mbus_framefmt *outfmt;
+ const struct imx_media_pixfmt *outcc;

+
+ outfmt = &priv->format_mbus[priv->output_pad];

+ outcc = priv->cc[priv->output_pad];
+ return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
+}
+
+static int camif_queue_setup(struct vb2_queue *vq,
+ unsigned int *nbuffers, unsigned int *nplanes,
+ unsigned int sizes[], struct device *alloc_devs[])
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ unsigned int count = *nbuffers;
+ u32 sizeimage;
+
+ if (!priv->src_sd)
+ return -EPIPE;
+
+ if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ sizeimage = camif_get_sizeimage(priv);
+
+ count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
+ count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
+
+ while (sizeimage * count > VID_MEM_LIMIT)
+ count--;
+
+ if (count < IMX_MEDIA_MIN_RING_BUFS)
+ return -EINVAL;
+
+ *nplanes = 1;
+ *nbuffers = count;
+ sizes[0] = sizeimage;

+
+ return 0;
+}
+

+static int camif_buf_init(struct vb2_buffer *vb)
+{
+ struct vb2_queue *vq = vb->vb2_queue;
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ struct imx_media_buffer *buf = to_imx_media_vb(vb);
+
+ if (!priv->src_sd)
+ return -EPIPE;
+
+ INIT_LIST_HEAD(&buf->list);

+
+ return 0;
+}
+

+static int camif_buf_prepare(struct vb2_buffer *vb)
+{
+ struct vb2_queue *vq = vb->vb2_queue;
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ u32 sizeimage = camif_get_sizeimage(priv);
+ int ret;
+
+ if (!priv->src_sd)
+ return -EPIPE;
+
+ if (vb2_plane_size(vb, 0) < sizeimage) {
+ v4l2_err(&priv->sd,
+ "data will not fit into plane (%lu < %lu)\n",
+ vb2_plane_size(vb, 0), (long)sizeimage);

+ return -EINVAL;
+ }
+

+ vb2_set_plane_payload(vb, 0, sizeimage);
+
+ if (!priv->in_ring) {
+ priv->in_ring = imx_media_alloc_dma_buf_ring(
+ priv->md, &priv->src_sd->entity, &priv->sd.entity,
+ sizeimage, vq->num_buffers, false);
+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ return ret;
+ }
+ }
+

+ ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
+ if (ret)
+ goto free_ring;

+
+ return 0;
+

+free_ring:
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+ return ret;
+}
+

+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+ struct imx_media_buffer *buf = to_imx_media_vb(vb);

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->q_lock, flags);
+
+ list_add_tail(&buf->list, &priv->ready_q);
+
+ spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ u32 sizeimage = camif_get_sizeimage(priv);
+ struct imx_media_buffer *buf, *tmp;
+ unsigned long flags;
+ int ret;
+
+ if (vb2_is_streaming(vq))
+ return 0;
+
+ if (!priv->src_sd)
+ return -EPIPE;
+
+ if (!priv->in_ring) {
+ priv->in_ring = imx_media_alloc_dma_buf_ring(
+ priv->md, &priv->src_sd->entity, &priv->sd.entity,
+ sizeimage, vq->num_buffers, false);
+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;
+ goto return_bufs;
+ }
+ }
+
+ ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+ &priv->mp, true);
+ if (ret) {
+ v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
+ ret);
+ goto free_ring;
+ }
+
+ priv->stop = false;

+
+ return 0;
+

+free_ring:
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;
+return_bufs:
+ spin_lock_irqsave(&priv->q_lock, flags);
+ list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
+ list_del(&buf->list);
+ vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
+ }
+ spin_unlock_irqrestore(&priv->q_lock, flags);

+ return ret;
+}
+

+static void camif_stop_streaming(struct vb2_queue *vq)
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ struct imx_media_buffer *frame;
+ unsigned long flags;
+ int ret;
+
+ if (!vb2_is_streaming(vq))
+ return;
+
+ spin_lock_irqsave(&priv->q_lock, flags);
+ priv->stop = true;
+ spin_unlock_irqrestore(&priv->q_lock, flags);
+
+ ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+ &priv->mp, false);
+ if (ret)
+ v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
+ ret);
+
+ if (priv->in_ring) {
+ v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;
+ }
+
+ /* release all active buffers */
+ spin_lock_irqsave(&priv->q_lock, flags);
+ while (!list_empty(&priv->ready_q)) {
+ frame = list_entry(priv->ready_q.next,
+ struct imx_media_buffer, list);
+ list_del(&frame->list);
+ vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
+ }
+ spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static struct vb2_ops camif_qops = {
+ .queue_setup = camif_queue_setup,
+ .buf_init = camif_buf_init,
+ .buf_prepare = camif_buf_prepare,
+ .buf_queue = camif_buf_queue,
+ .wait_prepare = vb2_ops_wait_prepare,
+ .wait_finish = vb2_ops_wait_finish,
+ .start_streaming = camif_start_streaming,
+ .stop_streaming = camif_stop_streaming,
+};
+
+/*
+ * File operations
+ */
+static int camif_open(struct file *file)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ int ret;
+
+ if (mutex_lock_interruptible(&priv->mutex))
+ return -ERESTARTSYS;
+
+ ret = v4l2_fh_open(file);
+ if (ret)
+ v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
+
+ mutex_unlock(&priv->mutex);

+ return ret;
+}
+

+static int camif_release(struct file *file)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct vb2_queue *vq = &priv->buffer_queue;

+ int ret = 0;
+

+ mutex_lock(&priv->mutex);
+
+ if (file->private_data == vq->owner) {
+ vb2_queue_release(vq);
+ vq->owner = NULL;
+ }
+
+ v4l2_fh_release(file);
+ mutex_unlock(&priv->mutex);

+ return ret;
+}
+

+static const struct v4l2_file_operations camif_fops = {
+ .owner = THIS_MODULE,
+ .open = camif_open,
+ .release = camif_release,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap,
+};
+
+static struct video_device camif_videodev = {
+ .fops = &camif_fops,
+ .ioctl_ops = &camif_ioctl_ops,
+ .minor = -1,
+ .release = video_device_release,
+ .vfl_dir = VFL_DIR_RX,
+ .tvnorms = V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
+};
+
+/*
+ * Subdev and media entity operations
+ */
+
+static void camif_new_dma_buf(struct camif_priv *priv)
+{
+ struct imx_media_dma_buf *dmabuf;
+ struct imx_media_buffer *buf;
+ enum vb2_buffer_state state;
+ struct vb2_buffer *vb;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->q_lock, flags);
+
+ if (priv->stop || list_empty(&priv->ready_q))
+ goto unlock;
+
+ dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
+ if (!dmabuf)
+ goto unlock;
+
+ vb = dmabuf->vb;
+ buf = to_imx_media_vb(vb);
+ if (list_empty(&buf->list)) {
+ dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
+ vb->index);

+ goto unlock;
+ }
+

+ dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
+ vb->timestamp = ktime_get_ns();
+ list_del_init(&buf->list);
+ state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
+ VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
+ vb2_buffer_done(vb, state);
+unlock:
+ spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct imx_media_dma_buf_ring **ring;
+
+ switch (cmd) {
+ case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+ if (!priv->in_ring)
+ return -EINVAL;
+ ring = (struct imx_media_dma_buf_ring **)arg;
+ *ring = priv->in_ring;
+ break;
+ case IMX_MEDIA_NEW_DMA_BUF:
+ camif_new_dma_buf(priv);
+ break;
+ case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+ /* src indicates sink buffer ring can be freed */
+ if (!priv->in_ring)
+ return 0;
+ v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int camif_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct video_device *vfd = &priv->vfd;
+ struct v4l2_subdev *remote_sd;

+ int ret = 0;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+ local->entity->name);
+

+ if (is_media_entity_v4l2_video_device(remote->entity))
+ return 0;
+
+ WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);

+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ /* reset controls to refresh with inherited from subdevs */
+ v4l2_ctrl_handler_free(vfd->ctrl_handler);
+ v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);

+
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ ret = imx_media_inherit_controls(priv->md, vfd,
+ &priv->src_sd->entity);

+ } else {
+ priv->src_sd = NULL;
+ }

+ return ret;
+}
+

+static int camif_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{
+ if (code->pad >= CAMIF_NUM_PADS)
+ return -EINVAL;
+
+ return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= CAMIF_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int camif_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ const struct imx_media_pixfmt *cc;
+ u32 code;
+
+ if (sdformat->pad >= CAMIF_NUM_PADS)
+ return -EINVAL;
+
+ if (vb2_is_busy(&priv->buffer_queue)) {
+ v4l2_err(&priv->sd, "%s queue busy\n", __func__);

+ return -EBUSY;
+ }
+

+ cc = imx_media_find_format(0, sdformat->format.code, true, true);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, true);
+ cc = imx_media_find_format(0, code, true, true);
+ sdformat->format.code = cc->codes[0];
+ }
+
+ /* Output pad mirrors input pad, no limitations on input pads */

+ if (sdformat->pad == priv->output_pad)

+ sdformat->format = priv->format_mbus[priv->input_pad];

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;

+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int camif_init_pads(struct camif_priv *priv)
+{
+ struct video_device *vfd = &priv->vfd;

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);

+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {

+ v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+ imxsd->num_sink_pads, imxsd->num_src_pads);

+ return -EINVAL;
+ }
+

+ priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
+ if (ret) {
+ v4l2_err(&priv->sd, "failed to init device node pad\n");

+ return ret;
+ }
+

+ for (i = 0; i < CAMIF_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];
+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;
+
+ /* set a default mbus format */
+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);

+ if (ret)
+ return ret;
+ }
+

+ return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
+ priv->pad);
+}
+
+static int camif_registered(struct v4l2_subdev *sd)
+{
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct vb2_queue *vq = &priv->buffer_queue;
+ struct video_device *vfd = &priv->vfd;

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ int ret;

+
+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ vfd->v4l2_dev = sd->v4l2_dev;
+
+ ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+ if (ret) {
+ v4l2_err(sd, "Failed to register video device\n");

+ return ret;
+ }
+

+ vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ vq->io_modes = VB2_MMAP | VB2_DMABUF;
+ vq->drv_priv = priv;
+ vq->buf_struct_size = sizeof(struct imx_media_buffer);
+ vq->ops = &camif_qops;
+ vq->mem_ops = &vb2_dma_contig_memops;
+ vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+ vq->lock = &priv->mutex;
+ vq->min_buffers_needed = 2;
+ vq->dev = priv->dev;
+
+ ret = vb2_queue_init(vq);
+ if (ret) {
+ v4l2_err(sd, "vb2_queue_init failed\n");
+ goto unreg;
+ }
+
+ INIT_LIST_HEAD(&priv->ready_q);
+
+ ret = camif_init_pads(priv);
+ if (ret) {
+ v4l2_err(sd, "camif_init_pads failed\n");
+ goto unreg;
+ }
+
+ /* create the link to our device node */
+ ret = media_create_pad_link(&sd->entity, priv->output_pad,
+ &vfd->entity, 0,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (ret) {
+ v4l2_err(sd, "failed to create link to device node\n");
+ goto unreg;
+ }
+
+ /* setup default pad formats */

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
+ &priv->cc[priv->output_pad]);
+ if (ret)
+ goto unreg;
+
+ *infmt = *outfmt;
+ priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
+
+ priv->current_std = V4L2_STD_UNKNOWN;
+
+ v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
+ video_device_node_name(vfd));
+
+ vfd->ctrl_handler = &priv->ctrl_hdlr;
+
+ return 0;
+unreg:
+ video_unregister_device(vfd);

+ return ret;
+}
+

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+ .enum_mbus_code = camif_enum_mbus_code,
+ .get_fmt = camif_get_fmt,
+ .set_fmt = camif_set_fmt,
+};
+
+static struct v4l2_subdev_ops camif_subdev_ops = {
+ .pad = &camif_pad_ops,
+ .core = &camif_core_ops,
+};
+
+static int camif_probe(struct platform_device *pdev)
+{
+ struct imx_media_internal_sd_platformdata *pdata;
+ struct camif_priv *priv;
+ struct video_device *vfd;

+ int ret;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+

+ platform_set_drvdata(pdev, priv);

+ priv->dev = &pdev->dev;
+

+ pdata = priv->dev->platform_data;
+

+ mutex_init(&priv->mutex);
+ spin_lock_init(&priv->q_lock);
+
+ v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
+ v4l2_set_subdevdata(&priv->sd, priv);
+ priv->sd.internal_ops = &camif_internal_ops;
+ priv->sd.entity.ops = &camif_entity_ops;

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ /* get our group id and camif id */

+ priv->sd.grp_id = pdata->grp_id;

+ priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));

+ snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+ "%s devnode", pdata->sd_name);
+

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+

+ vfd = &priv->vfd;
+ *vfd = camif_videodev;
+ vfd->lock = &priv->mutex;
+ vfd->queue = &priv->buffer_queue;
+
+ video_set_drvdata(vfd, priv);

+
+ v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+

+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)
+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ return ret;
+}
+

+static int camif_remove(struct platform_device *pdev)
+{
+ struct camif_priv *priv =
+ (struct camif_priv *)platform_get_drvdata(pdev);
+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(&priv->sd);

+
+ return 0;
+}
+

+static const struct platform_device_id camif_ids[] = {
+ { .name = DEVICE_NAME },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, camif_ids);
+
+static struct platform_driver imx_camif_driver = {
+ .probe = camif_probe,
+ .remove = camif_remove,
+ .driver = {
+ .name = DEVICE_NAME,

+ .owner = THIS_MODULE,
+ },
+};
+

+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Until the OV5652 sensor module compatible with the SabreSD becomes
available for testing, the ov5642 node is currently disabled.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6dl-sabresd.dts | 5 ++
arch/arm/boot/dts/imx6q-sabresd.dts | 5 ++
arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 ++++++++++++++++++++++++++++++++-
3 files changed, 123 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl-sabresd.dts b/arch/arm/boot/dts/imx6dl-sabresd.dts
index 1e45f2f..6cf7a50 100644
--- a/arch/arm/boot/dts/imx6dl-sabresd.dts
+++ b/arch/arm/boot/dts/imx6dl-sabresd.dts
@@ -15,3 +15,8 @@
model = "Freescale i.MX6 DualLite SABRE Smart Device Board";
compatible = "fsl,imx6dl-sabresd", "fsl,imx6dl";
};
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6q-sabresd.dts b/arch/arm/boot/dts/imx6q-sabresd.dts
index 9cbdfe7..8c1d7ad 100644
--- a/arch/arm/boot/dts/imx6q-sabresd.dts
+++ b/arch/arm/boot/dts/imx6q-sabresd.dts
@@ -23,3 +23,8 @@
&sata {
status = "okay";
};
+
+&ipu1_csi1_from_mipi_vc1 {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
index 55ef535..39b4228 100644
--- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
@@ -10,6 +10,7 @@
* http://www.gnu.org/copyleft/gpl.html
*/

+#include <dt-bindings/clock/imx6qdl-clock.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/input/input.h>

@@ -146,6 +147,33 @@

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;

+ data-shift = <12>; /* Lines 19:12 used */
+ hsync-active = <1>;
+ vsync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
+&mipi_csi {

+ status = "okay";
+};

+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+ remote-endpoint = <&ov5640_to_mipi_csi>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
+
&audmux {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_audmux>;
@@ -214,7 +242,33 @@
0x8014 /* 4:FN_DMICCDAT */
0x0000 /* 5:Default */
>;
- };
+ };
+
+ camera: ov5642@3c {
+ compatible = "ovti,ov5642";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5642>;
+ clocks = <&clks IMX6QDL_CLK_CKO>;
+ clock-names = "xclk";
+ reg = <0x3c>;
+ xclk = <24000000>;
+ DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+ AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
+ rev B board is VGEN5 */
+ DVDD-supply = <&vgen2_reg>; /* 1.5v*/
+ pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>; /* SD1_DAT0 */
+ reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>; /* SD1_DAT1 */
+ status = "disabled";
+
+ port {
+ ov5642_to_ipu1_csi0_mux: endpoint {

+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;

+ hsync-active = <1>;
+ vsync-active = <1>;
+ };
+ };
+ };
};

&i2c2 {
@@ -322,6 +376,34 @@
};
};
};
+
+ mipi_camera: ov5640@3c {
+ compatible = "ovti,ov5640_mipi";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5640>;
+ reg = <0x3c>;
+ clocks = <&clks IMX6QDL_CLK_CKO>;
+ clock-names = "xclk";
+ xclk = <24000000>;
+ DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+ AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
+ rev B board is VGEN5 */
+ DVDD-supply = <&vgen2_reg>; /* 1.5v*/
+ pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>; /* SD1_DAT2 */
+ reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>; /* SD1_CLK */
+
+ port {

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ ov5640_to_mipi_csi: endpoint@1 {
+ reg = <1>;
+ remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+ };
+ };
+ };
};

&i2c3 {
@@ -426,6 +508,36 @@
>;
};

+ pinctrl_ov5640: ov5640grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x80000000
+ MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x80000000
+ >;
+ };
+
+ pinctrl_ov5642: ov5642grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x80000000
+ MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x80000000
+ >;
+ };
+

+ pinctrl_ipu1_csi0: ipu1grp-csi0 {
+ fsl,pins = <

+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x80000000
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x80000000
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x80000000
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x80000000
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x80000000
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x80000000
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x80000000
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x80000000
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x80000000
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x80000000
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x80000000
+ >;
+ };
+

pinctrl_pcie: pciegrp {
fsl,pins = <
MX6QDL_PAD_GPIO_17__GPIO7_IO12 0x1b0b0
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:06 PM12/29/16

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

Both hang off the same i2c2 bus, so they require different (and non-
default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Note there is a pin conflict with GPIO6. This pin functions as a power
input pin to the OV5642, but ENET uses it as the h/w workaround for
erratum ERR006687, to wake-up the ARM cores on normal RX and TX packet
done events (see 6261c4c8). So workaround 6261c4c8 is reverted here to
support the OV5642, and the "fsl,err006687-workaround-present" boolean
also must be removed. The result is that the CPUidle driver will no longer
allow entering the deep idle states on the sabrelite.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6dl-sabrelite.dts | 5 ++
arch/arm/boot/dts/imx6q-sabrelite.dts | 6 ++
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 122 ++++++++++++++++++++++++++++++-
3 files changed, 129 insertions(+), 4 deletions(-)

diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
index 0f06ca5..fec2524 100644
--- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
@@ -48,3 +48,8 @@
model = "Freescale i.MX6 DualLite SABRE Lite Board";
compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";

};
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

diff --git a/arch/arm/boot/dts/imx6q-sabrelite.dts b/arch/arm/boot/dts/imx6q-sabrelite.dts
index 66d10d8..9e2d26d 100644
--- a/arch/arm/boot/dts/imx6q-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6q-sabrelite.dts
@@ -52,3 +52,9 @@

&sata {
status = "okay";
};
+
+&ipu1_csi1_from_mipi_vc1 {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

+
diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 1f9076e..4a50bb1 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -39,6 +39,8 @@
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
+

+#include <dt-bindings/clock/imx6qdl-clock.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/input/input.h>

@@ -96,6 +98,15 @@
};
};

+ mipi_xclk: mipi_xclk {
+ compatible = "pwm-clock";
+ #clock-cells = <0>;
+ clock-frequency = <22000000>;
+ clock-output-names = "mipi_pwm3";
+ pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */

+ status = "okay";
+ };
+

gpio-keys {
compatible = "gpio-keys";
pinctrl-names = "default";
@@ -220,6 +231,22 @@

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;
+ data-shift = <12>; /* Lines 19:12 used */
+ hsync-active = <1>;

+ vync-active = <1>;

+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+

&audmux {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_audmux>;

@@ -271,9 +298,6 @@
txd1-skew-ps = <0>;
txd2-skew-ps = <0>;
txd3-skew-ps = <0>;
- interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
- <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;
- fsl,err006687-workaround-present;
status = "okay";
};

@@ -302,6 +326,52 @@
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_i2c2>;
status = "okay";
+
+ camera: ov5642@42 {

+ compatible = "ovti,ov5642";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5642>;

+ clocks = <&clks IMX6QDL_CLK_CKO2>;
+ clock-names = "xclk";
+ reg = <0x42>;
+ xclk = <24000000>;
+ reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+ pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
+ gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;

+
+ port {
+ ov5642_to_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;
+ hsync-active = <1>;
+ vsync-active = <1>;
+ };
+ };
+ };

+
+ mipi_camera: ov5640@40 {

+ compatible = "ovti,ov5640_mipi";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5640>;

+ clocks = <&mipi_xclk>;
+ clock-names = "xclk";
+ reg = <0x40>;
+ xclk = <22000000>;
+ reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+ pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */

+
+ port {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ ov5640_to_mipi_csi: endpoint@1 {
+ reg = <1>;
+ remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+ };
+ };
+ };
};

&i2c3 {

@@ -374,7 +444,6 @@
MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL 0x1b030
/* Phy reset */
MX6QDL_PAD_EIM_D23__GPIO3_IO23 0x000b0
- MX6QDL_PAD_GPIO_6__ENET_IRQ 0x000b1
>;
};

@@ -449,6 +518,39 @@
>;
};

+ pinctrl_ov5642: ov5642grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x80000000

+ MX6QDL_PAD_GPIO_6__GPIO1_IO06 0x80000000
+ MX6QDL_PAD_GPIO_8__GPIO1_IO08 0x80000000
+ MX6QDL_PAD_GPIO_3__CCM_CLKO2 0x80000000

+ >;
+ };
+
+ pinctrl_ipu1_csi0: ipu1grp-csi0 {
+ fsl,pins = <
+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x80000000
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x80000000
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x80000000
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x80000000
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x80000000
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x80000000
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x80000000
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x80000000
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x80000000
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x80000000
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x80000000

+ MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x80000000
+ >;
+ };
+

+ pinctrl_ov5640: ov5640grp {
+ fsl,pins = <

+ MX6QDL_PAD_NANDF_D5__GPIO2_IO05 0x000b0
+ MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
+ >;
+ };
+
pinctrl_pwm1: pwm1grp {
fsl,pins = <
MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
@@ -605,3 +707,15 @@
vmmc-supply = <&reg_3p3v>;
status = "okay";

};
+
+&mipi_csi {
+ status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+ remote-endpoint = <&ov5640_to_mipi_csi>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
+

--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:07 PM12/29/16

This is a set of three media entity subdevice drivers for the i.MX
Image Converter. The i.MX IC module contains three independent
"tasks":

- Pre-processing Encode task: video frames are routed directly from
the CSI and can be scaled, color-space converted, and rotated.
Scaled output is limited to 1024x1024 resolution. Output frames
can be routed to various sinks including the post-processing subdev
task described below.

- Pre-processing Viewfinder task: this task can perform the same
conversions as the pre-process encode task, but in addition can
be used for hardware motion compensated deinterlacing. Frames can
come either directly from the CSIs or from memory via IDMAC channels.
Scaled output is limited to 1024x1024 resolution. Output frames can
be routed to various sinks including the post-processing subdev task.

- Post-processing task: same conversions as pre-process encode. However
this subdevice driver sends frames to the i.MX IPU image converter
which supports image tiling, which allows scaled output up to
4096x4096 resolution. Output frames can be routed to the camera
interface driver or a mem2mem driver.
---
drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/imx-ic-common.c | 113 +++
drivers/staging/media/imx/imx-ic-pp.c | 636 ++++++++++++++++
drivers/staging/media/imx/imx-ic-prpenc.c | 1037 +++++++++++++++++++++++++
drivers/staging/media/imx/imx-ic-prpvf.c | 1181 +++++++++++++++++++++++++++++
drivers/staging/media/imx/imx-ic.h | 36 +
6 files changed, 3005 insertions(+)

create mode 100644 drivers/staging/media/imx/imx-ic-common.c
create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
create mode 100644 drivers/staging/media/imx/imx-ic.h

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 3559d7b..d2a962c 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -1,8 +1,10 @@

imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \

imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o

obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
new file mode 100644
index 0000000..1b40558
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-common.c
@@ -0,0 +1,113 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>

+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+ [IC_TASK_ENCODER] = &imx_ic_prpenc_ops,
+ [IC_TASK_VIEWFINDER] = &imx_ic_prpvf_ops,
+ [IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+
+static int imx_ic_probe(struct platform_device *pdev)

+{
+ struct imx_media_internal_sd_platformdata *pdata;

+ struct imx_ic_priv *priv;

+ int ret;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+

+ platform_set_drvdata(pdev, &priv->sd);

+ priv->dev = &pdev->dev;
+

+ /* get our ipu_id, grp_id and IC task id */

+ pdata = priv->dev->platform_data;

+ priv->ipu_id = pdata->ipu_id;

+ switch (pdata->grp_id) {
+ case IMX_MEDIA_GRP_ID_IC_PRPENC:
+ priv->task_id = IC_TASK_ENCODER;

+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPVF:

+ priv->task_id = IC_TASK_VIEWFINDER;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+ priv->task_id = IC_TASK_POST_PROCESSOR;

+ break;
+ default:
+ return -EINVAL;
+ }
+

+ v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
+ v4l2_set_subdevdata(&priv->sd, priv);
+ priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
+ priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ priv->sd.grp_id = pdata->grp_id;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+

+ ret = ic_ops[priv->task_id]->init(priv);

+ if (ret)
+ return ret;
+

+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ goto remove;
+
+ return 0;
+remove:
+ ic_ops[priv->task_id]->remove(priv);

+ return ret;
+}
+

+static int imx_ic_remove(struct platform_device *pdev)
+{

+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+
+ ic_ops[priv->task_id]->remove(priv);
+

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(sd);

+
+ return 0;
+}
+

+static const struct platform_device_id imx_ic_ids[] = {
+ { .name = "imx-ipuv3-ic" },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, imx_ic_ids);
+
+static struct platform_driver imx_ic_driver = {
+ .probe = imx_ic_probe,
+ .remove = imx_ic_remove,
+ .id_table = imx_ic_ids,
+ .driver = {
+ .name = "imx-ipuv3-ic",

+ .owner = THIS_MODULE,
+ },
+};

+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-ic");
diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
new file mode 100644
index 0000000..bc93696
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-pp.c
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/timer.h>
+#include <linux/sched.h>
+#include <linux/slab.h>

+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+#include <media/imx.h>
+#include <video/imx-ipu-image-convert.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {
+ struct imx_media_dev *md;
+ struct imx_ic_priv *ic_priv;
+ int pp_id;
+
+ struct ipu_soc *ipu;
+ struct ipu_image_convert_ctx *ic_ctx;
+
+ struct media_pad pad[PP_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ /* our dma buffer sink ring */
+ struct imx_media_dma_buf_ring *in_ring;
+ /* the dma buffer ring we send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct ipu_image_convert_run *out_run;
+
+ struct imx_media_dma_buf *inbuf; /* last input buffer */
+
+ bool stream_on; /* streaming is on */

+ bool stop; /* streaming is stopping */

+ spinlock_t irqlock;
+
+ struct v4l2_subdev *src_sd;
+ struct v4l2_subdev *sink_sd;
+
+ struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
+ const struct imx_media_pixfmt *cc[PP_NUM_PADS];
+
+ /* motion select control */
+ struct v4l2_ctrl_handler ctrl_hdlr;
+ int rotation; /* degrees */
+ bool hflip;
+ bool vflip;
+
+ /* derived from rotation, hflip, vflip controls */
+ enum ipu_rotate_mode rot_mode;
+};
+
+static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+ return ic_priv->task_priv;
+}
+
+static void pp_convert_complete(struct ipu_image_convert_run *run,
+ void *data)
+{
+ struct pp_priv *priv = data;
+ struct imx_media_dma_buf *done;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);

+
+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))

+ imx_media_dma_buf_done(done, run->status ?
+ IMX_MEDIA_BUF_STATUS_ERROR :
+ IMX_MEDIA_BUF_STATUS_DONE);
+
+ /* we're done with the inbuf, queue it back */
+ imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
+

+ spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+

+static void pp_queue_conversion(struct pp_priv *priv,
+ struct imx_media_dma_buf *inbuf)
+{
+ struct ipu_image_convert_run *run;
+ struct imx_media_dma_buf *outbuf;
+
+ /* get next queued buffer and make it active */
+ outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(outbuf);
+ priv->inbuf = inbuf;
+
+ run = &priv->out_run[outbuf->index];
+ run->ctx = priv->ic_ctx;
+ run->in_phys = inbuf->phys;
+ run->out_phys = outbuf->phys;
+ ipu_image_convert_queue(run);
+}
+
+static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct pp_priv *priv = sd_to_priv(sd);
+ struct imx_media_dma_buf_ring **ring;
+ struct imx_media_dma_buf *buf;

+ unsigned long flags;
+

+ switch (cmd) {
+ case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:

+ /* src asks for a buffer ring */
+ if (!priv->in_ring)

+ return -EINVAL;
+ ring = (struct imx_media_dma_buf_ring **)arg;
+ *ring = priv->in_ring;
+ break;
+ case IMX_MEDIA_NEW_DMA_BUF:

+ /* src hands us a new buffer */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ if (!priv->stop &&
+ !imx_media_dma_buf_get_active(priv->out_ring)) {
+ buf = imx_media_dma_buf_dequeue(priv->in_ring);
+ if (buf)
+ pp_queue_conversion(priv, buf);
+ }
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+ break;
+ case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+ /* src indicates sink buffer ring can be freed */
+ if (!priv->in_ring)
+ return 0;
+ v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int pp_start(struct pp_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ struct ipu_image image_in, image_out;
+ const struct imx_media_pixfmt *incc;
+ struct v4l2_mbus_framefmt *infmt;
+ int i, in_size, ret;

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ return ret;
+

+ imx_media_mbus_fmt_to_ipu_image(&image_in,

+ &priv->format_mbus[priv->input_pad]);

+ imx_media_mbus_fmt_to_ipu_image(&image_out,
+ &priv->format_mbus[priv->output_pad]);
+
+ priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+ priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
+ IC_TASK_POST_PROCESSOR,
+ &image_in, &image_out,
+ priv->rot_mode,
+ pp_convert_complete, priv);
+ if (IS_ERR(priv->ic_ctx))
+ return PTR_ERR(priv->ic_ctx);
+

+ infmt = &priv->format_mbus[priv->input_pad];

+ incc = priv->cc[priv->input_pad];

+ in_size = (infmt->width * incc->bpp * infmt->height) >> 3;

+
+ if (priv->in_ring) {

+ v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ }

+
+ priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
+ &priv->src_sd->entity,
+ &ic_priv->sd.entity,
+ in_size,
+ IMX_MEDIA_MIN_RING_BUFS,
+ true);
+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&ic_priv->sd,
+ "failed to alloc dma-buf ring\n");

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ goto out_unprep;
+ }
+
+ for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+ imx_media_dma_buf_queue(priv->in_ring, i);
+
+ priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+ sizeof(*priv->out_run), GFP_KERNEL);
+ if (!priv->out_run) {
+ v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+ ret = -ENOMEM;
+ goto out_free_ring;

+ }
+
+ priv->stop = false;
+
+ return 0;
+

+out_free_ring:

+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+out_unprep:
+ ipu_image_convert_unprepare(priv->ic_ctx);

+ return ret;
+}
+

+static void pp_stop(struct pp_priv *priv)
+{

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);

+ priv->stop = true;

+ spin_unlock_irqrestore(&priv->irqlock, flags);
+
+ ipu_image_convert_unprepare(priv->ic_ctx);
+ kfree(priv->out_run);

+
+ priv->out_ring = NULL;
+

+ /* inform sink that its sink buffer ring can now be freed */

+ v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+

+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct pp_priv *priv = sd_to_priv(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = pp_start(priv);

+ else if (!enable && priv->stream_on)

+ pp_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int pp_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ const struct imx_media_pixfmt *cc;

+ u32 fourcc;
+ int ret;
+
+ if (code->pad >= PP_NUM_PADS)
+ return -EINVAL;
+
+ ret = ipu_image_convert_enum_format(code->index, &fourcc);

+ if (ret)
+ return ret;
+

+ /* convert returned fourcc to mbus code */
+ cc = imx_media_find_format(fourcc, 0, true, true);
+ if (WARN_ON(!cc))
+ return -EINVAL;
+
+ code->code = cc->codes[0];

+ return 0;
+}
+

+static int pp_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct pp_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PP_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int pp_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct pp_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ struct ipu_image test_in, test_out;
+ u32 code;
+
+ if (sdformat->pad >= PP_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (sdformat->pad == priv->output_pad) {

+ imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
+ imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+ ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+ imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
+ } else {
+ imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
+ imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+ ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+ imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
+ }

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ if (sdformat->pad == priv->output_pad) {

+ *outfmt = sdformat->format;
+ imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+ } else {
+ *infmt = sdformat->format;
+ imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+ }

+ priv->cc[sdformat->pad] = cc;
+ }
+
+ return 0;
+}
+

+static int pp_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct pp_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ }

+ } else {

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ } else {
+ priv->src_sd = NULL;
+ }
+ }
+

+ return 0;
+}
+

+static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct pp_priv *priv = container_of(ctrl->handler,
+ struct pp_priv, ctrl_hdlr);
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ enum ipu_rotate_mode rot_mode;
+ bool hflip, vflip;
+ int rotation, ret;
+
+ rotation = priv->rotation;
+ hflip = priv->hflip;
+ vflip = priv->vflip;
+
+ switch (ctrl->id) {
+ case V4L2_CID_HFLIP:
+ hflip = (ctrl->val == 1);
+ break;
+ case V4L2_CID_VFLIP:
+ vflip = (ctrl->val == 1);
+ break;
+ case V4L2_CID_ROTATE:
+ rotation = ctrl->val;
+ break;
+ default:
+ v4l2_err(&ic_priv->sd, "Invalid control\n");

+ return -EINVAL;
+ }
+

+ ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);

+ if (ret)
+ return ret;
+

+ if (rot_mode != priv->rot_mode) {

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ struct ipu_image test_in, test_out;
+
+ /* can't change rotation mid-streaming */
+ if (priv->stream_on)
+ return -EBUSY;
+
+ /*
+ * make sure this rotation will work with current input/output
+ * formats before setting
+ */

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+ imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+ ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);

+ if (ret)
+ return ret;
+

+ priv->rot_mode = rot_mode;
+ priv->rotation = rotation;
+ priv->hflip = hflip;
+ priv->vflip = vflip;

+ }
+
+ return 0;
+}
+

+static const struct v4l2_ctrl_ops pp_ctrl_ops = {
+ .s_ctrl = pp_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config pp_std_ctrl[] = {
+ {
+ .id = V4L2_CID_HFLIP,
+ .name = "Horizontal Flip",
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ .def = 0,
+ .min = 0,
+ .max = 1,
+ .step = 1,
+ }, {
+ .id = V4L2_CID_VFLIP,
+ .name = "Vertical Flip",
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ .def = 0,
+ .min = 0,
+ .max = 1,
+ .step = 1,
+ }, {
+ .id = V4L2_CID_ROTATE,
+ .name = "Rotation",
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .def = 0,
+ .min = 0,
+ .max = 270,
+ .step = 90,
+ },
+};
+
+#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
+
+static int pp_init_controls(struct pp_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+ const struct v4l2_ctrl_config *c;

+ int i, ret;
+

+ v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
+
+ for (i = 0; i < PP_NUM_CONTROLS; i++) {
+ c = &pp_std_ctrl[i];
+ v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
+ c->id, c->min, c->max, c->step, c->def);
+ }
+
+ ic_priv->sd.ctrl_handler = hdlr;
+
+ if (hdlr->error) {
+ ret = hdlr->error;
+ v4l2_ctrl_handler_free(hdlr);

+ return ret;
+ }
+

+ v4l2_ctrl_handler_setup(hdlr);

+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int pp_registered(struct v4l2_subdev *sd)
+{
+ struct pp_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < PP_NUM_PADS; i++) {

+ ret = pp_init_controls(priv);

+ if (ret)
+ return ret;
+

+ ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);

+ if (ret)
+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ return ret;
+}
+

+static struct v4l2_subdev_pad_ops pp_pad_ops = {
+ .enum_mbus_code = pp_enum_mbus_code,
+ .get_fmt = pp_get_fmt,
+ .set_fmt = pp_set_fmt,
+};
+
+static struct v4l2_subdev_video_ops pp_video_ops = {
+ .s_stream = pp_s_stream,
+};
+
+static struct v4l2_subdev_core_ops pp_core_ops = {
+ .ioctl = pp_ioctl,
+};
+
+static struct media_entity_operations pp_entity_ops = {
+ .link_setup = pp_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+ if (!priv)
+ return -ENOMEM;
+

+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;
+ spin_lock_init(&priv->irqlock);
+
+ /* get our PP id */
+ priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;

+
+ return 0;
+}
+

+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+ struct pp_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

+
+struct imx_ic_ops imx_ic_pp_ops = {
+ .subdev_ops = &pp_subdev_ops,
+ .internal_ops = &pp_internal_ops,
+ .entity_ops = &pp_entity_ops,
+ .init = pp_init,
+ .remove = pp_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
new file mode 100644
index 0000000..36c361e
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpenc.c
@@ -0,0 +1,1037 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.

+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/timer.h>
+#include <linux/sched.h>
+#include <linux/slab.h>

+#include <linux/interrupt.h>

+#include <linux/spinlock.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PRPENC_NUM_PADS 2
+
+#define MAX_W_IC 1024
+#define MAX_H_IC 1024
+#define MAX_W_SINK 4096
+#define MAX_H_SINK 4096
+
+struct prpenc_priv {
+ struct imx_media_dev *md;
+ struct imx_ic_priv *ic_priv;
+
+ /* IPU units we require */
+ struct ipu_soc *ipu;
+ struct ipu_ic *ic_enc;
+
+ struct media_pad pad[PRPENC_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ struct ipuv3_channel *enc_ch;
+ struct ipuv3_channel *enc_rot_in_ch;
+ struct ipuv3_channel *enc_rot_out_ch;

+
+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct imx_media_dma_buf *next;
+
+ int ipu_buf_num; /* ipu double buffer index: 0-1 */
+

+ struct v4l2_subdev *src_sd;
+ struct v4l2_subdev *sink_sd;
+
+ /* the CSI id at link validate */
+ int csi_id;

+
+ /* the attached sensor at stream on */

+ struct imx_media_subdev *sensor;
+

+ struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
+ const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
+
+ struct imx_media_dma_buf rot_buf[2];
+
+ /* controls */
+ struct v4l2_ctrl_handler ctrl_hdlr;
+ int rotation; /* degrees */
+ bool hflip;
+ bool vflip;
+
+ /* derived from rotation, hflip, vflip controls */
+ enum ipu_rotate_mode rot_mode;

+
+ spinlock_t irqlock;
+
+ struct timer_list eof_timeout_timer;
+ int eof_irq;
+ int nfb4eof_irq;
+

+ bool stream_on; /* streaming is on */

+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;

+};
+
+static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+ return ic_priv->task_priv;
+}
+
+static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
+{
+ if (!IS_ERR_OR_NULL(priv->ic_enc))
+ ipu_ic_put(priv->ic_enc);
+ priv->ic_enc = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->enc_ch))
+ ipu_idmac_put(priv->enc_ch);
+ priv->enc_ch = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
+ ipu_idmac_put(priv->enc_rot_in_ch);
+ priv->enc_rot_in_ch = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
+ ipu_idmac_put(priv->enc_rot_out_ch);
+ priv->enc_rot_out_ch = NULL;
+}
+
+static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ int ret;
+
+ priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+ priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
+ if (IS_ERR(priv->ic_enc)) {
+ v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
+ ret = PTR_ERR(priv->ic_enc);

+ goto out;
+ }
+

+ priv->enc_ch = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+ if (IS_ERR(priv->enc_ch)) {
+ v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+ IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+ ret = PTR_ERR(priv->enc_ch);

+ goto out;
+ }
+

+ priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_MEM_ROT_ENC);
+ if (IS_ERR(priv->enc_rot_in_ch)) {
+ v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+ IPUV3_CHANNEL_MEM_ROT_ENC);
+ ret = PTR_ERR(priv->enc_rot_in_ch);

+ goto out;
+ }
+

+ priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_ROT_ENC_MEM);
+ if (IS_ERR(priv->enc_rot_out_ch)) {
+ v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+ IPUV3_CHANNEL_ROT_ENC_MEM);
+ ret = PTR_ERR(priv->enc_rot_out_ch);
+ goto out;

+ }
+
+ return 0;

+out:
+ prpenc_put_ipu_resources(priv);

+ return ret;
+}
+

+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+ struct prpenc_priv *priv = dev_id;

+ struct imx_media_dma_buf *done, *next;

+ struct ipuv3_channel *channel;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);
+

+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;

+ goto unlock;
+ }
+

+ /* inform CSI of this EOF so it can monitor frame intervals */
+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+

+ channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+ priv->enc_rot_out_ch : priv->enc_ch;

+
+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))
+ imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+ /* priv->next buffer is now the active one */
+ imx_media_dma_buf_set_active(priv->next);
+
+ /* bump the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+

+ if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
+ ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);

+
+ /* get next queued buffer */
+ next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+

+ ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+ ipu_idmac_select_buffer(channel, priv->ipu_buf_num);

+
+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ priv->next = next;
+
+unlock:
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+ return IRQ_HANDLED;
+}
+

+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct prpenc_priv *priv = dev_id;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void prpenc_eof_timeout(unsigned long data)
+{
+ struct prpenc_priv *priv = (struct prpenc_priv *)data;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void prpenc_setup_channel(struct prpenc_priv *priv,
+ struct ipuv3_channel *channel,
+ enum ipu_rotate_mode rot_mode,
+ dma_addr_t addr0, dma_addr_t addr1,
+ bool rot_swap_width_height)
+{

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ unsigned int burst_size;
+ struct ipu_image image;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (rot_swap_width_height)
+ swap(outfmt->width, outfmt->height);
+
+ ipu_cpmem_zero(channel);

+
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+

+ image.phys0 = addr0;
+ image.phys1 = addr1;
+ ipu_cpmem_set_image(channel, &image);
+
+ if (channel == priv->enc_rot_in_ch ||
+ channel == priv->enc_rot_out_ch) {
+ burst_size = 8;
+ ipu_cpmem_set_block_mode(channel);
+ } else {

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;
+ }
+

+ ipu_cpmem_set_burstsize(channel, burst_size);
+
+ if (rot_mode)
+ ipu_cpmem_set_rotation(channel, rot_mode);

+
+ if (outfmt->field == V4L2_FIELD_NONE &&
+ (V4L2_FIELD_HAS_BOTH(infmt->field) ||

+ infmt->field == V4L2_FIELD_ALTERNATE) &&
+ channel == priv->enc_ch)
+ ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+
+ ipu_ic_task_idma_init(priv->ic_enc, channel,
+ outfmt->width, outfmt->height,
+ burst_size, rot_mode);
+ ipu_cpmem_set_axi_id(channel, 1);
+
+ ipu_idmac_set_double_buffer(channel, true);
+
+ if (rot_swap_width_height)
+ swap(outfmt->width, outfmt->height);
+}
+
+static int prpenc_setup_rotation(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ struct imx_media_dma_buf *buf0, *buf1;

+ int out_size, ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
+
+ ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
+ return ret;
+ }
+ ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
+ goto free_rot0;
+ }
+
+ ret = ipu_ic_task_init(priv->ic_enc,
+ infmt->width, infmt->height,
+ outfmt->height, outfmt->width,
+ incc->cs, outcc->cs);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+ goto free_rot1;
+ }
+
+ /* init the IC ENC-->MEM IDMAC channel */
+ prpenc_setup_channel(priv, priv->enc_ch,
+ IPU_ROTATE_NONE,
+ priv->rot_buf[0].phys,
+ priv->rot_buf[1].phys,
+ true);
+
+ /* init the MEM-->IC ENC ROT IDMAC channel */
+ prpenc_setup_channel(priv, priv->enc_rot_in_ch,
+ priv->rot_mode,
+ priv->rot_buf[0].phys,
+ priv->rot_buf[1].phys,
+ true);

+
+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+ /* init the destination IC ENC ROT-->MEM IDMAC channel */
+ prpenc_setup_channel(priv, priv->enc_rot_out_ch,
+ IPU_ROTATE_NONE,
+ buf0->phys, buf1->phys,
+ false);
+
+ /* now link IC ENC-->MEM to MEM-->IC ENC ROT */
+ ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
+
+ /* enable the IC */
+ ipu_ic_enable(priv->ic_enc);

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->enc_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_ch, 1);
+ ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);

+
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->enc_ch);
+ ipu_idmac_enable_channel(priv->enc_rot_in_ch);
+ ipu_idmac_enable_channel(priv->enc_rot_out_ch);
+
+ /* and finally enable the IC PRPENC task */
+ ipu_ic_task_enable(priv->ic_enc);

+
+ return 0;
+

+free_rot1:
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);

+ return ret;
+}
+

+static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
+{
+ ipu_ic_task_disable(priv->ic_enc);
+
+ ipu_idmac_disable_channel(priv->enc_ch);
+ ipu_idmac_disable_channel(priv->enc_rot_in_ch);
+ ipu_idmac_disable_channel(priv->enc_rot_out_ch);
+
+ ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
+
+ ipu_ic_disable(priv->ic_enc);
+
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+}
+
+static int prpenc_setup_norotation(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ struct imx_media_dma_buf *buf0, *buf1;

+ int ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ ret = ipu_ic_task_init(priv->ic_enc,
+ infmt->width, infmt->height,
+ outfmt->width, outfmt->height,
+ incc->cs, outcc->cs);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);

+ return ret;
+ }
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+ /* init the IC PRP-->MEM IDMAC channel */
+ prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
+ buf0->phys, buf1->phys,
+ false);
+
+ ipu_cpmem_dump(priv->enc_ch);
+ ipu_ic_dump(priv->ic_enc);
+ ipu_dump(priv->ipu);
+
+ ipu_ic_enable(priv->ic_enc);

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->enc_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_ch, 1);

+
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->enc_ch);
+
+ /* enable the IC ENCODE task */
+ ipu_ic_task_enable(priv->ic_enc);

+
+ return 0;
+}
+

+static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
+{
+ ipu_ic_task_disable(priv->ic_enc);
+ ipu_idmac_disable_channel(priv->enc_ch);
+ ipu_ic_disable(priv->ic_enc);
+}
+
+static int prpenc_start(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");
+ return -EINVAL;
+ }
+
+ ret = prpenc_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ /* set IC to receive from CSI */
+ ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ goto out_put_ipu;
+
+ priv->ipu_buf_num = 0;
+

+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ ret = prpenc_setup_rotation(priv);
+ else
+ ret = prpenc_setup_norotation(priv);
+ if (ret)
+ goto out_put_ipu;

+
+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->enc_ch,
+ IPU_IRQ_NFB4EOF);
+ ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+ prpenc_nfb4eof_interrupt, 0,
+ "imx-ic-prpenc-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_unsetup;
+ }
+

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ priv->eof_irq = ipu_idmac_channel_irq(
+ priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
+ else
+ priv->eof_irq = ipu_idmac_channel_irq(
+ priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
+
+ ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
+ prpenc_eof_interrupt, 0,
+ "imx-ic-prpenc-eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,
+ "Error registering eof irq: %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ /* start the EOF timeout timer */

+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+out_free_nfb4eof_irq:
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ prpenc_unsetup_rotation(priv);
+ else
+ prpenc_unsetup_norotation(priv);
+out_put_ipu:
+ prpenc_put_ipu_resources(priv);

+ return ret;
+}
+

+static void prpenc_stop(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ unsigned long flags;
+ int ret;
+

+ /* mark next EOF interrupt as the last before stream off */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ priv->last_eof = true;
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+
+ /*

+ * and then wait for interrupt handler to mark completion.
+ */
+ ret = wait_for_completion_timeout(
+ &priv->last_eof_comp,

+ msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+ if (ret == 0)

+ v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+ devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+
+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ prpenc_unsetup_rotation(priv);
+ else
+ prpenc_unsetup_norotation(priv);
+
+ prpenc_put_ipu_resources(priv);

+
+ /* cancel the EOF timeout timer */
+ del_timer_sync(&priv->eof_timeout_timer);
+
+ priv->out_ring = NULL;
+
+ /* inform sink that the buffer ring can now be freed */
+ v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+

+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);
+ bool allow_planar;
+
+ if (code->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+
+ allow_planar = (code->pad == priv->output_pad);
+

+ return imx_media_enum_format(&code->code, code->index,

+ true, allow_planar);
+}
+
+static int prpenc_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PRPENC_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int prpenc_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ bool allow_planar;
+ u32 code;
+
+ if (sdformat->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+

+ cc = imx_media_find_format(0, sdformat->format.code,

+ true, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, false);
+ cc = imx_media_find_format(0, code, true, false);

+ sdformat->format.code = cc->codes[0];
+ }
+

+ if (sdformat->pad == priv->output_pad) {

+ sdformat->format.width = min_t(__u32,
+ sdformat->format.width,
+ MAX_W_IC);
+ sdformat->format.height = min_t(__u32,
+ sdformat->format.height,
+ MAX_H_IC);
+

+ if (sdformat->format.field != V4L2_FIELD_NONE)

+ sdformat->format.field = infmt->field;
+

+ /* IC resizer cannot downsize more than 4:1 */
+ if (ipu_rot_mode_is_irt(priv->rot_mode)) {
+ sdformat->format.width = max_t(__u32,
+ sdformat->format.width,
+ infmt->height / 4);
+ sdformat->format.height = max_t(__u32,
+ sdformat->format.height,
+ infmt->width / 4);
+ } else {
+ sdformat->format.width = max_t(__u32,
+ sdformat->format.width,
+ infmt->width / 4);
+ sdformat->format.height = max_t(__u32,
+ sdformat->format.height,
+ infmt->height / 4);
+ }
+ } else {
+ sdformat->format.width = min_t(__u32,
+ sdformat->format.width,
+ MAX_W_SINK);
+ sdformat->format.height = min_t(__u32,
+ sdformat->format.height,
+ MAX_H_SINK);
+ }

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ priv->cc[sdformat->pad] = cc;
+ }
+
+ return 0;
+}
+

+static int prpenc_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpenc_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {

+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+
+ /* this is sink pad */

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ } else {
+ priv->src_sd = NULL;

+ return 0;
+ }
+

+ switch (remote_sd->grp_id) {

+ case IMX_MEDIA_GRP_ID_CSI0:
+ priv->csi_id = 0;
+ break;
+ case IMX_MEDIA_GRP_ID_CSI1:
+ priv->csi_id = 1;

+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int prpenc_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpenc_priv *priv = ic_priv->task_priv;
+ struct v4l2_mbus_config sensor_mbus_cfg;
+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link,
+ source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);

+ priv->sensor = NULL;

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {

+ int vc_num = 0;

+ /* see NOTE in imx-csi.c */

+#if 0
+ vc_num = imx_media_find_mipi_csi2_channel(

+ priv->md, &ic_priv->sd.entity);

+ if (vc_num < 0)
+ return vc_num;
+#endif

+ /* only virtual channel 0 can be sent to IC */
+ if (vc_num != 0)
+ return -EINVAL;
+ } else {
+ /*
+ * only 8-bit pixels can be sent to IC for parallel
+ * busses
+ */
+ if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)

+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct prpenc_priv *priv = container_of(ctrl->handler,
+ struct prpenc_priv, ctrl_hdlr);
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ enum ipu_rotate_mode rot_mode;
+ bool hflip, vflip;
+ int rotation, ret;
+
+ rotation = priv->rotation;
+ hflip = priv->hflip;
+ vflip = priv->vflip;
+
+ switch (ctrl->id) {
+ case V4L2_CID_HFLIP:
+ hflip = (ctrl->val == 1);
+ break;
+ case V4L2_CID_VFLIP:
+ vflip = (ctrl->val == 1);
+ break;
+ case V4L2_CID_ROTATE:
+ rotation = ctrl->val;
+ break;
+ default:
+ v4l2_err(&ic_priv->sd, "Invalid control\n");

+ return -EINVAL;
+ }
+

+ ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);

+ if (ret)
+ return ret;
+

+ if (rot_mode != priv->rot_mode) {
+ /* can't change rotation mid-streaming */
+ if (priv->stream_on)
+ return -EBUSY;
+
+ priv->rot_mode = rot_mode;
+ priv->rotation = rotation;
+ priv->hflip = hflip;
+ priv->vflip = vflip;

+ }
+
+ return 0;
+}
+

+static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
+ .s_ctrl = prpenc_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
+ {
+ .id = V4L2_CID_HFLIP,
+ .name = "Horizontal Flip",
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ .def = 0,
+ .min = 0,
+ .max = 1,
+ .step = 1,
+ }, {
+ .id = V4L2_CID_VFLIP,
+ .name = "Vertical Flip",
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ .def = 0,
+ .min = 0,
+ .max = 1,
+ .step = 1,
+ }, {
+ .id = V4L2_CID_ROTATE,
+ .name = "Rotation",
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .def = 0,
+ .min = 0,
+ .max = 270,
+ .step = 90,
+ },
+};
+
+#define PRPENC_NUM_CONTROLS ARRAY_SIZE(prpenc_std_ctrl)
+
+static int prpenc_init_controls(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+ const struct v4l2_ctrl_config *c;

+ int i, ret;
+

+ v4l2_ctrl_handler_init(hdlr, PRPENC_NUM_CONTROLS);
+
+ for (i = 0; i < PRPENC_NUM_CONTROLS; i++) {
+ c = &prpenc_std_ctrl[i];
+ v4l2_ctrl_new_std(hdlr, &prpenc_ctrl_ops,
+ c->id, c->min, c->max, c->step, c->def);
+ }
+
+ ic_priv->sd.ctrl_handler = hdlr;
+
+ if (hdlr->error) {
+ ret = hdlr->error;
+ goto out_free;
+ }
+
+ v4l2_ctrl_handler_setup(hdlr);
+ return 0;
+
+out_free:
+ v4l2_ctrl_handler_free(hdlr);

+ return ret;
+}
+

+static int prpenc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct prpenc_priv *priv = sd_to_priv(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = prpenc_start(priv);

+ else if (!enable && priv->stream_on)

+ prpenc_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int prpenc_registered(struct v4l2_subdev *sd)
+{
+ struct prpenc_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < PRPENC_NUM_PADS; i++) {

+ ret = prpenc_init_controls(priv);

+ if (ret)
+ return ret;
+

+ ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);

+ if (ret)
+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ return ret;
+}
+

+static struct v4l2_subdev_pad_ops prpenc_pad_ops = {
+ .enum_mbus_code = prpenc_enum_mbus_code,
+ .get_fmt = prpenc_get_fmt,
+ .set_fmt = prpenc_set_fmt,
+ .link_validate = prpenc_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpenc_video_ops = {
+ .s_stream = prpenc_s_stream,
+};
+
+static struct media_entity_operations prpenc_entity_ops = {
+ .link_setup = prpenc_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_ops prpenc_subdev_ops = {
+ .video = &prpenc_video_ops,
+ .pad = &prpenc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpenc_internal_ops = {
+ .registered = prpenc_registered,
+};
+
+static int prpenc_init(struct imx_ic_priv *ic_priv)
+{
+ struct prpenc_priv *priv;
+
+ priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);

+ if (!priv)
+ return -ENOMEM;
+

+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;
+
+ spin_lock_init(&priv->irqlock);

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = prpenc_eof_timeout;

+
+ return 0;
+}
+

+static void prpenc_remove(struct imx_ic_priv *ic_priv)
+{
+ struct prpenc_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

+
+struct imx_ic_ops imx_ic_prpenc_ops = {
+ .subdev_ops = &prpenc_subdev_ops,
+ .internal_ops = &prpenc_internal_ops,
+ .entity_ops = &prpenc_entity_ops,
+ .init = prpenc_init,
+ .remove = prpenc_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpvf.c b/drivers/staging/media/imx/imx-ic-prpvf.c
new file mode 100644
index 0000000..18d0af5
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpvf.c
@@ -0,0 +1,1181 @@
+/*
+ * V4L2 IC Deinterlacer Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+/*
+ * This subdev implements two different video pipelines:
+ *
+ * CSI -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends a single interlaced field F(n-1)
+ * directly to the VDIC (and optionally the following field F(n)
+ * can be sent to memory via IDMAC channel 13). So only two fields
+ * can be processed by the VDIC. This pipeline only works in VDIC's
+ * high motion mode, which only requires a single field for processing.
+ * The other motion modes (low and medium) require three fields, so this
+ * pipeline does not work in those modes. Also, it is not clear how this
+ * pipeline can deal with the various field orders (sequential BT/TB,
+ * interlaced BT/TB).
+ *
+ * CSI -> CH[0-3] -> MEM -> CH8,9,10 -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends raw and full frames to memory buffers
+ * via the SMFC channels 0-3. Fields from these frames are then
+ * transferred to the VDIC via IDMAC channels 8,9,10. The VDIC requires
+ * three fields: previous field F(n-1), current field F(n), and next
+ * field F(n+1), so we need three raw frames in memory: two completed frames
+ * to send F(n-1), F(n), F(n+1) to the VDIC, and a third frame for active
+ * CSI capture while the completed fields are sent through the VDIC->IC for
+ * processing.
+ *
+ * The "direct" CSI->VDIC pipeline requires less memory bus bandwidth
+ * (just 1 channel vs. 5 channels for indirect pipeline), but it can
+ * only be used in high motion mode, and it only processes a single
+ * field (so half the original image resolution is lost).
+ */
+
+struct prpvf_priv;
+
+struct prpvf_pipeline_ops {
+ int (*setup)(struct prpvf_priv *priv);
+ void (*start)(struct prpvf_priv *priv);
+ void (*stop)(struct prpvf_priv *priv);
+ void (*disable)(struct prpvf_priv *priv);
+};
+
+#define PRPVF_NUM_PADS 2
+
+#define MAX_W_IC 1024
+#define MAX_H_IC 1024
+#define MAX_W_VDIC 968
+#define MAX_H_VDIC 2048
+
+struct prpvf_priv {
+ struct imx_media_dev *md;
+ struct imx_ic_priv *ic_priv;
+
+ /* IPU units we require */
+ struct ipu_soc *ipu;
+ struct ipu_ic *ic_vf;
+ struct ipu_vdi *vdi;
+
+ struct media_pad pad[PRPVF_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ struct ipuv3_channel *vdi_in_ch_p; /* F(n-1) transfer channel */
+ struct ipuv3_channel *vdi_in_ch; /* F(n) transfer channel */
+ struct ipuv3_channel *vdi_in_ch_n; /* F(n+1) transfer channel */
+ struct ipuv3_channel *prpvf_out_ch;/* final progressive frame channel */
+
+ /* pipeline operations */
+ struct prpvf_pipeline_ops *ops;
+
+ /* our dma buffer sink ring */
+ struct imx_media_dma_buf_ring *in_ring;

+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+

+ /* ipu buf num for double-buffering (csi-direct path only) */
+ int ipu_buf_num;
+ struct imx_media_dma_buf *next_out_buf;
+
+ /* current and last input buffers indirect path */
+ struct imx_media_dma_buf *curr_in_buf;
+ struct imx_media_dma_buf *last_in_buf;
+
+ /*
+ * translated field type, input line stride, and field size
+ * for indirect path
+ */
+ u32 fieldtype;
+ u32 in_stride;
+ u32 field_size;
+
+ struct v4l2_subdev *src_sd;
+ /* the sink that will receive the progressive out buffers */

+ struct v4l2_subdev *sink_sd;
+

+ /* the attached CSI at stream on */
+ struct v4l2_subdev *csi_sd;

+
+ /* the attached sensor at stream on */

+ struct imx_media_subdev *sensor;
+

+ /* the video standard from sensor at time of streamon */
+ v4l2_std_id std;
+
+ struct v4l2_mbus_framefmt format_mbus[PRPVF_NUM_PADS];
+ const struct imx_media_pixfmt *cc[PRPVF_NUM_PADS];
+
+ bool csi_direct; /* using direct CSI->VDIC->IC pipeline */
+
+ /* motion select control */
+ struct v4l2_ctrl_handler ctrl_hdlr;
+ enum ipu_motion_sel motion;
+

+ struct timer_list eof_timeout_timer;
+

+ int nfb4eof_irq; /* CSI or PRPVF channel NFB4EOF IRQ */
+ int out_eof_irq; /* PRPVF channel EOF IRQ */
+ spinlock_t irqlock;
+

+ bool stream_on; /* streaming is on */

+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;

+};
+
+static inline struct prpvf_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+ return ic_priv->task_priv;
+}
+
+static void prpvf_put_ipu_resources(struct prpvf_priv *priv)
+{
+ if (!IS_ERR_OR_NULL(priv->ic_vf))
+ ipu_ic_put(priv->ic_vf);
+ priv->ic_vf = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->vdi_in_ch_p))
+ ipu_idmac_put(priv->vdi_in_ch_p);
+ priv->vdi_in_ch_p = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->vdi_in_ch))
+ ipu_idmac_put(priv->vdi_in_ch);
+ priv->vdi_in_ch = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->vdi_in_ch_n))
+ ipu_idmac_put(priv->vdi_in_ch_n);
+ priv->vdi_in_ch_n = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->prpvf_out_ch))
+ ipu_idmac_put(priv->prpvf_out_ch);
+ priv->prpvf_out_ch = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->vdi))
+ ipu_vdi_put(priv->vdi);
+ priv->vdi = NULL;
+}
+
+static int prpvf_get_ipu_resources(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ int ret, err_chan;
+
+ priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+ priv->ic_vf = ipu_ic_get(priv->ipu, IC_TASK_VIEWFINDER);
+ if (IS_ERR(priv->ic_vf)) {
+ v4l2_err(&ic_priv->sd, "failed to get IC VF\n");
+ ret = PTR_ERR(priv->ic_vf);

+ goto out;
+ }
+

+ priv->vdi = ipu_vdi_get(priv->ipu);
+ if (IS_ERR(priv->vdi)) {
+ v4l2_err(&ic_priv->sd, "failed to get VDIC\n");
+ ret = PTR_ERR(priv->vdi);

+ goto out;
+ }
+

+ priv->prpvf_out_ch = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_IC_PRP_VF_MEM);
+ if (IS_ERR(priv->prpvf_out_ch)) {
+ err_chan = IPUV3_CHANNEL_IC_PRP_VF_MEM;
+ ret = PTR_ERR(priv->prpvf_out_ch);
+ goto out_err_chan;
+ }
+
+ if (!priv->csi_direct) {
+ priv->vdi_in_ch_p = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_MEM_VDI_PREV);
+ if (IS_ERR(priv->vdi_in_ch_p)) {
+ err_chan = IPUV3_CHANNEL_MEM_VDI_PREV;
+ ret = PTR_ERR(priv->vdi_in_ch_p);
+ goto out_err_chan;
+ }
+
+ priv->vdi_in_ch = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_MEM_VDI_CUR);
+ if (IS_ERR(priv->vdi_in_ch)) {
+ err_chan = IPUV3_CHANNEL_MEM_VDI_CUR;
+ ret = PTR_ERR(priv->vdi_in_ch);
+ goto out_err_chan;
+ }
+
+ priv->vdi_in_ch_n = ipu_idmac_get(priv->ipu,
+ IPUV3_CHANNEL_MEM_VDI_NEXT);
+ if (IS_ERR(priv->vdi_in_ch_n)) {
+ err_chan = IPUV3_CHANNEL_MEM_VDI_NEXT;
+ ret = PTR_ERR(priv->vdi_in_ch_n);
+ goto out_err_chan;
+ }

+ }
+
+ return 0;
+

+out_err_chan:
+ v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n", err_chan);
+out:
+ prpvf_put_ipu_resources(priv);

+ return ret;
+}
+

+static void prepare_vdi_in_buffers(struct prpvf_priv *priv,
+ struct imx_media_dma_buf *curr)
+{
+ dma_addr_t prev_phys, curr_phys, next_phys;
+ struct imx_media_dma_buf *last;
+
+ last = priv->last_in_buf ? priv->last_in_buf : curr;
+ priv->curr_in_buf = curr;
+
+ switch (priv->fieldtype) {
+ case V4L2_FIELD_SEQ_TB:
+ prev_phys = last->phys;
+ curr_phys = curr->phys + priv->field_size;
+ next_phys = curr->phys;
+ break;
+ case V4L2_FIELD_SEQ_BT:
+ prev_phys = last->phys + priv->field_size;
+ curr_phys = curr->phys;
+ next_phys = curr->phys + priv->field_size;
+ break;
+ case V4L2_FIELD_INTERLACED_BT:
+ prev_phys = last->phys + priv->in_stride;
+ curr_phys = curr->phys;
+ next_phys = curr->phys + priv->in_stride;
+ break;
+ default:
+ /* assume V4L2_FIELD_INTERLACED_TB */
+ prev_phys = last->phys;
+ curr_phys = curr->phys + priv->in_stride;
+ next_phys = curr->phys;
+ break;
+ }
+
+ ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
+ ipu_cpmem_set_buffer(priv->vdi_in_ch, 0, curr_phys);
+ ipu_cpmem_set_buffer(priv->vdi_in_ch_n, 0, next_phys);
+
+ ipu_idmac_select_buffer(priv->vdi_in_ch_p, 0);
+ ipu_idmac_select_buffer(priv->vdi_in_ch, 0);
+ ipu_idmac_select_buffer(priv->vdi_in_ch_n, 0);
+}
+
+static void prepare_prpvf_out_buffer(struct prpvf_priv *priv)
+{
+ struct imx_media_dma_buf *buf;
+
+ /* get next buffer to prepare */
+ buf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ if (!priv->csi_direct) {
+ /*
+ * indirect does not use double-buffering, so this
+ * buffer is now the active one
+ */
+ imx_media_dma_buf_set_active(buf);
+ } else {
+ priv->next_out_buf = buf;
+ }
+
+ ipu_cpmem_set_buffer(priv->prpvf_out_ch, priv->ipu_buf_num, buf->phys);
+ ipu_idmac_select_buffer(priv->prpvf_out_ch, priv->ipu_buf_num);
+}
+
+/* prpvf_out_ch EOF interrupt (progressive frame ready) */
+static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
+{
+ struct prpvf_priv *priv = dev_id;
+ struct imx_media_dma_buf *done;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);
+

+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;

+ goto unlock;
+ }
+
+

+ if (priv->csi_direct) {

+ /* inform CSI of this EOF so it can monitor frame intervals */

+ /* FIXME: frames are comming in twice as fast in direct path! */

+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+ }
+
+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))
+ imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+

+ if (!priv->csi_direct) {
+ /* we're done with the input buffer, queue it back */
+ imx_media_dma_buf_queue(priv->in_ring,
+ priv->curr_in_buf->index);
+
+ /* current input buffer is now last */
+ priv->last_in_buf = priv->curr_in_buf;
+ } else {
+ /*
+ * priv->next buffer is now the active one due
+ * to IPU double-buffering
+ */
+ imx_media_dma_buf_set_active(priv->next_out_buf);
+ }

+
+ /* bump the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+

+ if (priv->csi_direct) {
+ prepare_prpvf_out_buffer(priv);

+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ }

+
+unlock:
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+ return IRQ_HANDLED;
+}
+

+static long prpvf_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);
+ struct imx_media_dma_buf_ring **ring;
+ struct imx_media_dma_buf *buf;

+ unsigned long flags;
+

+ switch (cmd) {
+ case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+ if (!priv->in_ring)
+ return -EINVAL;
+ ring = (struct imx_media_dma_buf_ring **)arg;
+ *ring = priv->in_ring;
+ break;
+ case IMX_MEDIA_NEW_DMA_BUF:

+ spin_lock_irqsave(&priv->irqlock, flags);
+ if (!imx_media_dma_buf_get_active(priv->out_ring)) {
+ buf = imx_media_dma_buf_dequeue(priv->in_ring);
+ if (buf) {
+ prepare_vdi_in_buffers(priv, buf);
+ prepare_prpvf_out_buffer(priv);
+ }
+ }
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+static irqreturn_t nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct prpvf_priv *priv = dev_id;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void prpvf_eof_timeout(unsigned long data)
+{
+ struct prpvf_priv *priv = (struct prpvf_priv *)data;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void setup_vdi_channel(struct prpvf_priv *priv,
+ struct ipuv3_channel *channel,
+ dma_addr_t phys0, dma_addr_t phys1,
+ bool out_chan)
+{

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ unsigned int burst_size;
+ struct ipu_image image;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (out_chan) {
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+ } else {
+ /* one field to VDIC channels */
+ infmt->height /= 2;
+ imx_media_mbus_fmt_to_ipu_image(&image, infmt);
+ infmt->height *= 2;
+ }
+ image.phys0 = phys0;
+ image.phys1 = phys1;
+
+ ipu_cpmem_zero(channel);
+ ipu_cpmem_set_image(channel, &image);
+
+ if (out_chan) {

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;

+ ipu_cpmem_set_burstsize(channel, burst_size);
+ ipu_ic_task_idma_init(priv->ic_vf, channel,
+ outfmt->width, outfmt->height,
+ burst_size, IPU_ROTATE_NONE);
+ } else {
+ burst_size = (infmt->width & 0xf) ? 8 : 16;
+ ipu_cpmem_set_burstsize(channel, burst_size);
+ }
+
+ ipu_cpmem_set_axi_id(channel, 1);
+
+ ipu_idmac_set_double_buffer(channel, priv->csi_direct && out_chan);
+}
+
+static int prpvf_setup_direct(struct prpvf_priv *priv)
+{
+ struct imx_media_dma_buf *buf0, *buf1;
+
+ /* set VDIC to receive from CSI for direct path */
+ ipu_fsu_link(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+ IPUV3_CHANNEL_CSI_VDI_PREV);

+
+ priv->ipu_buf_num = 0;
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);

+ priv->next_out_buf = buf1;
+
+ /* init the prpvf out channel */
+ setup_vdi_channel(priv, priv->prpvf_out_ch,
+ buf0->phys, buf1->phys, true);

+
+ return 0;
+}
+

+static void prpvf_start_direct(struct prpvf_priv *priv)

+{
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->prpvf_out_ch, 0);
+ ipu_idmac_select_buffer(priv->prpvf_out_ch, 1);

+
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_stop_direct(struct prpvf_priv *priv)
+{
+ ipu_idmac_disable_channel(priv->prpvf_out_ch);
+}
+
+static void prpvf_disable_direct(struct prpvf_priv *priv)
+{
+ ipu_fsu_unlink(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+ IPUV3_CHANNEL_CSI_VDI_PREV);
+}
+
+static int prpvf_setup_indirect(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ struct v4l2_mbus_framefmt *infmt;
+ const struct imx_media_pixfmt *incc;
+ int in_size, i, ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];

+ incc = priv->cc[priv->input_pad];
+

+
+ priv->ipu_buf_num = 0;

+
+ if (priv->in_ring) {

+ v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ }

+ priv->in_ring = imx_media_alloc_dma_buf_ring(
+ priv->md, &priv->src_sd->entity,

+ &ic_priv->sd.entity,
+ in_size, IMX_MEDIA_MIN_RING_BUFS_PRPVF, true);
+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&ic_priv->sd, "failed to alloc dma-buf ring\n");

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;
+ return ret;
+ }
+

+ for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS_PRPVF; i++)
+ imx_media_dma_buf_queue(priv->in_ring, i);
+
+ priv->last_in_buf = NULL;
+ priv->curr_in_buf = NULL;
+
+ /* translate V4L2_FIELD_ALTERNATE to SEQ_TB or SEQ_BT */
+ priv->fieldtype = infmt->field;
+ if (infmt->field == V4L2_FIELD_ALTERNATE)
+ priv->fieldtype = (priv->std & V4L2_STD_525_60) ?
+ V4L2_FIELD_SEQ_TB : V4L2_FIELD_SEQ_BT;
+
+ /* init the vdi-in channels */
+ setup_vdi_channel(priv, priv->vdi_in_ch_p, 0, 0, false);
+ setup_vdi_channel(priv, priv->vdi_in_ch, 0, 0, false);
+ setup_vdi_channel(priv, priv->vdi_in_ch_n, 0, 0, false);
+
+ /* init the prpvf out channel */
+ setup_vdi_channel(priv, priv->prpvf_out_ch, 0, 0, true);

+
+ return 0;
+}
+

+static void prpvf_start_indirect(struct prpvf_priv *priv)

+{
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->prpvf_out_ch);
+ ipu_idmac_enable_channel(priv->vdi_in_ch_p);
+ ipu_idmac_enable_channel(priv->vdi_in_ch);
+ ipu_idmac_enable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_stop_indirect(struct prpvf_priv *priv)
+{
+ /* disable channels */
+ ipu_idmac_disable_channel(priv->prpvf_out_ch);
+ ipu_idmac_disable_channel(priv->vdi_in_ch_p);
+ ipu_idmac_disable_channel(priv->vdi_in_ch);
+ ipu_idmac_disable_channel(priv->vdi_in_ch_n);
+}
+
+static void prpvf_disable_indirect(struct prpvf_priv *priv)
+{
+}
+
+static struct prpvf_pipeline_ops direct_ops = {
+ .setup = prpvf_setup_direct,
+ .start = prpvf_start_direct,
+ .stop = prpvf_stop_direct,
+ .disable = prpvf_disable_direct,
+};
+
+static struct prpvf_pipeline_ops indirect_ops = {
+ .setup = prpvf_setup_indirect,
+ .start = prpvf_start_indirect,
+ .stop = prpvf_stop_indirect,
+ .disable = prpvf_disable_indirect,
+};
+
+static int prpvf_start(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;
+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");
+ return -EINVAL;
+ }
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ priv->ops = priv->csi_direct ? &direct_ops : &indirect_ops;
+
+ ret = prpvf_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ /* set IC to receive from VDIC */
+ ipu_set_ic_src_mux(priv->ipu, 0, true);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ goto out_put_ipu;

+
+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ /* request EOF irq for prpvf out channel */
+ priv->out_eof_irq = ipu_idmac_channel_irq(priv->ipu,
+ priv->prpvf_out_ch,
+ IPU_IRQ_EOF);
+ ret = devm_request_irq(ic_priv->dev, priv->out_eof_irq,
+ prpvf_out_eof_interrupt, 0,
+ "imx-ic-prpvf-out-eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,
+ "Error registering out eof irq: %d\n", ret);
+ goto out_put_ipu;
+ }
+
+ /* request NFB4EOF irq */

+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->prpvf_out_ch,
+ IPU_IRQ_NFB4EOF);
+ ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+ nfb4eof_interrupt, 0,
+ "imx-ic-prpvf-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,
+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_free_eof_irq;
+ }
+
+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_std, &priv->std);
+ if (ret)
+ goto out_free_nfb4eof_irq;
+
+ /* init the VDIC */
+ ipu_vdi_setup(priv->vdi, infmt->code,
+ infmt->width, infmt->height);
+ ipu_vdi_set_field_order(priv->vdi, priv->std, infmt->field);
+ ipu_vdi_set_motion(priv->vdi, priv->motion);
+
+ ret = ipu_ic_task_init(priv->ic_vf,
+ infmt->width, infmt->height,
+ outfmt->width, outfmt->height,
+ incc->cs, outcc->cs);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ ret = priv->ops->setup(priv);
+ if (ret)
+ goto out_free_nfb4eof_irq;
+
+ ipu_vdi_enable(priv->vdi);
+ ipu_ic_enable(priv->ic_vf);
+
+ priv->ops->start(priv);
+
+ /* enable the IC VF task */
+ ipu_ic_task_enable(priv->ic_vf);
+
+ /* start the EOF timeout timer */

+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+out_free_nfb4eof_irq:
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_free_eof_irq:
+ devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+out_put_ipu:
+ prpvf_put_ipu_resources(priv);

+ return ret;
+}
+

+static void prpvf_stop(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ unsigned long flags;
+ int ret;
+

+ /* mark next EOF interrupt as the last before stream off */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ priv->last_eof = true;
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+
+ /*

+ * and then wait for interrupt handler to mark completion.
+ */
+ ret = wait_for_completion_timeout(
+ &priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+ if (ret == 0)

+ v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+ ipu_ic_task_disable(priv->ic_vf);
+ priv->ops->stop(priv);
+ ipu_ic_disable(priv->ic_vf);
+ ipu_vdi_disable(priv->vdi);
+ priv->ops->disable(priv);
+
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+ devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+ prpvf_put_ipu_resources(priv);

+static int prpvf_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct prpvf_priv *priv = container_of(ctrl->handler,
+ struct prpvf_priv, ctrl_hdlr);
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ enum ipu_motion_sel motion;
+
+ switch (ctrl->id) {
+ case V4L2_CID_IMX_MOTION:
+ motion = ctrl->val;
+ if (motion != priv->motion) {
+ /* can't change motion control mid-streaming */
+ if (priv->stream_on)
+ return -EBUSY;
+ priv->motion = motion;
+ }
+ break;
+ default:
+ v4l2_err(&ic_priv->sd, "Invalid control\n");

+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static const struct v4l2_ctrl_ops prpvf_ctrl_ops = {
+ .s_ctrl = prpvf_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpvf_custom_ctrl[] = {
+ {
+ .ops = &prpvf_ctrl_ops,
+ .id = V4L2_CID_IMX_MOTION,
+ .name = "Motion Compensation",
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .def = MOTION_NONE,
+ .min = MOTION_NONE,
+ .max = HIGH_MOTION,
+ .step = 1,
+ },
+};
+
+#define PRPVF_NUM_CONTROLS ARRAY_SIZE(prpvf_custom_ctrl)
+
+static int prpvf_init_controls(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+ const struct v4l2_ctrl_config *c;

+ int i, ret;
+

+ v4l2_ctrl_handler_free(hdlr);
+ v4l2_ctrl_handler_init(hdlr, PRPVF_NUM_CONTROLS);
+
+ for (i = 0; i < PRPVF_NUM_CONTROLS; i++) {
+ c = &prpvf_custom_ctrl[i];
+ v4l2_ctrl_new_custom(hdlr, c, NULL);
+ }
+
+ ic_priv->sd.ctrl_handler = hdlr;
+
+ if (hdlr->error) {
+ ret = hdlr->error;
+ goto out_free;
+ }
+
+ v4l2_ctrl_handler_setup(hdlr);
+ return 0;
+
+out_free:
+ v4l2_ctrl_handler_free(hdlr);

+ return ret;
+}
+

+static int prpvf_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = prpvf_start(priv);

+ else if (!enable && priv->stream_on)

+ prpvf_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int prpvf_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);
+ bool allow_planar, allow_rgb;
+
+ if (code->pad >= PRPVF_NUM_PADS)
+ return -EINVAL;
+
+ allow_planar = (code->pad == priv->output_pad);
+ allow_rgb = allow_planar;

+
+ return imx_media_enum_format(&code->code, code->index,

+ allow_rgb, allow_planar);
+}
+
+static int prpvf_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PRPVF_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int prpvf_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ bool allow_planar, allow_rgb;
+ u32 code;
+
+ if (sdformat->pad >= PRPVF_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+ allow_rgb = allow_planar;

+
+ cc = imx_media_find_format(0, sdformat->format.code,

+ allow_rgb, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, false, false);
+ cc = imx_media_find_format(0, code, false, false);

+ sdformat->format.code = cc->codes[0];
+ }
+

+ if (sdformat->pad == priv->output_pad) {

+ sdformat->format.width = min_t(__u32,
+ sdformat->format.width,
+ MAX_W_IC);
+ sdformat->format.height = min_t(__u32,
+ sdformat->format.height,
+ MAX_H_IC);
+ /* IC resizer cannot downsize more than 4:1 */
+ sdformat->format.width = max_t(__u32, sdformat->format.width,
+ infmt->width / 4);
+ sdformat->format.height = max_t(__u32, sdformat->format.height,
+ infmt->height / 4);
+
+ /* output is always progressive! */
+ sdformat->format.field = V4L2_FIELD_NONE;
+ } else {
+ sdformat->format.width = min_t(__u32,
+ sdformat->format.width,
+ MAX_W_VDIC);
+ sdformat->format.height = min_t(__u32,
+ sdformat->format.height,
+ MAX_H_VDIC);
+
+ /* input must be interlaced! Choose alternate if not */
+ if (!V4L2_FIELD_HAS_BOTH(sdformat->format.field))
+ sdformat->format.field = V4L2_FIELD_ALTERNATE;
+ }

+static int prpvf_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpvf_priv *priv = ic_priv->task_priv;
+ struct v4l2_subdev *remote_sd;
+ int ret;
+
+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {

+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+
+ /* this is sink pad */

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+

+ priv->csi_direct = ((priv->src_sd->grp_id &
+ IMX_MEDIA_GRP_ID_CSI) != 0);
+
+ ret = prpvf_init_controls(priv);

+ if (ret)
+ return ret;

+ } else {
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ priv->src_sd = NULL;
+ }
+

+ return 0;
+}
+

+static int prpvf_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpvf_priv *priv = ic_priv->task_priv;
+ struct v4l2_mbus_config sensor_mbus_cfg;
+ struct imx_media_subdev *csi;
+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link,
+ source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);

+ priv->sensor = NULL;

+ return ret;
+ }
+

+ if (!priv->csi_direct) {
+ csi = imx_media_find_pipeline_subdev(
+ priv->md, &ic_priv->sd.entity, IMX_MEDIA_GRP_ID_CSI);
+ if (IS_ERR(csi)) {
+ v4l2_err(&ic_priv->sd, "no CSI attached\n");
+ ret = PTR_ERR(csi);

+ return ret;
+ }
+

+ priv->csi_sd = csi->sd;

+ return 0;
+ }
+

+ priv->csi_sd = priv->src_sd;
+
+ if (priv->motion != HIGH_MOTION) {
+ v4l2_err(&ic_priv->sd,
+ "direct CSI pipeline requires HIGH_MOTION\n");

+ return -EINVAL;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {

+ int vc_num = 0;

+ /* see NOTE in imx-csi.c */

+#if 0
+ vc_num = imx_media_find_mipi_csi2_channel(

+ priv->md, &ic_priv->sd.entity);

+ if (vc_num < 0)
+ return vc_num;
+#endif

+ /* only virtual channel 0 can be sent to IC */
+ if (vc_num != 0)
+ return -EINVAL;
+ } else {
+ /* only 8-bit pixels can be sent to IC for parallel busses */
+ if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)

+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int prpvf_registered(struct v4l2_subdev *sd)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < PRPVF_NUM_PADS; i++) {

+ return media_entity_pads_init(&sd->entity, PRPVF_NUM_PADS, priv->pad);
+}
+
+static struct v4l2_subdev_pad_ops prpvf_pad_ops = {
+ .enum_mbus_code = prpvf_enum_mbus_code,
+ .get_fmt = prpvf_get_fmt,
+ .set_fmt = prpvf_set_fmt,
+ .link_validate = prpvf_link_validate,
+};
+
+static struct v4l2_subdev_video_ops prpvf_video_ops = {
+ .s_stream = prpvf_s_stream,
+};
+
+static struct v4l2_subdev_core_ops prpvf_core_ops = {
+ .ioctl = prpvf_ioctl,
+};
+
+static struct media_entity_operations prpvf_entity_ops = {
+ .link_setup = prpvf_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_ops prpvf_subdev_ops = {
+ .video = &prpvf_video_ops,
+ .pad = &prpvf_pad_ops,
+ .core = &prpvf_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops prpvf_internal_ops = {
+ .registered = prpvf_registered,
+};
+
+static int prpvf_init(struct imx_ic_priv *ic_priv)
+{
+ struct prpvf_priv *priv;
+
+ priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);

+ if (!priv)
+ return -ENOMEM;
+

+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;
+
+ spin_lock_init(&priv->irqlock);

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = prpvf_eof_timeout;

+
+ return 0;
+}
+

+static void prpvf_remove(struct imx_ic_priv *ic_priv)
+{
+ struct prpvf_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

+
+struct imx_ic_ops imx_ic_prpvf_ops = {
+ .subdev_ops = &prpvf_subdev_ops,
+ .internal_ops = &prpvf_internal_ops,
+ .entity_ops = &prpvf_entity_ops,
+ .init = prpvf_init,
+ .remove = prpvf_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
new file mode 100644
index 0000000..9aed5f5
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic.h
@@ -0,0 +1,36 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.

+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+
+struct imx_ic_priv {
+ struct device *dev;
+ struct v4l2_subdev sd;
+ int ipu_id;
+ int task_id;
+ void *task_priv;
+};
+
+struct imx_ic_ops {
+ struct v4l2_subdev_ops *subdev_ops;
+ struct v4l2_subdev_internal_ops *internal_ops;
+ struct media_entity_operations *entity_ops;
+
+ int (*init)(struct imx_ic_priv *ic_priv);
+ void (*remove)(struct imx_ic_priv *ic_priv);
+};
+
+extern struct imx_ic_ops imx_ic_prpenc_ops;
+extern struct imx_ic_ops imx_ic_prpvf_ops;
+extern struct imx_ic_ops imx_ic_pp_ops;
+
+#endif
+
--
2.7.4

Steve Longerbeam

unread,

Dec 29, 2016, 5:40:08 PM12/29/16

Add pinctrl groups for both GPT input capture channels.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
1 file changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 516bac6..83ac2ff 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -457,6 +457,18 @@
>;
};

+ pinctrl_gpt_input_capture0: gptinputcapture0grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1 0x80000000
+ >;
+ };
+
+ pinctrl_gpt_input_capture1: gptinputcapture1grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2 0x80000000
+ >;
+ };
+
pinctrl_spdif: spdifgrp {
fsl,pins = <
MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
--
2.7.4

Lothar Waßmann

unread,

Dec 30, 2016, 2:30:05 AM12/30/16

Hi,

The pin is already initialized to the inactive state thru the
GPIOD_OUT_LOW flag in devm_gpiod_get_optional(), so this call to
gpiod_set_value() is useless.

Lothar Waßmann

Linus Walleij

unread,

Dec 30, 2016, 8:20:05 AM12/30/16

On Thu, Dec 29, 2016 at 11:27 PM, Steve Longerbeam
<slong...@gmail.com> wrote:

> Add optional reset-gpios pin control. If present, de-assert the
> specified reset gpio pin to bring the chip out of reset.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> Cc: Linus Walleij <linus....@linaro.org>
> Cc: Alexandre Courbot <gnu...@gmail.com>
> Cc: linux...@vger.kernel.org
> Cc: linux-...@vger.kernel.org

This seems like a separate patch from the other 19 patches so please send it
separately so I can handle logistics easier in the future.

> @@ -133,6 +134,7 @@ struct pca953x_chip {
> const char *const *names;
> unsigned long driver_data;
> struct regulator *regulator;
> + struct gpio_desc *reset_gpio;

Why do you even keep this around in the device state container?

As you only use it in the probe() function, use a local variable.

The descriptor will be free():ed by the devm infrastructure anyways.

> + /* see if we need to de-assert a reset pin */
> + chip->reset_gpio = devm_gpiod_get_optional(&client->dev,
> + "reset",
> + GPIOD_OUT_LOW);
> + if (IS_ERR(chip->reset_gpio)) {
> + dev_err(&client->dev, "request for reset pin failed\n");
> + return PTR_ERR(chip->reset_gpio);
> + }

Nice.

> + if (chip->reset_gpio) {
> + /* bring chip out of reset */
> + dev_info(&client->dev, "releasing reset\n");
> + gpiod_set_value(chip->reset_gpio, 0);
> + }

Is this really needed given that you set it low in the
devm_gpiod_get_optional()?

Yours,
Linus Walleij

Steve Longerbeam

unread,

Dec 30, 2016, 1:10:05 PM12/30/16

Hi Linus, Lothar,

On 12/30/2016 05:17 AM, Linus Walleij wrote:
> On Thu, Dec 29, 2016 at 11:27 PM, Steve Longerbeam
> <slong...@gmail.com> wrote:
>
>> Add optional reset-gpios pin control. If present, de-assert the
>> specified reset gpio pin to bring the chip out of reset.
>>
>> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
>> Cc: Linus Walleij <linus....@linaro.org>
>> Cc: Alexandre Courbot <gnu...@gmail.com>
>> Cc: linux...@vger.kernel.org
>> Cc: linux-...@vger.kernel.org
> This seems like a separate patch from the other 19 patches so please send it
> separately so I can handle logistics easier in the future.

Ok, I'll send the next version separately.

>
>
>> @@ -133,6 +134,7 @@ struct pca953x_chip {
>> const char *const *names;
>> unsigned long driver_data;
>> struct regulator *regulator;
>> + struct gpio_desc *reset_gpio;
> Why do you even keep this around in the device state container?
>
> As you only use it in the probe() function, use a local variable.
>
> The descriptor will be free():ed by the devm infrastructure anyways.

I think my reasoning for putting the gpio handle into the device
struct was for possibly using it for run-time reset control at some
point. But that could be done later if needed, so I'll go ahead and
make it local.

>> + /* see if we need to de-assert a reset pin */
>> + chip->reset_gpio = devm_gpiod_get_optional(&client->dev,
>> + "reset",
>> + GPIOD_OUT_LOW);
>> + if (IS_ERR(chip->reset_gpio)) {
>> + dev_err(&client->dev, "request for reset pin failed\n");
>> + return PTR_ERR(chip->reset_gpio);
>> + }
> Nice.
>
>> + if (chip->reset_gpio) {
>> + /* bring chip out of reset */
>> + dev_info(&client->dev, "releasing reset\n");
>> + gpiod_set_value(chip->reset_gpio, 0);
>> + }
> Is this really needed given that you set it low in the
> devm_gpiod_get_optional()?

Yep, this is left over from a previous iteration, but it isn't needed now.
I'll remove it.

Steve

Fabio Estevam

unread,

Jan 2, 2017, 4:20:06 PM1/2/17

Hi Steve,

On Thu, Dec 29, 2016 at 8:27 PM, Steve Longerbeam <slong...@gmail.com> wrote:
> This is a media driver for video capture on i.MX.
>
> Refer to Documentation/media/v4l-drivers/imx.rst for example capture
> pipelines on SabreSD, SabreAuto, and SabreLite reference platforms.
>
> This patchset includes the OF graph layout as proposed by Philipp Zabel,
> with only minor changes which are enumerated in the patch header.

Patches 13, 14 and 19 miss your Signed-off-by tag.

Tested the whole series on a mx6qsabresd:

Tested-by: Fabio Estevam <fabio....@nxp.com>

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:05 PM1/3/17

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:05 PM1/3/17

Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
for sensors with a MIPI CSI2 interface.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 1 +
drivers/staging/media/imx/imx-mipi-csi2.c | 509 ++++++++++++++++++++++++++++++
2 files changed, 510 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index fe9e992..0decef7 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c

new file mode 100644

index 0000000..84df16e
--- /dev/null
+++ b/drivers/staging/media/imx/imx-mipi-csi2.c
@@ -0,0 +1,509 @@
+/*
+ * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.

+ *
+ * Copyright (c) 2012-2014 Mentor Graphics Inc.

+#include <linux/export.h>
+#include <linux/types.h>
+#include <linux/init.h>
+#include <linux/platform_device.h>
+#include <linux/err.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/clk.h>
+#include <linux/list.h>
+#include <linux/irq.h>
+#include <linux/of_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>

+#include <media/v4l2-subdev.h>
+#include <media/v4l2-async.h>
+#include <asm/mach/irq.h>

+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+

+/*
+ * there must be 5 pads: 1 input pad from sensor, and
+ * the 4 virtual channel output pads
+ */
+#define CSI2_NUM_SINK_PADS 1
+#define CSI2_NUM_SRC_PADS 4
+#define CSI2_NUM_PADS 5
+
+struct imxcsi2_dev {

+ struct device *dev;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[CSI2_NUM_PADS];
+ struct v4l2_mbus_framefmt format_mbus;

+ struct v4l2_subdev *src_sd;

+ struct v4l2_subdev *sink_sd[CSI2_NUM_SRC_PADS];
+ int input_pad;
+ struct clk *dphy_clk;
+ struct clk *cfg_clk;
+ struct clk *pix_clk; /* what is this? */
+ void __iomem *base;
+ int intr1;
+ int intr2;
+ struct v4l2_of_bus_mipi_csi2 bus;

+ bool on;

+ bool stream_on;
+};
+
+#define DEVICE_NAME "imx-mipi-csi2"
+
+/* Register offsets */
+#define CSI2_VERSION 0x000
+#define CSI2_N_LANES 0x004
+#define CSI2_PHY_SHUTDOWNZ 0x008
+#define CSI2_DPHY_RSTZ 0x00c
+#define CSI2_RESETN 0x010
+#define CSI2_PHY_STATE 0x014
+#define CSI2_DATA_IDS_1 0x018
+#define CSI2_DATA_IDS_2 0x01c
+#define CSI2_ERR1 0x020
+#define CSI2_ERR2 0x024
+#define CSI2_MSK1 0x028
+#define CSI2_MSK2 0x02c
+#define CSI2_PHY_TST_CTRL0 0x030
+#define CSI2_PHY_TST_CTRL1 0x034
+#define CSI2_SFT_RESET 0xf00
+

+static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
+{
+ return container_of(sdev, struct imxcsi2_dev, sd);
+}
+

+ imxcsi2_enable(csi2, true);
+}
+

+static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
+{
+ u32 reg;

+ int i;
+

+ /* wait for mipi sensor ready */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+ if (reg != 0x200)
+ break;
+ usleep_range(10000, 20000);

+ }
+

+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for clock lane timeout, phy_state = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ /* wait for mipi stable */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_ERR1);
+ if (reg == 0x0)
+ break;
+ usleep_range(10000, 20000);

+ }
+

+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for controller timeout, err1 = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ /* finally let's wait for active clock on the clock lane */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+ if (reg & (1 << 8))
+ break;
+ usleep_range(10000, 20000);

+ }
+

+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for active clock timeout, phy_state = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
+ imxcsi2_read(csi2, CSI2_VERSION));

+
+ return 0;
+}
+
+/*

+ * V4L2 subdev operations
+ */
+
+static int imxcsi2_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (csi2->sink_sd[local->index])
+ return -EBUSY;

+ csi2->sink_sd[local->index] = remote_sd;
+ } else {
+ csi2->sink_sd[local->index] = NULL;

+ }
+ } else {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (csi2->src_sd)
+ return -EBUSY;
+ csi2->src_sd = remote_sd;
+ } else {
+ csi2->src_sd = NULL;

+ }
+ }
+
+ return 0;
+}
+

+static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+

+ if (on && !csi2->on) {
+ v4l2_info(&csi2->sd, "power ON\n");
+ clk_prepare_enable(csi2->cfg_clk);
+ clk_prepare_enable(csi2->dphy_clk);
+ imxcsi2_set_lanes(csi2);
+ imxcsi2_reset(csi2);
+ } else if (!on && csi2->on) {
+ v4l2_info(&csi2->sd, "power OFF\n");
+ imxcsi2_enable(csi2, false);
+ clk_disable_unprepare(csi2->dphy_clk);
+ clk_disable_unprepare(csi2->cfg_clk);
+ }
+
+ csi2->on = on;

+ return 0;
+}
+

+static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
+{

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);

+ int i, ret = 0;
+

+ if (!csi2->src_sd)
+ return -EPIPE;
+ for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
+ if (csi2->sink_sd[i])
+ break;
+ }
+ if (i >= CSI2_NUM_SRC_PADS)

+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !csi2->stream_on) {
+ clk_prepare_enable(csi2->pix_clk);
+ ret = imxcsi2_dphy_wait(csi2);
+ if (ret)
+ clk_disable_unprepare(csi2->pix_clk);
+ } else if (!enable && csi2->stream_on) {
+ clk_disable_unprepare(csi2->pix_clk);
+ }

+
+ if (!ret)
+ csi2->stream_on = enable;

+ return ret;
+}
+

+static int imxcsi2_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ sdformat->format = csi2->format_mbus;

+
+ return 0;
+}
+

+static int imxcsi2_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ if (sdformat->pad >= CSI2_NUM_PADS)
+ return -EINVAL;
+
+ if (csi2->stream_on)
+ return -EBUSY;
+

+ /* Output pads mirror active input pad, no limits on input pads */
+ if (sdformat->pad != csi2->input_pad)
+ sdformat->format = csi2->format_mbus;

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY)
+ cfg->try_fmt = sdformat->format;
+ else

+ csi2->format_mbus = sdformat->format;

+
+ return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int imxcsi2_registered(struct v4l2_subdev *sd)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+
+ /* get media device */

+ csi2->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(csi2->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 4)
+ return -EINVAL;
+
+ for (i = 0; i < CSI2_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ csi2->pad[i] = pad->pad;
+ if (csi2->pad[i].flags & MEDIA_PAD_FL_SINK)
+ csi2->input_pad = i;
+ }

+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&csi2->format_mbus,
+ 640, 480, 0, V4L2_FIELD_NONE, NULL);

+ if (ret)
+ return ret;
+

+ return media_entity_pads_init(&sd->entity, CSI2_NUM_PADS, csi2->pad);
+}
+

+static struct media_entity_operations imxcsi2_entity_ops = {
+ .link_setup = imxcsi2_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_core_ops imxcsi2_core_ops = {
+ .s_power = imxcsi2_s_power,

+};
+

+static struct v4l2_subdev_video_ops imxcsi2_video_ops = {
+ .s_stream = imxcsi2_s_stream,

+};
+

+static struct v4l2_subdev_pad_ops imxcsi2_pad_ops = {
+ .get_fmt = imxcsi2_get_fmt,
+ .set_fmt = imxcsi2_set_fmt,

+};
+

+static struct v4l2_subdev_ops imxcsi2_subdev_ops = {
+ .core = &imxcsi2_core_ops,
+ .video = &imxcsi2_video_ops,
+ .pad = &imxcsi2_pad_ops,

+};
+

+static struct v4l2_subdev_internal_ops imxcsi2_internal_ops = {
+ .registered = imxcsi2_registered,
+};
+
+static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
+{
+ struct device_node *node = csi2->dev->of_node;
+ struct device_node *epnode;
+ struct v4l2_of_endpoint ep;

+ int ret = 0;
+

+ epnode = of_graph_get_next_endpoint(node, NULL);
+ if (!epnode) {

+ v4l2_err(&csi2->sd, "failed to get endpoint node\n");

+ return -EINVAL;
+ }
+

+ v4l2_of_parse_endpoint(epnode, &ep);
+ if (ep.bus_type != V4L2_MBUS_CSI2) {

+ v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");
+ ret = -EINVAL;

+ goto out;
+ }
+

+ csi2->bus = ep.bus.mipi_csi2;
+
+ v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
+ v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);
+out:
+ of_node_put(epnode);

+ return ret;
+}
+

+static int imxcsi2_probe(struct platform_device *pdev)
+{

+ struct imxcsi2_dev *csi2;
+ struct resource *res;

+ int ret;
+

+ csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+ if (!csi2)

+ return -ENOMEM;
+
+ csi2->dev = &pdev->dev;
+

+ v4l2_subdev_init(&csi2->sd, &imxcsi2_subdev_ops);
+ v4l2_set_subdevdata(&csi2->sd, &pdev->dev);
+ csi2->sd.internal_ops = &imxcsi2_internal_ops;
+ csi2->sd.entity.ops = &imxcsi2_entity_ops;
+ csi2->sd.dev = &pdev->dev;
+ csi2->sd.owner = THIS_MODULE;
+ csi2->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+ strcpy(csi2->sd.name, DEVICE_NAME);
+ /* FIXME: this is a mipi-csi2 receiver, function isn't right */
+ csi2->sd.entity.function = MEDIA_ENT_F_ATV_DECODER;
+ csi2->sd.grp_id = IMX_MEDIA_GRP_ID_CSI2;
+
+ ret = imxcsi2_parse_endpoints(csi2);

+ if (ret)
+ return ret;
+

+ csi2->cfg_clk = devm_clk_get(&pdev->dev, "cfg_clk");
+ if (IS_ERR(csi2->cfg_clk)) {

+ v4l2_err(&csi2->sd, "failed to get cfg clock\n");
+ ret = PTR_ERR(csi2->cfg_clk);

+ return ret;
+ }
+

+ csi2->dphy_clk = devm_clk_get(&pdev->dev, "dphy_clk");
+ if (IS_ERR(csi2->dphy_clk)) {

+ v4l2_err(&csi2->sd, "failed to get dphy clock\n");
+ ret = PTR_ERR(csi2->dphy_clk);

+ return ret;
+ }
+

+ csi2->pix_clk = devm_clk_get(&pdev->dev, "pix_clk");
+ if (IS_ERR(csi2->pix_clk)) {

+ v4l2_err(&csi2->sd, "failed to get pixel clock\n");
+ ret = PTR_ERR(csi2->pix_clk);

+ return ret;
+ }
+

+ csi2->intr1 = platform_get_irq(pdev, 0);
+ csi2->intr2 = platform_get_irq(pdev, 1);
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+
+ if (!res || csi2->intr1 < 0 || csi2->intr2 < 0) {
+ v4l2_err(&csi2->sd, "failed to get platform resources\n");
+ return -ENODEV;
+ }
+
+ csi2->base = devm_ioremap(&pdev->dev, res->start, PAGE_SIZE);
+ if (!csi2->base) {
+ v4l2_err(&csi2->sd, "failed to map CSI-2 registers\n");

+ return -ENOMEM;
+ }
+

+ platform_set_drvdata(pdev, &csi2->sd);
+

+ return v4l2_async_register_subdev(&csi2->sd);
+}
+
+static int imxcsi2_remove(struct platform_device *pdev)

+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ imxcsi2_s_power(sd, 0);
+
+ v4l2_async_unregister_subdev(&csi2->sd);

+ media_entity_cleanup(&csi2->sd.entity);

+ v4l2_device_unregister_subdev(sd);
+
+ return 0;
+}
+

+static const struct of_device_id imxcsi2_dt_ids[] = {

+ { .compatible = "fsl,imx-mipi-csi2", },

+ { /* sentinel */ }
+};

+MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
+
+static struct platform_driver imxcsi2_driver = {

+ .driver = {
+ .name = DEVICE_NAME,
+ .owner = THIS_MODULE,

+ .of_match_table = imxcsi2_dt_ids,
+ },
+ .probe = imxcsi2_probe,
+ .remove = imxcsi2_remove,
+};
+
+module_platform_driver(imxcsi2_driver);
+

+MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");
+

--
2.7.4

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:05 PM1/3/17

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
Both hang off the same i2c2 bus, so they require different (and non-
default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on
mipi_csi. It is set to transmit over MIPI virtual channel 1.

Note there is a pin conflict with GPIO6. This pin functions as a power
input pin to the OV5642, but ENET uses it as the h/w workaround for
erratum ERR006687, to wake-up the ARM cores on normal RX and TX packet
done events (see 6261c4c8). So workaround 6261c4c8 is reverted here to
support the OV5642, and the "fsl,err006687-workaround-present" boolean
also must be removed. The result is that the CPUidle driver will no longer
allow entering the deep idle states on the sabrelite.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

+ MX6QDL_PAD_GPIO_3__CCM_CLKO2 0x80000000
+ >;
+ };
+

+ pinctrl_ipu1_csi0: ipu1grp-csi0 {
+ fsl,pins = <
+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x80000000
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x80000000
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x80000000
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x80000000
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x80000000
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x80000000
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x80000000
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x80000000
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x80000000
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x80000000
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x80000000

+ MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x80000000
+ >;
+ };
+

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:06 PM1/3/17

Hi Fabio,

On 01/02/2017 01:09 PM, Fabio Estevam wrote:
> Hi Steve,
>
> On Thu, Dec 29, 2016 at 8:27 PM, Steve Longerbeam <slong...@gmail.com> wrote:
>> This is a media driver for video capture on i.MX.
>>
>> Refer to Documentation/media/v4l-drivers/imx.rst for example capture
>> pipelines on SabreSD, SabreAuto, and SabreLite reference platforms.
>>
>> This patchset includes the OF graph layout as proposed by Philipp Zabel,
>> with only minor changes which are enumerated in the patch header.
> Patches 13, 14 and 19 miss your Signed-off-by tag.

thanks for catching. I've added them to version 2.

Steve

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:06 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

gpio-controller;
#gpio-cells = <2>;
};
+

+ camera: adv7180@21 {
+ compatible = "adi,adv7180";
+ reg = <0x21>;
+ powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+ interrupt-parent = <&gpio1>;
+ interrupts = <27 0x8>;
+ inputs = <0x00 0x02>;
+ input-names = "ADV7180 Composite on Ain1",
+ "ADV7180 Composite on Ain3";
+
+ port {

+ adv7180_to_ipu1_csi0_mux: endpoint {

+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;

+ };
+ };
+ };
};

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;

+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+ bus-width = <8>;

+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+

&clks {
assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
<&clks IMX6QDL_PLL4_BYPASS>,
@@ -451,6 +483,30 @@
>;
};

+ pinctrl_ipu1_csi0: ipu1grp-csi0 {
+ fsl,pins = <

+ MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04 0x80000000
+ MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05 0x80000000
+ MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06 0x80000000
+ MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07 0x80000000
+ MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08 0x80000000
+ MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09 0x80000000
+ MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10 0x80000000
+ MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11 0x80000000

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:06 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu1_csi0_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ };
+ };
+ };
+

+ ipu1_csi1_mux: ipu1_csi1_mux@34 {
+ compatible = "imx-video-mux";
+ reg = <0x34 0x38>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu1_csi1_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ };
+ };
+ };
+};
+

+&ipu1_csi1 {
+ ipu1_csi1_from_ipu1_csi1_mux: endpoint {
+ remote-endpoint = <&ipu1_csi1_mux_to_ipu1_csi1>;
+ };
+};
+
+&mipi_csi {
+ port@0 {
+ reg = <0>;
+
+ mipi_csi_from_mipi_sensor: endpoint {
+ };
+ };
+
+ port@1 {

+ reg = <1>;

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ mipi_vc0_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+ };
+
+ mipi_vc0_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc0>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ mipi_vc1_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc1>;
+ };
+
+ mipi_vc1_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc1>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ mipi_vc2_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc2>;
+ };
+
+ mipi_vc2_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc2>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;

+ #address-cells = <1>;
+ #size-cells = <0>;
+

diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..56a314f 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi

@@ -143,10 +143,18 @@

ipu2_csi0: port@0 {
reg = <0>;
+
+ ipu2_csi0_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu2_csi0>;
+ };
};

ipu2_csi1: port@1 {

reg = <1>;
+

+ ipu2_csi1_from_ipu2_csi1_mux: endpoint {
+ remote-endpoint = <&ipu2_csi1_mux_to_ipu2_csi1>;
+ };
};

ipu2_di0: port@2 {
@@ -266,6 +274,76 @@
};
};

+&gpr {
+ ipu1_csi0_mux: ipu1_csi0_mux@4 {
+ compatible = "imx-video-mux";
+ reg = <0x04 0x80000>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu1_csi0_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ ipu1_csi0_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+ remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;

+ };
+ };
+ };
+

+ ipu2_csi1_mux: ipu2_csi1_mux@4 {
+ compatible = "imx-video-mux";
+ reg = <0x04 0x100000>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu2_csi1_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu2_csi1_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ ipu2_csi1_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu2_csi1_mux_to_ipu2_csi1: endpoint {
+ remote-endpoint = <&ipu2_csi1_from_ipu2_csi1_mux>;

+ };
+ };
+ };
+};
+

+&ipu1_csi1 {
+ ipu1_csi1_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi1>;
+ };
+};
+
&ldb {
clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
<&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -312,6 +390,47 @@
};
};

+&mipi_csi {
+ port@0 {
+ reg = <0>;
+
+ mipi_csi_from_mipi_sensor: endpoint {
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 7b546e3..89218a4 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi

@@ -799,8 +799,10 @@
};

gpr: iomuxc-gpr@020e0000 {
- compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+ compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
reg = <0x020e0000 0x38>;

+ #address-cells = <1>;
+ #size-cells = <0>;

};

iomuxc: iomuxc@020e0000 {
@@ -1127,6 +1129,8 @@
mipi_csi: mipi@021dc000 {

compatible = "fsl,imx-mipi-csi2";
reg = <0x021dc000 0x4000>;

+ #address-cells = <1>;
+ #size-cells = <0>;

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:06 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 1 +
drivers/staging/media/imx/imx-smfc.c | 739 +++++++++++++++++++++++++++++++++++
2 files changed, 740 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-smfc.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 133672a..3559d7b 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

diff --git a/drivers/staging/media/imx/imx-smfc.c b/drivers/staging/media/imx/imx-smfc.c

new file mode 100644
index 0000000..565048c
--- /dev/null
+++ b/drivers/staging/media/imx/imx-smfc.c
@@ -0,0 +1,739 @@
+/*
+ * V4L2 Capture SMFC Subdev for Freescale i.MX5/6 SOC
+ *

+ * This subdevice handles capture of raw/unconverted video frames
+ * from the CSI, directly to memory via the Sensor Multi-FIFO Controller.

+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.

+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/timer.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/interrupt.h>
+#include <linux/spinlock.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>
+#include <media/imx.h>

+#include "imx-media.h"
+
+/*

+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W 176
+#define MIN_H 144
+#define MAX_W 8192
+#define MAX_H 4096
+#define W_ALIGN 4 /* multiple of 16 pixels */
+#define H_ALIGN 1 /* multiple of 2 lines */
+#define S_ALIGN 1 /* multiple of 2 */
+
+#define SMFC_NUM_PADS 2
+
+struct imx_smfc_priv {

+ struct device *dev;
+ struct ipu_soc *ipu;

+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[SMFC_NUM_PADS];
+ int ipu_id;
+ int smfc_id;

+ int input_pad;
+ int output_pad;
+

+ struct ipuv3_channel *smfc_ch;
+ struct ipu_smfc *smfc;

+
+ struct v4l2_mbus_framefmt format_mbus[SMFC_NUM_PADS];
+ const struct imx_media_pixfmt *cc[SMFC_NUM_PADS];
+

+ struct v4l2_mbus_config sensor_mbus_cfg;
+

+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct imx_media_dma_buf *next;
+
+ int ipu_buf_num; /* ipu double buffer index: 0-1 */
+

+ /* the sink that will receive the dma buffers */
+ struct v4l2_subdev *sink_sd;

+ struct v4l2_subdev *src_sd;
+

+ /*

+ * the CSI id and mipi virtual channel number at
+ * link validate

+ */
+ int csi_id;
+ int vc_num;

+
+ /* the attached sensor at stream on */
+ struct imx_media_subdev *sensor;
+

+ spinlock_t irqlock;

+ struct timer_list eof_timeout_timer;
+ int eof_irq;
+ int nfb4eof_irq;
+
+ bool stream_on; /* streaming is on */
+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;
+};
+

+static void imx_smfc_put_ipu_resources(struct imx_smfc_priv *priv)
+{

+ if (!IS_ERR_OR_NULL(priv->smfc_ch))
+ ipu_idmac_put(priv->smfc_ch);

+ priv->smfc_ch = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->smfc))
+ ipu_smfc_put(priv->smfc);

+ priv->smfc = NULL;
+}
+

+static int imx_smfc_get_ipu_resources(struct imx_smfc_priv *priv)
+{

+ int ch_num, ret;
+

+ priv->ipu = priv->md->ipu[priv->ipu_id];
+
+ ch_num = IPUV3_CHANNEL_CSI0 + priv->smfc_id;
+
+ priv->smfc = ipu_smfc_get(priv->ipu, ch_num);
+ if (IS_ERR(priv->smfc)) {

+ v4l2_err(&priv->sd, "failed to get SMFC\n");
+ ret = PTR_ERR(priv->smfc);

+ goto out;
+ }
+

+ priv->smfc_ch = ipu_idmac_get(priv->ipu, ch_num);
+ if (IS_ERR(priv->smfc_ch)) {
+ v4l2_err(&priv->sd, "could not get IDMAC channel %u\n", ch_num);
+ ret = PTR_ERR(priv->smfc_ch);

+ goto out;
+ }
+

+ return 0;
+out:
+ imx_smfc_put_ipu_resources(priv);

+ return ret;
+}
+

+static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
+{
+ struct imx_smfc_priv *priv = dev_id;

+ struct imx_media_dma_buf *done, *next;

+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->irqlock, flags);
+
+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;
+ goto unlock;
+ }
+
+ /* inform CSI of this EOF so it can monitor frame intervals */
+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+

+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))
+ imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+ /* priv->next buffer is now the active one */
+ imx_media_dma_buf_set_active(priv->next);
+
+ /* bump the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+

+ if (ipu_idmac_buffer_is_ready(priv->smfc_ch, priv->ipu_buf_num))

+ ipu_idmac_clear_buffer(priv->smfc_ch, priv->ipu_buf_num);

+
+ /* get next queued buffer */
+ next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+

+ ipu_cpmem_set_buffer(priv->smfc_ch, priv->ipu_buf_num, next->phys);
+ ipu_idmac_select_buffer(priv->smfc_ch, priv->ipu_buf_num);

+
+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ priv->next = next;
+
+unlock:
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+ return IRQ_HANDLED;
+}
+

+static irqreturn_t imx_smfc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct imx_smfc_priv *priv = dev_id;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void imx_smfc_eof_timeout(unsigned long data)
+{

+ struct imx_smfc_priv *priv = (struct imx_smfc_priv *)data;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ v4l2_err(&priv->sd, "EOF timeout\n");
+

+ v4l2_subdev_notify_event(&priv->sd, &ev);
+}
+
+/* init the SMFC IDMAC channel */
+static void imx_smfc_setup_channel(struct imx_smfc_priv *priv)

+{
+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ struct imx_media_dma_buf *buf0, *buf1;

+ unsigned int burst_size;
+ struct ipu_image image;

+ bool passthrough;

+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ ipu_cpmem_zero(priv->smfc_ch);

+
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+ image.phys0 = buf0->phys;
+ image.phys1 = buf1->phys;
+ ipu_cpmem_set_image(priv->smfc_ch, &image);
+

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;
+

+ ipu_cpmem_set_burstsize(priv->smfc_ch, burst_size);
+
+ /*

+ * If the sensor uses 16-bit parallel CSI bus, we must handle
+ * the data internally in the IPU as 16-bit generic, aka
+ * passthrough mode.
+ */
+ passthrough = (priv->sensor_mbus_cfg.type != V4L2_MBUS_CSI2 &&

+ priv->sensor->sensor_ep.bus.parallel.bus_width >= 16);
+

+ if (passthrough)
+ ipu_cpmem_set_format_passthrough(priv->smfc_ch, 16);

+
+ /*

+ * Set the channel for the direct CSI-->memory via SMFC
+ * use-case to very high priority, by enabling the watermark
+ * signal in the SMFC, enabling WM in the channel, and setting
+ * the channel priority to high.
+ *
+ * Refer to the i.mx6 rev. D TRM Table 36-8: Calculated priority
+ * value.

+ *

+ * The WM's are set very low by intention here to ensure that
+ * the SMFC FIFOs do not overflow.
+ */
+ ipu_smfc_set_watermark(priv->smfc, 0x02, 0x01);
+ ipu_cpmem_set_high_priority(priv->smfc_ch);
+ ipu_idmac_enable_watermark(priv->smfc_ch, true);
+ ipu_cpmem_set_axi_id(priv->smfc_ch, 0);
+ ipu_idmac_lock_enable(priv->smfc_ch, 8);
+
+ burst_size = ipu_cpmem_get_burstsize(priv->smfc_ch);
+ burst_size = passthrough ?
+ (burst_size >> 3) - 1 : (burst_size >> 2) - 1;
+
+ ipu_smfc_set_burstsize(priv->smfc, burst_size);

+
+ if (outfmt->field == V4L2_FIELD_NONE &&
+ (V4L2_FIELD_HAS_BOTH(infmt->field) ||

+ infmt->field == V4L2_FIELD_ALTERNATE))
+ ipu_cpmem_interlaced_scan(priv->smfc_ch,
+ image.pix.bytesperline);
+
+ ipu_idmac_set_double_buffer(priv->smfc_ch, true);
+}
+
+static void imx_smfc_unsetup(struct imx_smfc_priv *priv)
+{
+ ipu_idmac_disable_channel(priv->smfc_ch);
+ ipu_smfc_disable(priv->smfc);
+}
+
+static void imx_smfc_setup(struct imx_smfc_priv *priv)
+{
+ imx_smfc_setup_channel(priv);
+
+ ipu_cpmem_dump(priv->smfc_ch);

+ ipu_dump(priv->ipu);
+
+ ipu_smfc_enable(priv->smfc);

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->smfc_ch, 0);
+ ipu_idmac_select_buffer(priv->smfc_ch, 1);

+
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->smfc_ch);
+}
+
+static int imx_smfc_start(struct imx_smfc_priv *priv)
+{

+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return -EINVAL;
+ }
+

+ ret = imx_smfc_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ ipu_smfc_map_channel(priv->smfc, priv->csi_id, priv->vc_num);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);
+ if (ret)
+ goto out_put_ipu;
+

+ priv->ipu_buf_num = 0;
+

+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ imx_smfc_setup(priv);

+
+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->smfc_ch,
+ IPU_IRQ_NFB4EOF);

+ ret = devm_request_irq(priv->dev, priv->nfb4eof_irq,
+ imx_smfc_nfb4eof_interrupt, 0,

+ "imx-smfc-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_unsetup;
+ }
+

+ priv->eof_irq = ipu_idmac_channel_irq(priv->ipu, priv->smfc_ch,

+ IPU_IRQ_EOF);
+

+ ret = devm_request_irq(priv->dev, priv->eof_irq,
+ imx_smfc_eof_interrupt, 0,

+ "imx-smfc-eof", priv);
+ if (ret) {
+ v4l2_err(&priv->sd,

+ "Error registering eof irq: %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ /* start the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+out_free_nfb4eof_irq:
+ devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+ imx_smfc_unsetup(priv);
+out_put_ipu:
+ imx_smfc_put_ipu_resources(priv);

+ return ret;
+}
+

+static void imx_smfc_stop(struct imx_smfc_priv *priv)

+{
+ unsigned long flags;

+ int ret;
+

+ /* mark next EOF interrupt as the last before stream off */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ priv->last_eof = true;
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+
+ /*
+ * and then wait for interrupt handler to mark completion.
+ */
+ ret = wait_for_completion_timeout(
+ &priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+ if (ret == 0)

+ v4l2_warn(&priv->sd, "wait last EOF timeout\n");
+
+ devm_free_irq(priv->dev, priv->eof_irq, priv);
+ devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+
+ imx_smfc_unsetup(priv);

+ imx_smfc_put_ipu_resources(priv);
+}
+
+static int imx_smfc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !priv->stream_on)

+ ret = imx_smfc_start(priv);
+ else if (!enable && priv->stream_on)
+ imx_smfc_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int imx_smfc_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,

+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (code->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+

+ return imx_media_enum_format(&code->code, code->index,

+ true, code->pad == priv->output_pad);
+}
+
+static int imx_smfc_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+

+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int imx_smfc_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc, *incc;

+ bool allow_planar;
+ u32 code;

+
+ if (sdformat->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+ allow_planar = (sdformat->pad == priv->output_pad);
+
+ cc = imx_media_find_format(0, sdformat->format.code,
+ true, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, false);
+ cc = imx_media_find_format(0, code, true, false);
+ sdformat->format.code = cc->codes[0];
+ }
+

+ v4l_bound_align_image(&sdformat->format.width, MIN_W, MAX_W,
+ W_ALIGN, &sdformat->format.height,
+ MIN_H, MAX_H, H_ALIGN, S_ALIGN);

+
+ if (sdformat->pad == priv->output_pad) {

+ incc = priv->cc[priv->input_pad];

+ sdformat->format.width = infmt->width;
+ sdformat->format.height = infmt->height;

+ if (sdformat->format.field != V4L2_FIELD_NONE)
+ sdformat->format.field = infmt->field;

+ if (cc->cs != incc->cs) {

+ sdformat->format.code = infmt->code;

+ cc = imx_media_find_format(0, sdformat->format.code,

+ true, false);
+ }
+ }

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int imx_smfc_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;

+ return 0;
+ }
+

+ /* must attach to CSI source */
+ if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_CSI))

+ return -EINVAL;
+

+ return 0;
+}
+

+static int imx_smfc_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+ int ret;
+

+ ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ switch (priv->src_sd->grp_id) {

+ case IMX_MEDIA_GRP_ID_CSI0:
+ priv->csi_id = 0;
+ break;
+ case IMX_MEDIA_GRP_ID_CSI1:
+ priv->csi_id = 1;
+ break;
+ default:

+ return -EINVAL;
+ }
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);
+ priv->sensor = NULL;

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ priv->vc_num = 0;
+ if (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {

+ /* see NOTE in imx-csi.c */
+#if 0

+ priv->vc_num = imx_media_find_mipi_csi2_channel(
+ priv->md, &priv->sd.entity);

+ if (priv->vc_num < 0)
+ return vc_num;
+#endif

+ }
+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int imx_smfc_registered(struct v4l2_subdev *sd)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i;

+
+ /* get media device */

+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+ return -EINVAL;
+
+ for (i = 0; i < SMFC_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;

+
+ /* set a default mbus format */

+ imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);
+ }

+
+ return media_entity_pads_init(&sd->entity, SMFC_NUM_PADS, priv->pad);
+}
+

+static struct media_entity_operations imx_smfc_entity_ops = {
+ .link_setup = imx_smfc_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_video_ops imx_smfc_video_ops = {
+ .s_stream = imx_smfc_s_stream,

+};
+

+static struct v4l2_subdev_pad_ops imx_smfc_pad_ops = {
+ .enum_mbus_code = imx_smfc_enum_mbus_code,
+ .get_fmt = imx_smfc_get_fmt,
+ .set_fmt = imx_smfc_set_fmt,
+ .link_validate = imx_smfc_link_validate,

+};
+

+static struct v4l2_subdev_ops imx_smfc_subdev_ops = {
+ .video = &imx_smfc_video_ops,
+ .pad = &imx_smfc_pad_ops,

+};
+

+static struct v4l2_subdev_internal_ops imx_smfc_internal_ops = {
+ .registered = imx_smfc_registered,
+};
+

+static int imx_smfc_probe(struct platform_device *pdev)
+{
+ struct imx_media_internal_sd_platformdata *pdata;
+ struct imx_smfc_priv *priv;
+

+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)

+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, &priv->sd);
+ priv->dev = &pdev->dev;
+

+ pdata = priv->dev->platform_data;
+ priv->ipu_id = pdata->ipu_id;
+

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = imx_smfc_eof_timeout;
+ spin_lock_init(&priv->irqlock);
+
+ v4l2_subdev_init(&priv->sd, &imx_smfc_subdev_ops);
+ v4l2_set_subdevdata(&priv->sd, priv);

+ priv->sd.internal_ops = &imx_smfc_internal_ops;
+ priv->sd.entity.ops = &imx_smfc_entity_ops;

+ /* FIXME: this the right function? */

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;
+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ /* get our group id and SMFC id */

+ priv->sd.grp_id = pdata->grp_id;

+ priv->smfc_id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+

+ return v4l2_async_register_subdev(&priv->sd);
+}
+
+static int imx_smfc_remove(struct platform_device *pdev)

+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imx_smfc_priv *priv = container_of(sd, struct imx_smfc_priv, sd);

+
+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(sd);
+
+ return 0;
+}
+

+static const struct platform_device_id imx_smfc_ids[] = {

+ { .name = "imx-ipuv3-smfc" },
+ { },
+};

+MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
+
+static struct platform_driver imx_smfc_driver = {
+ .probe = imx_smfc_probe,
+ .remove = imx_smfc_remove,
+ .id_table = imx_smfc_ids,

+ .driver = {
+ .name = "imx-ipuv3-smfc",
+ .owner = THIS_MODULE,

+ },
+};
+module_platform_driver(imx_smfc_driver);
+

+MODULE_DESCRIPTION("i.MX SMFC subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:07 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 2 +-
drivers/staging/media/imx/imx-camif.c | 1010 +++++++++++++++++++++++++++++++++
2 files changed, 1011 insertions(+), 1 deletion(-)
create mode 100644 drivers/staging/media/imx/imx-camif.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index d2a962c..fe9e992 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c

new file mode 100644
index 0000000..3cf167e
--- /dev/null
+++ b/drivers/staging/media/imx/imx-camif.c
@@ -0,0 +1,1010 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *

+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/timer.h>
+#include <linux/sched.h>
+#include <linux/slab.h>

+#include <linux/spinlock.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <linux/of_platform.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+#include <media/v4l2-event.h>
+#include <video/imx-ipu-v3.h>

+#include <media/imx.h>
+#include "imx-media.h"
+

+#define DEVICE_NAME "imx-media-camif"
+
+#define CAMIF_NUM_PADS 2
+
+#define CAMIF_DQ_TIMEOUT 5000
+
+struct camif_priv;
+
+struct camif_priv {

+ struct device *dev;

+ struct video_device vfd;
+ struct media_pipeline mp;

+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[CAMIF_NUM_PADS];
+ struct media_pad vd_pad;
+ int id;

+ int input_pad;
+ int output_pad;
+

+ struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+ const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+

+ /* dma buffer ring */
+ struct imx_media_dma_buf_ring *in_ring;

+ struct v4l2_subdev *src_sd;
+

+ /* controls inherited from subdevs */

+ struct v4l2_ctrl_handler ctrl_hdlr;
+

+ /* misc status */
+ int current_input; /* the current input */

+ v4l2_std_id current_std; /* current standard */

+ bool stop; /* streaming is stopping */

+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT SZ_64M
+
+static struct vb2_ops camif_qops;
+
+/*
+ * Video ioctls follow

+ */
+

+static int vidioc_querycap(struct file *file, void *fh,
+ struct v4l2_capability *cap)
+{
+ strncpy(cap->driver, DEVICE_NAME, sizeof(cap->driver) - 1);
+ strncpy(cap->card, DEVICE_NAME, sizeof(cap->card) - 1);
+ cap->bus_info[0] = 0;
+ cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+ cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

+
+ return 0;
+}
+

+static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_fmtdesc *f)
+{

+ const struct imx_media_pixfmt *cc;
+ u32 code;
+ int ret;
+

+ ret = imx_media_enum_format(&code, f->index, true, true);

+ if (ret)
+ return ret;

+ cc = imx_media_find_format(0, code, true, true);
+ if (!cc)
+ return -EINVAL;
+

+ f->pixelformat = cc->fourcc;

+
+ return 0;
+}
+

+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct v4l2_mbus_framefmt *outfmt;
+
+ /* user format is the same as the format from output pad */

+ outfmt = &priv->format_mbus[priv->output_pad];

+ return -EBUSY;
+ }
+

+ return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ return v4l2_subdev_call(sensor->sd, video, querystd, std);
+}
+
+static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
+{
+ struct camif_priv *priv = video_drvdata(file);
+
+ *std = priv->current_std;

+ return 0;
+}
+

+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (vb2_is_busy(&priv->buffer_queue))

+ return -EBUSY;
+

+ ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
+ if (ret < 0)

+ return ret;
+
+ priv->current_std = std;

+ return 0;
+}
+

+static int camif_enum_input(struct file *file, void *fh,
+ struct v4l2_input *input)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;

+ int index = input->index;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (index >= sensor->input.num)

+ return -EINVAL;
+

+ input->type = V4L2_INPUT_TYPE_CAMERA;
+ strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+ if (index == priv->current_input) {
+ v4l2_subdev_call(sensor->sd, video, g_input_status,
+ &input->status);
+ v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
+ } else {
+ input->status = V4L2_IN_ST_NO_SIGNAL;
+ input->std = V4L2_STD_UNKNOWN;

+ }
+
+ return 0;
+}
+

+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+ struct camif_priv *priv = video_drvdata(file);
+
+ *index = priv->current_input;

+ return 0;
+}
+

+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (index >= sensor->input.num)

+ return -EINVAL;
+
+ if (index == priv->current_input)
+ return 0;
+

+ /* select the sensor's input */
+ ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+ sensor->input.value[index], 0, 0);

+ if (!ret)
+ priv->current_input = index;
+

+ return ret;
+}
+

+static int camif_g_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *a)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)

+ return -EINVAL;
+

+ return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *a)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)

+ return -EINVAL;
+

+ return v4l2_subdev_call(sensor->sd, video, s_parm, a);

+}
+

+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+ struct v4l2_mbus_framefmt *outfmt;
+ const struct imx_media_pixfmt *outcc;
+

+ outfmt = &priv->format_mbus[priv->output_pad];

+ outcc = priv->cc[priv->output_pad];

+ return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
+}
+
+static int camif_queue_setup(struct vb2_queue *vq,
+ unsigned int *nbuffers, unsigned int *nplanes,
+ unsigned int sizes[], struct device *alloc_devs[])
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ unsigned int count = *nbuffers;
+ u32 sizeimage;
+
+ if (!priv->src_sd)

+ return -EPIPE;
+

+ if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)

+ return -EINVAL;
+

+ sizeimage = camif_get_sizeimage(priv);
+
+ count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
+ count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
+
+ while (sizeimage * count > VID_MEM_LIMIT)
+ count--;
+
+ if (count < IMX_MEDIA_MIN_RING_BUFS)

+ return -EINVAL;
+

+ *nplanes = 1;
+ *nbuffers = count;
+ sizes[0] = sizeimage;

+
+ return 0;
+}
+

+ return -EPIPE;
+
+ INIT_LIST_HEAD(&buf->list);

+
+ return 0;
+}
+

+static int camif_buf_prepare(struct vb2_buffer *vb)
+{
+ struct vb2_queue *vq = vb->vb2_queue;
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ u32 sizeimage = camif_get_sizeimage(priv);

+ int ret;
+
+ if (!priv->src_sd)
+ return -EPIPE;
+

+ if (vb2_plane_size(vb, 0) < sizeimage) {
+ v4l2_err(&priv->sd,
+ "data will not fit into plane (%lu < %lu)\n",
+ vb2_plane_size(vb, 0), (long)sizeimage);

+ return -EINVAL;
+ }
+

+ vb2_set_plane_payload(vb, 0, sizeimage);
+
+ if (!priv->in_ring) {

+ priv->in_ring = imx_media_alloc_dma_buf_ring(

+ priv->md, &priv->src_sd->entity, &priv->sd.entity,
+ sizeimage, vq->num_buffers, false);
+ if (IS_ERR(priv->in_ring)) {

+ v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ return ret;
+ }
+ }
+

+ ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);

+ if (ret)
+ goto free_ring;

+
+ return 0;
+

+free_ring:
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+ return ret;
+}
+

+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+ struct imx_media_buffer *buf = to_imx_media_vb(vb);

+ unsigned long flags;
+

+ unsigned long flags;
+ int ret;
+

+ if (vb2_is_streaming(vq))
+ return 0;
+
+ if (!priv->src_sd)
+ return -EPIPE;
+
+ if (!priv->in_ring) {

+ priv->in_ring = imx_media_alloc_dma_buf_ring(

+ priv->md, &priv->src_sd->entity, &priv->sd.entity,
+ sizeimage, vq->num_buffers, false);
+ if (IS_ERR(priv->in_ring)) {

+ v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ goto return_bufs;
+ }
+ }
+

+ ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+ &priv->mp, true);

+ if (ret) {

+ v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
+ ret);
+ goto free_ring;

+ }
+
+ priv->stop = false;

+
+ return 0;
+

+free_ring:

+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+return_bufs:
+ spin_lock_irqsave(&priv->q_lock, flags);
+ list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
+ list_del(&buf->list);
+ vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
+ }
+ spin_unlock_irqrestore(&priv->q_lock, flags);

+ return ret;
+}
+

+static void camif_stop_streaming(struct vb2_queue *vq)
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vq);
+ struct imx_media_buffer *frame;

+ unsigned long flags;
+ int ret;
+

+ if (!vb2_is_streaming(vq))
+ return;
+

+ spin_lock_irqsave(&priv->q_lock, flags);

+ priv->stop = true;

+ spin_unlock_irqrestore(&priv->q_lock, flags);
+
+ ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+ &priv->mp, false);
+ if (ret)
+ v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
+ ret);

+
+ if (priv->in_ring) {

+ v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);

+ priv->in_ring = NULL;
+ }
+

+ /* release all active buffers */
+ spin_lock_irqsave(&priv->q_lock, flags);
+ while (!list_empty(&priv->ready_q)) {
+ frame = list_entry(priv->ready_q.next,
+ struct imx_media_buffer, list);
+ list_del(&frame->list);
+ vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
+ }
+ spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static struct vb2_ops camif_qops = {
+ .queue_setup = camif_queue_setup,
+ .buf_init = camif_buf_init,
+ .buf_prepare = camif_buf_prepare,
+ .buf_queue = camif_buf_queue,
+ .wait_prepare = vb2_ops_wait_prepare,
+ .wait_finish = vb2_ops_wait_finish,
+ .start_streaming = camif_start_streaming,
+ .stop_streaming = camif_stop_streaming,

+};
+
+/*

+ * File operations
+ */
+static int camif_open(struct file *file)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ int ret;
+

+ if (mutex_lock_interruptible(&priv->mutex))
+ return -ERESTARTSYS;
+
+ ret = v4l2_fh_open(file);

+ if (ret)

+ v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
+
+ mutex_unlock(&priv->mutex);

+ return ret;
+}
+

+static int camif_release(struct file *file)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct vb2_queue *vq = &priv->buffer_queue;

+ int ret = 0;
+

+ mutex_lock(&priv->mutex);
+
+ if (file->private_data == vq->owner) {
+ vb2_queue_release(vq);

+ vq->owner = NULL;
+ }
+
+ v4l2_fh_release(file);
+ mutex_unlock(&priv->mutex);

+ return ret;
+}
+

+static const struct v4l2_file_operations camif_fops = {

+ .owner = THIS_MODULE,

+ .open = camif_open,
+ .release = camif_release,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap,
+};
+
+static struct video_device camif_videodev = {
+ .fops = &camif_fops,
+ .ioctl_ops = &camif_ioctl_ops,
+ .minor = -1,
+ .release = video_device_release,
+ .vfl_dir = VFL_DIR_RX,
+ .tvnorms = V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,

+};
+
+/*
+ * Subdev and media entity operations
+ */
+
+static void camif_new_dma_buf(struct camif_priv *priv)
+{

+ struct imx_media_dma_buf *dmabuf;
+ struct imx_media_buffer *buf;
+ enum vb2_buffer_state state;
+ struct vb2_buffer *vb;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->q_lock, flags);
+
+ if (priv->stop || list_empty(&priv->ready_q))
+ goto unlock;
+
+ dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
+ if (!dmabuf)
+ goto unlock;
+

+ vb = dmabuf->vb;
+ buf = to_imx_media_vb(vb);
+ if (list_empty(&buf->list)) {
+ dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
+ vb->index);

+ goto unlock;
+ }
+

+static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct imx_media_dma_buf_ring **ring;

+
+ switch (cmd) {
+ case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+ if (!priv->in_ring)
+ return -EINVAL;
+ ring = (struct imx_media_dma_buf_ring **)arg;
+ *ring = priv->in_ring;
+ break;
+ case IMX_MEDIA_NEW_DMA_BUF:

+ camif_new_dma_buf(priv);

+ break;
+ default:
+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int camif_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct video_device *vfd = &priv->vfd;

+ struct v4l2_subdev *remote_sd;

+ int ret = 0;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+ local->entity->name);
+

+ if (is_media_entity_v4l2_video_device(remote->entity))
+ return 0;
+
+ WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);

+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ /* reset controls to refresh with inherited from subdevs */
+ v4l2_ctrl_handler_free(vfd->ctrl_handler);
+ v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);

+
+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ ret = imx_media_inherit_controls(priv->md, vfd,
+ &priv->src_sd->entity);

+ } else {
+ priv->src_sd = NULL;
+ }

+ return ret;
+}
+

+static int camif_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ if (code->pad >= CAMIF_NUM_PADS)

+ return -EINVAL;
+
+ return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= CAMIF_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int camif_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ const struct imx_media_pixfmt *cc;
+ u32 code;
+
+ if (sdformat->pad >= CAMIF_NUM_PADS)
+ return -EINVAL;
+

+ if (vb2_is_busy(&priv->buffer_queue)) {
+ v4l2_err(&priv->sd, "%s queue busy\n", __func__);

+ return -EBUSY;
+ }
+

+ cc = imx_media_find_format(0, sdformat->format.code, true, true);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, true);
+ cc = imx_media_find_format(0, code, true, true);

+ sdformat->format.code = cc->codes[0];
+ }
+

+ /* Output pad mirrors input pad, no limitations on input pads */

+ if (sdformat->pad == priv->output_pad)

+ sdformat->format = priv->format_mbus[priv->input_pad];

+static int camif_init_pads(struct camif_priv *priv)
+{
+ struct video_device *vfd = &priv->vfd;

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i, ret;
+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {

+ v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+ imxsd->num_sink_pads, imxsd->num_src_pads);

+ return -EINVAL;
+ }
+

+ priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);

+ if (ret) {

+ v4l2_err(&priv->sd, "failed to init device node pad\n");

+ return ret;
+ }
+

+ for (i = 0; i < CAMIF_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];
+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;
+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,
+ &priv->cc[i]);

+ if (ret)
+ return ret;
+ }
+

+ return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,

+ priv->pad);
+}
+
+static int camif_registered(struct v4l2_subdev *sd)
+{

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct vb2_queue *vq = &priv->buffer_queue;
+ struct video_device *vfd = &priv->vfd;

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ int ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ vfd->v4l2_dev = sd->v4l2_dev;
+
+ ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+ if (ret) {
+ v4l2_err(sd, "Failed to register video device\n");

+ return ret;
+ }
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
+ &priv->cc[priv->output_pad]);
+ if (ret)

+ goto unreg;
+
+ *infmt = *outfmt;

+ priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
+

+ priv->current_std = V4L2_STD_UNKNOWN;
+
+ v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
+ video_device_node_name(vfd));
+
+ vfd->ctrl_handler = &priv->ctrl_hdlr;
+
+ return 0;
+unreg:
+ video_unregister_device(vfd);

+ return ret;
+}
+

+static void camif_unregistered(struct v4l2_subdev *sd)
+{

+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct video_device *vfd = &priv->vfd;
+
+ mutex_lock(&priv->mutex);
+
+ if (video_is_registered(vfd)) {
+ video_unregister_device(vfd);
+ media_entity_cleanup(&vfd->entity);
+ }
+
+ mutex_unlock(&priv->mutex);

+}
+

+static const struct v4l2_subdev_internal_ops camif_internal_ops = {
+ .registered = camif_registered,
+ .unregistered = camif_unregistered,

+};
+

+static struct v4l2_subdev_core_ops camif_core_ops = {
+ .ioctl = camif_ioctl,

+};
+

+static struct media_entity_operations camif_entity_ops = {
+ .link_setup = camif_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+ .enum_mbus_code = camif_enum_mbus_code,
+ .get_fmt = camif_get_fmt,
+ .set_fmt = camif_set_fmt,

+};
+

+static struct v4l2_subdev_ops camif_subdev_ops = {
+ .pad = &camif_pad_ops,
+ .core = &camif_core_ops,
+};
+

+static int camif_probe(struct platform_device *pdev)

+{
+ struct imx_media_internal_sd_platformdata *pdata;

+ struct camif_priv *priv;
+ struct video_device *vfd;

+ int ret;
+

+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+

+ platform_set_drvdata(pdev, priv);

+ priv->dev = &pdev->dev;
+
+ pdata = priv->dev->platform_data;
+

+ mutex_init(&priv->mutex);
+ spin_lock_init(&priv->q_lock);
+
+ v4l2_subdev_init(&priv->sd, &camif_subdev_ops);

+ v4l2_set_subdevdata(&priv->sd, priv);

+ priv->sd.internal_ops = &camif_internal_ops;
+ priv->sd.entity.ops = &camif_entity_ops;

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ /* get our group id and camif id */

+ priv->sd.grp_id = pdata->grp_id;

+ priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));

+ snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+ "%s devnode", pdata->sd_name);
+

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+

+ vfd = &priv->vfd;
+ *vfd = camif_videodev;
+ vfd->lock = &priv->mutex;
+ vfd->queue = &priv->buffer_queue;
+
+ video_set_drvdata(vfd, priv);
+

+ v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+
+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ return ret;
+}
+

+static int camif_remove(struct platform_device *pdev)
+{

+ struct camif_priv *priv =
+ (struct camif_priv *)platform_get_drvdata(pdev);
+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(&priv->sd);

+
+ return 0;
+}
+

+static const struct platform_device_id camif_ids[] = {

+ { .name = DEVICE_NAME },
+ { },
+};

+MODULE_DEVICE_TABLE(platform, camif_ids);
+
+static struct platform_driver imx_camif_driver = {
+ .probe = camif_probe,
+ .remove = camif_remove,

+ .driver = {
+ .name = DEVICE_NAME,

+ .owner = THIS_MODULE,
+ },
+};

+
+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

--
2.7.4

Steve Longerbeam

unread,

Jan 3, 2017, 4:00:07 PM1/3/17

are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same
conversions as the pre-process encode task, but in addition can
be used for hardware motion compensated deinterlacing. Frames can

come either directly from the CSI or from the SMFC entities (memory
buffers via IDMAC channels). Scaled output is limited to 1024x1024
resolution. Output frames can be routed to various sinks including
the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However

this entity sends frames to the i.MX IPU image converter which supports

image tiling, which allows scaled output up to 4096x4096 resolution.

Output frames can be routed to the camera interfaces.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/imx-ic-common.c | 113 +++
drivers/staging/media/imx/imx-ic-pp.c | 636 ++++++++++++++++
drivers/staging/media/imx/imx-ic-prpenc.c | 1037 +++++++++++++++++++++++++

drivers/staging/media/imx/imx-ic-prpvf.c | 1180 +++++++++++++++++++++++++++++
drivers/staging/media/imx/imx-ic.h | 36 +
6 files changed, 3004 insertions(+)

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 3559d7b..d2a962c 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile

@@ -1,8 +1,10 @@
imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o

obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
new file mode 100644

index 0000000..1b40558
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-common.c

@@ -0,0 +1,113 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+ [IC_TASK_ENCODER] = &imx_ic_prpenc_ops,
+ [IC_TASK_VIEWFINDER] = &imx_ic_prpvf_ops,
+ [IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+

+static int imx_ic_probe(struct platform_device *pdev)

+{
+ struct imx_media_internal_sd_platformdata *pdata;

+ struct imx_ic_priv *priv;

+ int ret;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+

+ platform_set_drvdata(pdev, &priv->sd);

+ priv->dev = &pdev->dev;
+

+ /* get our ipu_id, grp_id and IC task id */

+ pdata = priv->dev->platform_data;

+ priv->ipu_id = pdata->ipu_id;

+ switch (pdata->grp_id) {
+ case IMX_MEDIA_GRP_ID_IC_PRPENC:
+ priv->task_id = IC_TASK_ENCODER;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPVF:
+ priv->task_id = IC_TASK_VIEWFINDER;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+ priv->task_id = IC_TASK_POST_PROCESSOR;

+ break;
+ default:
+ return -EINVAL;
+ }
+

+ v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);

+ v4l2_set_subdevdata(&priv->sd, priv);

+ priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
+ priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ priv->sd.grp_id = pdata->grp_id;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+

+ ret = ic_ops[priv->task_id]->init(priv);

+ if (ret)
+ return ret;
+

+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ goto remove;
+
+ return 0;
+remove:
+ ic_ops[priv->task_id]->remove(priv);

+ return ret;
+}
+

+static int imx_ic_remove(struct platform_device *pdev)
+{

+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+
+ ic_ops[priv->task_id]->remove(priv);
+

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(sd);

+
+ return 0;
+}
+

+static const struct platform_device_id imx_ic_ids[] = {

+ { .name = "imx-ipuv3-ic" },
+ { },
+};

+MODULE_DEVICE_TABLE(platform, imx_ic_ids);
+
+static struct platform_driver imx_ic_driver = {
+ .probe = imx_ic_probe,
+ .remove = imx_ic_remove,
+ .id_table = imx_ic_ids,

+ .driver = {
+ .name = "imx-ipuv3-ic",

+ .owner = THIS_MODULE,
+ },
+};

+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-ic");
diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c

new file mode 100644
index 0000000..5ef0581
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-pp.c
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+#include <media/imx.h>
+#include <video/imx-ipu-image-convert.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {

+ struct imx_media_dev *md;

+ struct imx_ic_priv *ic_priv;
+ int pp_id;
+

+ struct ipu_soc *ipu;

+ struct ipu_image_convert_ctx *ic_ctx;
+
+ struct media_pad pad[PP_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ /* our dma buffer sink ring */
+ struct imx_media_dma_buf_ring *in_ring;
+ /* the dma buffer ring we send to sink */
+ struct imx_media_dma_buf_ring *out_ring;

+ struct ipu_image_convert_run *out_run;
+
+ struct imx_media_dma_buf *inbuf; /* last input buffer */

+
+ bool stream_on; /* streaming is on */

+ bool stop; /* streaming is stopping */

+ spinlock_t irqlock;
+
+ struct v4l2_subdev *src_sd;
+ struct v4l2_subdev *sink_sd;
+

+ struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
+ const struct imx_media_pixfmt *cc[PP_NUM_PADS];
+

+ /* motion select control */
+ struct v4l2_ctrl_handler ctrl_hdlr;

+ int rotation; /* degrees */
+ bool hflip;
+ bool vflip;
+
+ /* derived from rotation, hflip, vflip controls */
+ enum ipu_rotate_mode rot_mode;
+};
+

+static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+ return ic_priv->task_priv;
+}
+
+static void pp_convert_complete(struct ipu_image_convert_run *run,
+ void *data)
+{
+ struct pp_priv *priv = data;
+ struct imx_media_dma_buf *done;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);

+
+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))

+ spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+

+static void pp_queue_conversion(struct pp_priv *priv,
+ struct imx_media_dma_buf *inbuf)
+{
+ struct ipu_image_convert_run *run;
+ struct imx_media_dma_buf *outbuf;

+ /* get next queued buffer and make it active */
+ outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(outbuf);
+ priv->inbuf = inbuf;
+
+ run = &priv->out_run[outbuf->index];
+ run->ctx = priv->ic_ctx;
+ run->in_phys = inbuf->phys;
+ run->out_phys = outbuf->phys;
+ ipu_image_convert_queue(run);
+}
+

+static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct pp_priv *priv = sd_to_priv(sd);
+ struct imx_media_dma_buf_ring **ring;
+ struct imx_media_dma_buf *buf;

+ unsigned long flags;
+

+ switch (cmd) {
+ case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:

+ /* src asks for a buffer ring */
+ if (!priv->in_ring)

+ return -EINVAL;
+ ring = (struct imx_media_dma_buf_ring **)arg;
+ *ring = priv->in_ring;
+ break;
+ case IMX_MEDIA_NEW_DMA_BUF:

+ /* src hands us a new buffer */
+ spin_lock_irqsave(&priv->irqlock, flags);

+ if (!priv->stop &&
+ !imx_media_dma_buf_get_active(priv->out_ring)) {
+ buf = imx_media_dma_buf_dequeue(priv->in_ring);
+ if (buf)
+ pp_queue_conversion(priv, buf);
+ }
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ return ret;
+

+ imx_media_mbus_fmt_to_ipu_image(&image_in,
+ &priv->format_mbus[priv->input_pad]);
+ imx_media_mbus_fmt_to_ipu_image(&image_out,
+ &priv->format_mbus[priv->output_pad]);

+ priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+ priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
+ IC_TASK_POST_PROCESSOR,
+ &image_in, &image_out,
+ priv->rot_mode,
+ pp_convert_complete, priv);
+ if (IS_ERR(priv->ic_ctx))
+ return PTR_ERR(priv->ic_ctx);
+

+ infmt = &priv->format_mbus[priv->input_pad];

+ incc = priv->cc[priv->input_pad];

+ in_size = (infmt->width * incc->bpp * infmt->height) >> 3;

+
+ if (priv->in_ring) {

+ v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ }

+
+ priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
+ &priv->src_sd->entity,
+ &ic_priv->sd.entity,
+ in_size,
+ IMX_MEDIA_MIN_RING_BUFS,
+ true);

+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&ic_priv->sd,
+ "failed to alloc dma-buf ring\n");

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ goto out_unprep;
+ }
+

+ for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+ imx_media_dma_buf_queue(priv->in_ring, i);
+
+ priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+ sizeof(*priv->out_run), GFP_KERNEL);
+ if (!priv->out_run) {

+ v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+ ret = -ENOMEM;
+ goto out_free_ring;

+ }
+
+ priv->stop = false;
+
+ return 0;
+

+out_free_ring:

+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+out_unprep:
+ ipu_image_convert_unprepare(priv->ic_ctx);

+ return ret;
+}
+

+static void pp_stop(struct pp_priv *priv)
+{

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);

+ priv->stop = true;

+ spin_unlock_irqrestore(&priv->irqlock, flags);
+
+ ipu_image_convert_unprepare(priv->ic_ctx);
+ kfree(priv->out_run);

+
+ priv->out_ring = NULL;
+

+ /* inform sink that its sink buffer ring can now be freed */

+ v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+

+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct pp_priv *priv = sd_to_priv(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = pp_start(priv);

+ else if (!enable && priv->stream_on)

+ pp_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int pp_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ const struct imx_media_pixfmt *cc;

+ u32 fourcc;
+ int ret;
+
+ if (code->pad >= PP_NUM_PADS)
+ return -EINVAL;
+

+ ret = ipu_image_convert_enum_format(code->index, &fourcc);

+ if (ret)
+ return ret;
+

+ /* convert returned fourcc to mbus code */
+ cc = imx_media_find_format(fourcc, 0, true, true);
+ if (WARN_ON(!cc))

+ return -EINVAL;
+
+ code->code = cc->codes[0];

+ return 0;
+}
+

+static int pp_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct pp_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PP_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int pp_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct pp_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ struct ipu_image test_in, test_out;
+ u32 code;
+
+ if (sdformat->pad >= PP_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (sdformat->pad == priv->output_pad) {

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ if (sdformat->pad == priv->output_pad) {

+ *outfmt = sdformat->format;
+ imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+ } else {
+ *infmt = sdformat->format;
+ imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+ }

+ priv->cc[sdformat->pad] = cc;
+ }
+
+ return 0;
+}
+

+static int pp_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct pp_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ }

+ } else {

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ } else {
+ priv->src_sd = NULL;
+ }
+ }
+

+ return 0;
+}
+

+ break;
+ default:

+ v4l2_err(&ic_priv->sd, "Invalid control\n");

+ return -EINVAL;
+ }
+

+ ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);

+ if (ret)
+ return ret;
+

+ if (rot_mode != priv->rot_mode) {

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ struct ipu_image test_in, test_out;
+
+ /* can't change rotation mid-streaming */
+ if (priv->stream_on)

+ return -EBUSY;
+

+ /*
+ * make sure this rotation will work with current input/output
+ * formats before setting

+ */

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+ imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+ ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);

+ if (ret)
+ return ret;
+

+ priv->rot_mode = rot_mode;
+ priv->rotation = rotation;
+ priv->hflip = hflip;
+ priv->vflip = vflip;

+ }
+
+ return 0;
+}
+

+static const struct v4l2_ctrl_ops pp_ctrl_ops = {
+ .s_ctrl = pp_s_ctrl,

+};
+

+ int i, ret;
+

+ v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
+

+ for (i = 0; i < PP_NUM_CONTROLS; i++) {
+ c = &pp_std_ctrl[i];
+ v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
+ c->id, c->min, c->max, c->step, c->def);
+ }
+
+ ic_priv->sd.ctrl_handler = hdlr;
+
+ if (hdlr->error) {
+ ret = hdlr->error;
+ v4l2_ctrl_handler_free(hdlr);

+ return ret;
+ }
+

+ v4l2_ctrl_handler_setup(hdlr);

+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int pp_registered(struct v4l2_subdev *sd)
+{
+ struct pp_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < PP_NUM_PADS; i++) {

+ ret = pp_init_controls(priv);

+ if (ret)
+ return ret;
+

+ ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);

+ if (ret)
+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ return ret;
+}
+

+static struct v4l2_subdev_pad_ops pp_pad_ops = {
+ .enum_mbus_code = pp_enum_mbus_code,
+ .get_fmt = pp_get_fmt,
+ .set_fmt = pp_set_fmt,

+};
+

+static struct v4l2_subdev_video_ops pp_video_ops = {
+ .s_stream = pp_s_stream,

+};
+

+static struct v4l2_subdev_core_ops pp_core_ops = {
+ .ioctl = pp_ioctl,

+};
+

+static struct media_entity_operations pp_entity_ops = {
+ .link_setup = pp_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_ops pp_subdev_ops = {
+ .video = &pp_video_ops,
+ .pad = &pp_pad_ops,
+ .core = &pp_core_ops,

+};
+

+static struct v4l2_subdev_internal_ops pp_internal_ops = {
+ .registered = pp_registered,
+};
+
+static int pp_init(struct imx_ic_priv *ic_priv)
+{

+ struct pp_priv *priv;
+
+ priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);

+ if (!priv)
+ return -ENOMEM;
+

+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;

+ spin_lock_init(&priv->irqlock);
+

+ /* get our PP id */
+ priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;

+
+ return 0;
+}
+

+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+ struct pp_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

new file mode 100644
index 0000000..e17216b
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpenc.c
@@ -0,0 +1,1037 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *

+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.

+#include <linux/interrupt.h>

+#include <linux/spinlock.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+ struct imx_media_dev *md;

+ struct imx_ic_priv *ic_priv;
+
+ /* IPU units we require */

+ struct ipu_soc *ipu;

+ struct ipu_ic *ic_enc;
+
+ struct media_pad pad[PRPENC_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ struct ipuv3_channel *enc_ch;
+ struct ipuv3_channel *enc_rot_in_ch;
+ struct ipuv3_channel *enc_rot_out_ch;

+
+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct imx_media_dma_buf *next;
+
+ int ipu_buf_num; /* ipu double buffer index: 0-1 */
+

+ struct v4l2_subdev *src_sd;

+ struct v4l2_subdev *sink_sd;
+

+ /* the CSI id at link validate */
+ int csi_id;
+

+ /* the attached sensor at stream on */

+ struct imx_media_subdev *sensor;
+

+ struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
+ const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
+
+ struct imx_media_dma_buf rot_buf[2];
+

+ /* controls */
+ struct v4l2_ctrl_handler ctrl_hdlr;

+ int rotation; /* degrees */
+ bool hflip;
+ bool vflip;
+
+ /* derived from rotation, hflip, vflip controls */
+ enum ipu_rotate_mode rot_mode;

+
+ spinlock_t irqlock;
+
+ struct timer_list eof_timeout_timer;
+ int eof_irq;
+ int nfb4eof_irq;
+
+ bool stream_on; /* streaming is on */
+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;
+};
+

+static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+ return ic_priv->task_priv;
+}
+

+static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
+{

+ if (!IS_ERR_OR_NULL(priv->ic_enc))
+ ipu_ic_put(priv->ic_enc);

+ priv->ic_enc = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->enc_ch))
+ ipu_idmac_put(priv->enc_ch);

+ priv->enc_ch = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
+ ipu_idmac_put(priv->enc_rot_in_ch);

+ priv->enc_rot_in_ch = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
+ ipu_idmac_put(priv->enc_rot_out_ch);

+ priv->enc_rot_out_ch = NULL;
+}
+

+static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ int ret;
+

+ priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+ priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
+ if (IS_ERR(priv->ic_enc)) {

+ v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
+ ret = PTR_ERR(priv->ic_enc);

+ goto out;
+ }
+

+ }
+
+ return 0;

+out:
+ prpenc_put_ipu_resources(priv);

+ return ret;
+}
+

+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+ struct prpenc_priv *priv = dev_id;

+ struct imx_media_dma_buf *done, *next;

+ struct ipuv3_channel *channel;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);
+

+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;

+ goto unlock;
+ }
+

+ /* inform CSI of this EOF so it can monitor frame intervals */
+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+

+ channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+ priv->enc_rot_out_ch : priv->enc_ch;

+ if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))

+ ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);

+
+ /* get next queued buffer */
+ next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+

+ ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+ ipu_idmac_select_buffer(channel, priv->ipu_buf_num);

+
+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ priv->next = next;
+
+unlock:
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+ return IRQ_HANDLED;
+}
+

+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct prpenc_priv *priv = dev_id;

+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void prpenc_eof_timeout(unsigned long data)
+{

+ struct prpenc_priv *priv = (struct prpenc_priv *)data;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ v4l2_err(&ic_priv->sd, "EOF timeout\n");
+

+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void prpenc_setup_channel(struct prpenc_priv *priv,
+ struct ipuv3_channel *channel,
+ enum ipu_rotate_mode rot_mode,
+ dma_addr_t addr0, dma_addr_t addr1,
+ bool rot_swap_width_height)
+{

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ unsigned int burst_size;
+ struct ipu_image image;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (rot_swap_width_height)
+ swap(outfmt->width, outfmt->height);
+
+ ipu_cpmem_zero(channel);

+
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;
+ }
+

+ ipu_cpmem_set_burstsize(channel, burst_size);
+
+ if (rot_mode)
+ ipu_cpmem_set_rotation(channel, rot_mode);

+
+ if (outfmt->field == V4L2_FIELD_NONE &&
+ (V4L2_FIELD_HAS_BOTH(infmt->field) ||

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ struct imx_media_dma_buf *buf0, *buf1;

+ int out_size, ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ return ret;
+ }

+ ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
+ if (ret) {

+ v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
+ goto free_rot0;
+ }
+

+ ret = ipu_ic_task_init(priv->ic_enc,
+ infmt->width, infmt->height,
+ outfmt->height, outfmt->width,
+ incc->cs, outcc->cs);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+ goto free_rot1;
+ }
+
+ /* init the IC ENC-->MEM IDMAC channel */
+ prpenc_setup_channel(priv, priv->enc_ch,
+ IPU_ROTATE_NONE,
+ priv->rot_buf[0].phys,
+ priv->rot_buf[1].phys,
+ true);
+
+ /* init the MEM-->IC ENC ROT IDMAC channel */
+ prpenc_setup_channel(priv, priv->enc_rot_in_ch,
+ priv->rot_mode,
+ priv->rot_buf[0].phys,
+ priv->rot_buf[1].phys,
+ true);

+
+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+ /* enable the IC */
+ ipu_ic_enable(priv->ic_enc);

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->enc_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_ch, 1);
+ ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);

+ ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);

+
+ /* enable the channels */

+
+ return 0;
+

+free_rot1:
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:

+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);

+ return ret;
+}
+

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ struct imx_media_dma_buf *buf0, *buf1;

+ int ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ return ret;
+ }
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+ ipu_dump(priv->ipu);
+
+ ipu_ic_enable(priv->ic_enc);

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->enc_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_ch, 1);

+
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->enc_ch);
+

+ /* enable the IC ENCODE task */
+ ipu_ic_task_enable(priv->ic_enc);

+
+ return 0;
+}
+

+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");
+ return -EINVAL;
+ }
+
+ ret = prpenc_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ /* set IC to receive from CSI */
+ ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)

+ goto out_put_ipu;
+
+ priv->ipu_buf_num = 0;
+
+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ ret = prpenc_setup_rotation(priv);
+ else
+ ret = prpenc_setup_norotation(priv);

+ if (ret)
+ goto out_put_ipu;

+
+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->enc_ch,
+ IPU_IRQ_NFB4EOF);

+ ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+ prpenc_nfb4eof_interrupt, 0,

+ "imx-ic-prpenc-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_unsetup;
+ }
+

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ priv->eof_irq = ipu_idmac_channel_irq(
+ priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
+ else
+ priv->eof_irq = ipu_idmac_channel_irq(

+ priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
+

+ ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
+ prpenc_eof_interrupt, 0,

+ "imx-ic-prpenc-eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,

+
+ return 0;
+

+out_free_nfb4eof_irq:
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ prpenc_unsetup_rotation(priv);
+ else
+ prpenc_unsetup_norotation(priv);
+out_put_ipu:
+ prpenc_put_ipu_resources(priv);

+ return ret;
+}
+

+static void prpenc_stop(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ unsigned long flags;
+ int ret;
+

+ msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+ if (ret == 0)

+ v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+ devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ prpenc_unsetup_rotation(priv);
+ else
+ prpenc_unsetup_norotation(priv);
+
+ prpenc_put_ipu_resources(priv);

+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);
+ bool allow_planar;

+
+ if (code->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+
+ allow_planar = (code->pad == priv->output_pad);
+

+ return imx_media_enum_format(&code->code, code->index,

+ true, allow_planar);
+}
+

+static int prpenc_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PRPENC_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int prpenc_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ bool allow_planar;
+ u32 code;
+
+ if (sdformat->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+

+ cc = imx_media_find_format(0, sdformat->format.code,

+ true, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, false);
+ cc = imx_media_find_format(0, code, true, false);

+ sdformat->format.code = cc->codes[0];
+ }
+

+ if (sdformat->pad == priv->output_pad) {

+ sdformat->format.width = min_t(__u32,
+ sdformat->format.width,
+ MAX_W_IC);
+ sdformat->format.height = min_t(__u32,
+ sdformat->format.height,
+ MAX_H_IC);
+

+ if (sdformat->format.field != V4L2_FIELD_NONE)
+ sdformat->format.field = infmt->field;
+

+static int prpenc_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpenc_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {

+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+

+ /* this is sink pad */

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ } else {
+ priv->src_sd = NULL;

+ return 0;
+ }
+

+ switch (remote_sd->grp_id) {

+ case IMX_MEDIA_GRP_ID_CSI0:
+ priv->csi_id = 0;
+ break;
+ case IMX_MEDIA_GRP_ID_CSI1:
+ priv->csi_id = 1;

+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int prpenc_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpenc_priv *priv = ic_priv->task_priv;
+ struct v4l2_mbus_config sensor_mbus_cfg;

+ int ret;
+

+ ret = v4l2_subdev_link_validate_default(sd, link,
+ source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);

+ priv->sensor = NULL;

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+ int vc_num = 0;

+ /* see NOTE in imx-csi.c */
+#if 0

+ vc_num = imx_media_find_mipi_csi2_channel(
+ priv->md, &ic_priv->sd.entity);

+ if (vc_num < 0)
+ return vc_num;
+#endif

+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+ break;
+ default:

+ v4l2_err(&ic_priv->sd, "Invalid control\n");

+ return -EINVAL;
+ }
+

+ ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);

+ if (ret)
+ return ret;
+

+ if (rot_mode != priv->rot_mode) {
+ /* can't change rotation mid-streaming */
+ if (priv->stream_on)

+ return -EBUSY;
+

+ priv->rot_mode = rot_mode;
+ priv->rotation = rotation;
+ priv->hflip = hflip;
+ priv->vflip = vflip;

+ }
+
+ return 0;
+}
+

+static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
+ .s_ctrl = prpenc_s_ctrl,

+};
+

+ int i, ret;
+

+ v4l2_ctrl_handler_init(hdlr, PRPENC_NUM_CONTROLS);
+

+ for (i = 0; i < PRPENC_NUM_CONTROLS; i++) {
+ c = &prpenc_std_ctrl[i];
+ v4l2_ctrl_new_std(hdlr, &prpenc_ctrl_ops,
+ c->id, c->min, c->max, c->step, c->def);
+ }
+
+ ic_priv->sd.ctrl_handler = hdlr;
+
+ if (hdlr->error) {
+ ret = hdlr->error;
+ goto out_free;
+ }
+
+ v4l2_ctrl_handler_setup(hdlr);
+ return 0;
+
+out_free:
+ v4l2_ctrl_handler_free(hdlr);

+ return ret;
+}
+

+static int prpenc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct prpenc_priv *priv = sd_to_priv(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = prpenc_start(priv);

+ else if (!enable && priv->stream_on)

+ prpenc_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int prpenc_registered(struct v4l2_subdev *sd)
+{
+ struct prpenc_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < PRPENC_NUM_PADS; i++) {

+ ret = prpenc_init_controls(priv);

+ if (ret)
+ return ret;
+

+ ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);

+ if (ret)
+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ return ret;
+}
+

+static struct v4l2_subdev_pad_ops prpenc_pad_ops = {
+ .enum_mbus_code = prpenc_enum_mbus_code,
+ .get_fmt = prpenc_get_fmt,
+ .set_fmt = prpenc_set_fmt,
+ .link_validate = prpenc_link_validate,

+};
+

+static struct v4l2_subdev_video_ops prpenc_video_ops = {
+ .s_stream = prpenc_s_stream,

+};
+

+static struct media_entity_operations prpenc_entity_ops = {
+ .link_setup = prpenc_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_ops prpenc_subdev_ops = {
+ .video = &prpenc_video_ops,
+ .pad = &prpenc_pad_ops,

+};
+

+static struct v4l2_subdev_internal_ops prpenc_internal_ops = {
+ .registered = prpenc_registered,
+};
+
+static int prpenc_init(struct imx_ic_priv *ic_priv)
+{

+ struct prpenc_priv *priv;
+
+ priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);

+ if (!priv)
+ return -ENOMEM;
+

+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;
+
+ spin_lock_init(&priv->irqlock);

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = prpenc_eof_timeout;

+
+ return 0;
+}
+

+static void prpenc_remove(struct imx_ic_priv *ic_priv)
+{
+ struct prpenc_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

new file mode 100644
index 0000000..53ce006
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpvf.c
@@ -0,0 +1,1180 @@
+/*
+ * V4L2 IC Deinterlacer Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/consumer.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+ *

+ * The "direct" CSI->VDIC pipeline requires less memory bus bandwidth
+ * (just 1 channel vs. 5 channels for indirect pipeline), but it can
+ * only be used in high motion mode, and it only processes a single
+ * field (so half the original image resolution is lost).
+ */
+
+struct prpvf_priv;
+
+struct prpvf_pipeline_ops {
+ int (*setup)(struct prpvf_priv *priv);
+ void (*start)(struct prpvf_priv *priv);
+ void (*stop)(struct prpvf_priv *priv);
+ void (*disable)(struct prpvf_priv *priv);
+};
+
+#define PRPVF_NUM_PADS 2
+
+#define MAX_W_IC 1024
+#define MAX_H_IC 1024
+#define MAX_W_VDIC 968
+#define MAX_H_VDIC 2048
+
+struct prpvf_priv {

+ struct imx_media_dev *md;

+ struct imx_ic_priv *ic_priv;
+
+ /* IPU units we require */

+ struct ipu_soc *ipu;

+ struct ipu_ic *ic_vf;
+ struct ipu_vdi *vdi;
+
+ struct media_pad pad[PRPVF_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ /* our dma buffer sink ring */
+ struct imx_media_dma_buf_ring *in_ring;

+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+

+
+ /*

+ * translated field type, input line stride, and field size
+ * for indirect path
+ */
+ u32 fieldtype;
+ u32 in_stride;
+ u32 field_size;
+

+ struct v4l2_subdev *src_sd;
+ /* the sink that will receive the progressive out buffers */

+ struct v4l2_subdev *sink_sd;
+

+ /* the attached CSI at stream on */
+ struct v4l2_subdev *csi_sd;

+
+ /* the attached sensor at stream on */

+ struct imx_media_subdev *sensor;
+

+ /* the video standard from sensor at time of streamon */
+ v4l2_std_id std;

+ struct v4l2_mbus_framefmt format_mbus[PRPVF_NUM_PADS];
+ const struct imx_media_pixfmt *cc[PRPVF_NUM_PADS];
+
+ bool csi_direct; /* using direct CSI->VDIC->IC pipeline */
+

+ /* motion select control */
+ struct v4l2_ctrl_handler ctrl_hdlr;
+ enum ipu_motion_sel motion;
+

+ struct timer_list eof_timeout_timer;
+

+ int nfb4eof_irq; /* CSI or PRPVF channel NFB4EOF IRQ */
+ int out_eof_irq; /* PRPVF channel EOF IRQ */
+ spinlock_t irqlock;

+
+ bool stream_on; /* streaming is on */
+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;
+};
+

+static inline struct prpvf_priv *sd_to_priv(struct v4l2_subdev *sd)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+ return ic_priv->task_priv;
+}
+

+static void prpvf_put_ipu_resources(struct prpvf_priv *priv)
+{

+ if (!IS_ERR_OR_NULL(priv->ic_vf))
+ ipu_ic_put(priv->ic_vf);

+ priv->ic_vf = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->vdi_in_ch_p))
+ ipu_idmac_put(priv->vdi_in_ch_p);

+ priv->vdi_in_ch_p = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->vdi_in_ch))
+ ipu_idmac_put(priv->vdi_in_ch);

+ priv->vdi_in_ch = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->vdi_in_ch_n))
+ ipu_idmac_put(priv->vdi_in_ch_n);

+ priv->vdi_in_ch_n = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->prpvf_out_ch))
+ ipu_idmac_put(priv->prpvf_out_ch);

+ priv->prpvf_out_ch = NULL;
+

+ if (!IS_ERR_OR_NULL(priv->vdi))
+ ipu_vdi_put(priv->vdi);

+ priv->vdi = NULL;
+}
+

+static int prpvf_get_ipu_resources(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ int ret, err_chan;

+ priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+ priv->ic_vf = ipu_ic_get(priv->ipu, IC_TASK_VIEWFINDER);
+ if (IS_ERR(priv->ic_vf)) {

+ v4l2_err(&ic_priv->sd, "failed to get IC VF\n");
+ ret = PTR_ERR(priv->ic_vf);

+ goto out;
+ }
+

+ priv->vdi = ipu_vdi_get(priv->ipu);
+ if (IS_ERR(priv->vdi)) {

+ v4l2_err(&ic_priv->sd, "failed to get VDIC\n");
+ ret = PTR_ERR(priv->vdi);

+ goto out;
+ }
+

+ }
+
+ return 0;
+

+out_err_chan:
+ v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n", err_chan);
+out:
+ prpvf_put_ipu_resources(priv);

+ return ret;
+}
+

+ break;
+ default:

+ /* assume V4L2_FIELD_INTERLACED_TB */
+ prev_phys = last->phys;
+ curr_phys = curr->phys + priv->in_stride;
+ next_phys = curr->phys;
+ break;
+ }
+
+ ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
+ ipu_cpmem_set_buffer(priv->vdi_in_ch, 0, curr_phys);
+ ipu_cpmem_set_buffer(priv->vdi_in_ch_n, 0, next_phys);
+
+ ipu_idmac_select_buffer(priv->vdi_in_ch_p, 0);
+ ipu_idmac_select_buffer(priv->vdi_in_ch, 0);
+ ipu_idmac_select_buffer(priv->vdi_in_ch_n, 0);
+}
+

+static void prepare_prpvf_out_buffer(struct prpvf_priv *priv)
+{
+ struct imx_media_dma_buf *buf;
+

+ /* get next buffer to prepare */
+ buf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ if (!priv->csi_direct) {
+ /*
+ * indirect does not use double-buffering, so this
+ * buffer is now the active one
+ */
+ imx_media_dma_buf_set_active(buf);
+ } else {
+ priv->next_out_buf = buf;
+ }
+
+ ipu_cpmem_set_buffer(priv->prpvf_out_ch, priv->ipu_buf_num, buf->phys);
+ ipu_idmac_select_buffer(priv->prpvf_out_ch, priv->ipu_buf_num);
+}
+
+/* prpvf_out_ch EOF interrupt (progressive frame ready) */

+static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
+{
+ struct prpvf_priv *priv = dev_id;
+ struct imx_media_dma_buf *done;

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);
+

+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;

+ goto unlock;
+ }
+

+ if (priv->csi_direct) {

+ /* inform CSI of this EOF so it can monitor frame intervals */

+ /* FIXME: frames are coming in twice as fast in direct path! */

+
+ /* bump the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+

+ if (priv->csi_direct) {
+ prepare_prpvf_out_buffer(priv);

+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ }

+
+unlock:
+ spin_unlock_irqrestore(&priv->irqlock, flags);
+ return IRQ_HANDLED;
+}
+

+static long prpvf_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);
+ struct imx_media_dma_buf_ring **ring;
+ struct imx_media_dma_buf *buf;

+ unsigned long flags;
+

+static irqreturn_t nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct prpvf_priv *priv = dev_id;

+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void prpvf_eof_timeout(unsigned long data)
+{

+ struct prpvf_priv *priv = (struct prpvf_priv *)data;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ v4l2_err(&ic_priv->sd, "EOF timeout\n");
+

+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void setup_vdi_channel(struct prpvf_priv *priv,
+ struct ipuv3_channel *channel,
+ dma_addr_t phys0, dma_addr_t phys1,
+ bool out_chan)
+{

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ unsigned int burst_size;
+ struct ipu_image image;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;

+static int prpvf_setup_direct(struct prpvf_priv *priv)
+{

+ struct imx_media_dma_buf *buf0, *buf1;
+
+ /* set VDIC to receive from CSI for direct path */
+ ipu_fsu_link(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
+ IPUV3_CHANNEL_CSI_VDI_PREV);

+
+ priv->ipu_buf_num = 0;
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);

+ priv->next_out_buf = buf1;
+
+ /* init the prpvf out channel */
+ setup_vdi_channel(priv, priv->prpvf_out_ch,
+ buf0->phys, buf1->phys, true);

+
+ return 0;
+}
+

+static void prpvf_start_direct(struct prpvf_priv *priv)

+{
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->prpvf_out_ch, 0);
+ ipu_idmac_select_buffer(priv->prpvf_out_ch, 1);

+
+ /* enable the channels */

+ infmt = &priv->format_mbus[priv->input_pad];

+ incc = priv->cc[priv->input_pad];
+

+
+ priv->ipu_buf_num = 0;

+
+ if (priv->in_ring) {

+ v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ }

+ priv->in_ring = imx_media_alloc_dma_buf_ring(
+ priv->md, &priv->src_sd->entity,

+ &ic_priv->sd.entity,
+ in_size, IMX_MEDIA_MIN_RING_BUFS_PRPVF, true);

+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&ic_priv->sd, "failed to alloc dma-buf ring\n");

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;
+ return ret;
+ }
+

+ for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS_PRPVF; i++)
+ imx_media_dma_buf_queue(priv->in_ring, i);
+
+ priv->last_in_buf = NULL;

+ priv->curr_in_buf = NULL;
+

+ /* translate V4L2_FIELD_ALTERNATE to SEQ_TB or SEQ_BT */
+ priv->fieldtype = infmt->field;
+ if (infmt->field == V4L2_FIELD_ALTERNATE)
+ priv->fieldtype = (priv->std & V4L2_STD_525_60) ?
+ V4L2_FIELD_SEQ_TB : V4L2_FIELD_SEQ_BT;
+
+ /* init the vdi-in channels */
+ setup_vdi_channel(priv, priv->vdi_in_ch_p, 0, 0, false);
+ setup_vdi_channel(priv, priv->vdi_in_ch, 0, 0, false);
+ setup_vdi_channel(priv, priv->vdi_in_ch_n, 0, 0, false);
+
+ /* init the prpvf out channel */
+ setup_vdi_channel(priv, priv->prpvf_out_ch, 0, 0, true);

+
+ return 0;
+}
+

+static void prpvf_start_indirect(struct prpvf_priv *priv)

+{
+ /* enable the channels */

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");
+ return -EINVAL;
+ }
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ priv->ops = priv->csi_direct ? &direct_ops : &indirect_ops;
+
+ ret = prpvf_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ /* set IC to receive from VDIC */
+ ipu_set_ic_src_mux(priv->ipu, 0, true);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ goto out_put_ipu;

+
+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ "imx-ic-prpvf-out-eof", priv);
+ if (ret) {

+ v4l2_err(&ic_priv->sd,
+ "Error registering out eof irq: %d\n", ret);
+ goto out_put_ipu;
+ }
+
+ /* request NFB4EOF irq */

+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->prpvf_out_ch,
+ IPU_IRQ_NFB4EOF);

+ ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+ nfb4eof_interrupt, 0,

+ "imx-ic-prpvf-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);

+ goto out_free_eof_irq;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_std, &priv->std);

+ if (ret)

+ goto out_free_nfb4eof_irq;
+
+ /* init the VDIC */
+ ipu_vdi_setup(priv->vdi, infmt->code,
+ infmt->width, infmt->height);
+ ipu_vdi_set_field_order(priv->vdi, priv->std, infmt->field);
+ ipu_vdi_set_motion(priv->vdi, priv->motion);
+
+ ret = ipu_ic_task_init(priv->ic_vf,
+ infmt->width, infmt->height,
+ outfmt->width, outfmt->height,
+ incc->cs, outcc->cs);
+ if (ret) {

+ v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ ret = priv->ops->setup(priv);
+ if (ret)

+ goto out_free_nfb4eof_irq;
+
+ ipu_vdi_enable(priv->vdi);
+ ipu_ic_enable(priv->ic_vf);
+
+ priv->ops->start(priv);

+ /* enable the IC VF task */
+ ipu_ic_task_enable(priv->ic_vf);

+
+ /* start the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+ return ret;
+}
+

+static void prpvf_stop(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ unsigned long flags;
+ int ret;
+

+ v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+

+ ipu_ic_task_disable(priv->ic_vf);
+ priv->ops->stop(priv);
+ ipu_ic_disable(priv->ic_vf);
+ ipu_vdi_disable(priv->vdi);
+ priv->ops->disable(priv);
+
+ devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+ devm_free_irq(ic_priv->dev, priv->out_eof_irq, priv);
+ prpvf_put_ipu_resources(priv);

+ return -EBUSY;
+ priv->motion = motion;
+ }
+ break;
+ default:

+ v4l2_err(&ic_priv->sd, "Invalid control\n");

+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static const struct v4l2_ctrl_ops prpvf_ctrl_ops = {
+ .s_ctrl = prpvf_s_ctrl,

+};
+

+static const struct v4l2_ctrl_config prpvf_custom_ctrl[] = {
+ {
+ .ops = &prpvf_ctrl_ops,
+ .id = V4L2_CID_IMX_MOTION,
+ .name = "Motion Compensation",
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .def = MOTION_NONE,
+ .min = MOTION_NONE,
+ .max = HIGH_MOTION,
+ .step = 1,
+ },
+};
+
+#define PRPVF_NUM_CONTROLS ARRAY_SIZE(prpvf_custom_ctrl)
+
+static int prpvf_init_controls(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+ const struct v4l2_ctrl_config *c;

+ int i, ret;
+

+ v4l2_ctrl_handler_free(hdlr);
+ v4l2_ctrl_handler_init(hdlr, PRPVF_NUM_CONTROLS);
+

+ for (i = 0; i < PRPVF_NUM_CONTROLS; i++) {
+ c = &prpvf_custom_ctrl[i];
+ v4l2_ctrl_new_custom(hdlr, c, NULL);
+ }
+
+ ic_priv->sd.ctrl_handler = hdlr;
+
+ if (hdlr->error) {
+ ret = hdlr->error;
+ goto out_free;
+ }
+
+ v4l2_ctrl_handler_setup(hdlr);
+ return 0;
+
+out_free:
+ v4l2_ctrl_handler_free(hdlr);

+ return ret;
+}
+

+static int prpvf_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = prpvf_start(priv);

+ else if (!enable && priv->stream_on)

+ prpvf_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int prpvf_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);
+ bool allow_planar, allow_rgb;

+
+ if (code->pad >= PRPVF_NUM_PADS)
+ return -EINVAL;
+
+ allow_planar = (code->pad == priv->output_pad);
+ allow_rgb = allow_planar;

+
+ return imx_media_enum_format(&code->code, code->index,

+ allow_rgb, allow_planar);
+}
+

+static int prpvf_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PRPVF_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int prpvf_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ bool allow_planar, allow_rgb;
+ u32 code;
+
+ if (sdformat->pad >= PRPVF_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+ allow_rgb = allow_planar;

+
+ cc = imx_media_find_format(0, sdformat->format.code,

+ allow_rgb, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, false, false);
+ cc = imx_media_find_format(0, code, false, false);

+ sdformat->format.code = cc->codes[0];
+ }
+

+ if (sdformat->pad == priv->output_pad) {

+static int prpvf_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpvf_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+ int ret;
+
+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;

+ priv->sink_sd = remote_sd;
+ } else {

+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+

+ /* this is sink pad */

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+

+ priv->csi_direct = ((priv->src_sd->grp_id &

+ IMX_MEDIA_GRP_ID_CSI) != 0);
+
+ ret = prpvf_init_controls(priv);

+ if (ret)
+ return ret;

+ } else {
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ priv->src_sd = NULL;
+ }
+

+ return 0;
+}
+

+static int prpvf_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpvf_priv *priv = ic_priv->task_priv;

+ struct v4l2_mbus_config sensor_mbus_cfg;
+ struct imx_media_subdev *csi;
+ int ret;
+

+ ret = v4l2_subdev_link_validate_default(sd, link,
+ source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);

+ priv->sensor = NULL;

+ return ret;
+ }
+

+ if (!priv->csi_direct) {
+ csi = imx_media_find_pipeline_subdev(
+ priv->md, &ic_priv->sd.entity, IMX_MEDIA_GRP_ID_CSI);
+ if (IS_ERR(csi)) {

+ v4l2_err(&ic_priv->sd, "no CSI attached\n");
+ ret = PTR_ERR(csi);

+ return ret;
+ }
+

+ priv->csi_sd = csi->sd;

+ return 0;
+ }
+

+ priv->csi_sd = priv->src_sd;
+
+ if (priv->motion != HIGH_MOTION) {
+ v4l2_err(&ic_priv->sd,
+ "direct CSI pipeline requires HIGH_MOTION\n");

+ return -EINVAL;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+ int vc_num = 0;

+ /* see NOTE in imx-csi.c */
+#if 0

+ vc_num = imx_media_find_mipi_csi2_channel(
+ priv->md, &ic_priv->sd.entity);

+ if (vc_num < 0)
+ return vc_num;
+#endif

+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int prpvf_registered(struct v4l2_subdev *sd)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < PRPVF_NUM_PADS; i++) {

+ return media_entity_pads_init(&sd->entity, PRPVF_NUM_PADS, priv->pad);
+}
+

+static struct v4l2_subdev_pad_ops prpvf_pad_ops = {
+ .enum_mbus_code = prpvf_enum_mbus_code,
+ .get_fmt = prpvf_get_fmt,
+ .set_fmt = prpvf_set_fmt,
+ .link_validate = prpvf_link_validate,

+};
+

+static struct v4l2_subdev_video_ops prpvf_video_ops = {
+ .s_stream = prpvf_s_stream,

+};
+

+static struct v4l2_subdev_core_ops prpvf_core_ops = {
+ .ioctl = prpvf_ioctl,

+};
+

+static struct media_entity_operations prpvf_entity_ops = {
+ .link_setup = prpvf_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_ops prpvf_subdev_ops = {
+ .video = &prpvf_video_ops,
+ .pad = &prpvf_pad_ops,
+ .core = &prpvf_core_ops,

+};
+

+static struct v4l2_subdev_internal_ops prpvf_internal_ops = {
+ .registered = prpvf_registered,
+};
+
+static int prpvf_init(struct imx_ic_priv *ic_priv)
+{

+ struct prpvf_priv *priv;
+
+ priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);

+ if (!priv)
+ return -ENOMEM;
+

+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;
+
+ spin_lock_init(&priv->irqlock);

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = prpvf_eof_timeout;

+
+ return 0;
+}
+

+static void prpvf_remove(struct imx_ic_priv *ic_priv)
+{
+ struct prpvf_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

new file mode 100644

index 0000000..9aed5f5
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic.h

@@ -0,0 +1,36 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.

+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+
+struct imx_ic_priv {

+ struct device *dev;
+ struct v4l2_subdev sd;
+ int ipu_id;

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:05 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:05 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6q.dtsi | 4 ++++
arch/arm/boot/dts/imx6qdl.dtsi | 8 ++++++++
2 files changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index 56a314f..2fbe0b3 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi

@@ -454,3 +454,7 @@
&vpu {
compatible = "fsl,imx6q-vpu", "cnm,coda960";
};
+
+&media0 {
+ ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 89218a4..12ae045 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi

@@ -1296,5 +1296,13 @@
};
};
};
+
+ media0: media@0 {
+ compatible = "fsl,imx-media";
+ ports = <&ipu1_csi0>, <&ipu1_csi1>;

+ #address-cells = <1>;
+ #size-cells = <0>;
+ status = "okay";

+ };
};
};
--
2.7.4

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:05 PM1/3/17

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
clocks.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 53e6e63..7b546e3 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi

@@ -1125,7 +1125,14 @@
};

mipi_csi: mipi@021dc000 {

+ compatible = "fsl,imx-mipi-csi2";
reg = <0x021dc000 0x4000>;

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:05 PM1/3/17

This is a media entity subdevice for the i.MX Camera
Serial Interface module.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index ef9f11b..133672a 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \

obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c

new file mode 100644

index 0000000..975eafb
--- /dev/null
+++ b/drivers/staging/media/imx/imx-csi.c

@@ -0,0 +1,638 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC

+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>

+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-ctrls.h>

+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+#define CSI_NUM_PADS 2
+
+struct csi_priv {

+ struct device *dev;
+ struct ipu_soc *ipu;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[CSI_NUM_PADS];
+ struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];

+ struct v4l2_mbus_config sensor_mbus_cfg;

+ struct v4l2_rect crop;
+ struct ipu_csi *csi;
+ int csi_id;

+ int input_pad;
+ int output_pad;

+ bool power_on; /* power is on */

+ bool stream_on; /* streaming is on */
+

+ /* the sink for the captured frames */
+ struct v4l2_subdev *sink_sd;
+ enum ipu_csi_dest dest;

+ struct v4l2_subdev *src_sd;
+

+ struct v4l2_ctrl_handler ctrl_hdlr;
+ struct imx_media_fim *fim;

+
+ /* the attached sensor at stream on */
+ struct imx_media_subdev *sensor;
+};
+

+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{

+ return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */

+static int csi_setup(struct csi_priv *priv)
+{

+ struct v4l2_mbus_framefmt infmt;
+
+ ipu_csi_set_window(priv->csi, &priv->crop);

+
+ /*

+ * the ipu-csi doesn't understand ALTERNATE, but it only
+ * needs to know whether the stream is interlaced, so set
+ * to INTERLACED if infmt field is ALTERNATE.

+ */

+ infmt = priv->format_mbus[priv->input_pad];
+ if (infmt.field == V4L2_FIELD_ALTERNATE)
+ infmt.field = V4L2_FIELD_INTERLACED;
+
+ ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
+
+ ipu_csi_set_dest(priv->csi, priv->dest);
+
+ ipu_csi_dump(priv->csi);

+
+ return 0;
+}
+

+static int csi_start(struct csi_priv *priv)
+{

+ int ret;
+
+ if (!priv->sensor) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return -EINVAL;
+ }
+

+ ret = csi_setup(priv);

+ if (ret)
+ return ret;
+

+ /* start the frame interval monitor */
+ ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);

+ if (ret)
+ return ret;
+

+ ret = ipu_csi_enable(priv->csi);
+ if (ret) {
+ v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);

+ return ret;
+ }
+

+ return 0;
+}
+

+static void csi_stop(struct csi_priv *priv)
+{
+ /* stop the frame interval monitor */
+ imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+ ipu_csi_disable(priv->csi);
+}
+

+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+
+ if (!priv->src_sd || !priv->sink_sd)
+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = csi_start(priv);

+ else if (!enable && priv->stream_on)

+ csi_stop(priv);

+
+ if (!ret)
+ priv->stream_on = enable;
+ return ret;
+}
+

+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+

+ v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+

+ if (on != priv->power_on)
+ ret = imx_media_fim_set_power(priv->fim, on);

+
+ if (!ret)
+ priv->power_on = on;

+ return ret;
+}
+

+static int csi_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SINK) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;
+ return 0;
+ }
+

+ return 0;
+ }
+

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;

+ return 0;
+ }
+

+ /* set CSI destination */
+ switch (remote_sd->grp_id) {

+ case IMX_MEDIA_GRP_ID_SMFC0:
+ case IMX_MEDIA_GRP_ID_SMFC1:
+ case IMX_MEDIA_GRP_ID_SMFC2:
+ case IMX_MEDIA_GRP_ID_SMFC3:
+ priv->dest = IPU_CSI_DEST_IDMAC;

+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPVF:

+ priv->dest = IPU_CSI_DEST_VDIC;
+ break;

+ case IMX_MEDIA_GRP_ID_IC_PRPENC:
+ priv->dest = IPU_CSI_DEST_IC;

+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int csi_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ bool is_csi2;

+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);
+ priv->sensor = NULL;
+ return ret;
+ }
+
+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+ if (is_csi2) {

+ int vc_num = 0;
+ /*

+ * NOTE! It seems the virtual channels from the mipi csi-2
+ * receiver are used only for routing by the video mux's,
+ * or for hard-wired routing to the CSI's. Once the stream
+ * enters the CSI's however, they are treated internally
+ * in the IPU as virtual channel 0.

+ */
+#if 0
+ vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+ &priv->sd.entity);

+ if (vc_num < 0)
+ return vc_num;
+#endif

+ ipu_csi_set_mipi_datatype(priv->csi, vc_num,

+ &priv->format_mbus[priv->input_pad]);
+ }

+
+ /* select either parallel or MIPI-CSI2 as input to CSI */
+ ipu_set_csi_src_mux(priv->ipu, priv->csi_id, is_csi2);

+
+ return 0;
+}
+

+static int csi_try_crop(struct csi_priv *priv, struct v4l2_rect *crop)
+{

+ struct v4l2_mbus_framefmt *infmt;

+ struct imx_media_subdev *sensor;
+ v4l2_std_id std;

+ int ret;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");
+ return PTR_ERR(sensor);
+ }
+
+ ret = v4l2_subdev_call(sensor->sd, video, g_mbus_config,
+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+

+
+ /*

+ * FIXME: not sure why yet, but on interlaced bt.656,
+ * changing the vertical cropping causes loss of vertical
+ * sync, so fix it to NTSC/PAL active lines. NTSC contains
+ * 2 extra lines of active video that need to be cropped.

+ */
+ if (priv->sensor_mbus_cfg.type == V4L2_MBUS_BT656) {
+ ret = v4l2_subdev_call(sensor->sd, video, g_std, &std);

+ if (ret)
+ return ret;

+ }
+
+ return 0;
+}
+

+static int csi_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= CSI_NUM_PADS)

+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];
+
+ return 0;
+}
+

+static int csi_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ struct v4l2_rect crop;
+ int ret;
+
+ if (sdformat->pad >= CSI_NUM_PADS)

+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (sdformat->pad == priv->output_pad) {

+ sdformat->format.code = infmt->code;

+ sdformat->format.field = infmt->field;

+ crop.left = priv->crop.left;
+ crop.top = priv->crop.top;
+ crop.width = sdformat->format.width;
+ crop.height = sdformat->format.height;
+ ret = csi_try_crop(priv, &crop);

+ if (ret)
+ return ret;

+ sdformat->format.width = crop.width;
+ sdformat->format.height = crop.height;

+ }
+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;

+ /* Update the crop window if this is output pad */

+ if (sdformat->pad == priv->output_pad)

+ priv->crop = crop;

+ }
+
+ return 0;
+}
+

+static int csi_get_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,

+ struct v4l2_subdev_selection *sel)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt;
+

+ if (sel->pad != priv->output_pad)

+ return -EINVAL;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+

+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+

+static int csi_set_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,

+ struct v4l2_subdev_selection *sel)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ struct v4l2_mbus_framefmt *outfmt;
+ int ret;
+

+ if (sel->pad != priv->output_pad ||
+ sel->target != V4L2_SEL_TGT_CROP)

+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+ cfg->try_crop = sel->r;

+ return 0;
+ }
+

+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ ret = csi_try_crop(priv, &sel->r);

+ if (ret)
+ return ret;
+

+ }
+
+ return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int csi_registered(struct v4l2_subdev *sd)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+
+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+ if (IS_ERR(imxsd)) {

+ ret = PTR_ERR(imxsd);
+ goto put_csi;
+ }

+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {

+ ret = -EINVAL;
+ goto put_csi;
+ }
+
+ for (i = 0; i < CSI_NUM_PADS; i++) {

+ NULL);
+ if (ret)

+ goto put_csi;
+ }
+
+ priv->fim = imx_media_fim_init(&priv->sd);
+ if (IS_ERR(priv->fim)) {
+ ret = PTR_ERR(priv->fim);
+ goto put_csi;

+ }
+
+ ret = media_entity_pads_init(&sd->entity, CSI_NUM_PADS, priv->pad);
+ if (ret)

+ goto free_fim;
+
+ return 0;
+free_fim:
+ imx_media_fim_free(priv->fim);
+put_csi:
+ ipu_csi_put(priv->csi);

+ return ret;
+}
+

+static struct media_entity_operations csi_entity_ops = {
+ .link_setup = csi_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_core_ops csi_core_ops = {
+ .s_power = csi_s_power,
+ .interrupt_service_routine = csi_eof_isr,

+};
+

+static struct v4l2_subdev_video_ops csi_video_ops = {
+ .s_stream = csi_s_stream,

+};
+

+static struct v4l2_subdev_pad_ops csi_pad_ops = {
+ .get_fmt = csi_get_fmt,
+ .set_fmt = csi_set_fmt,
+ .get_selection = csi_get_selection,
+ .set_selection = csi_set_selection,
+ .link_validate = csi_link_validate,

+};
+

+static struct v4l2_subdev_ops csi_subdev_ops = {
+ .core = &csi_core_ops,
+ .video = &csi_video_ops,
+ .pad = &csi_pad_ops,

+};
+

+static struct v4l2_subdev_internal_ops csi_internal_ops = {
+ .registered = csi_registered,
+};
+

+static int imx_csi_probe(struct platform_device *pdev)
+{
+ struct ipu_client_platformdata *pdata;
+ struct csi_priv *priv;

+ int ret;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, &priv->sd);
+ priv->dev = &pdev->dev;
+

+ /* get parent IPU */
+ priv->ipu = dev_get_drvdata(priv->dev->parent);

+
+ /* get our CSI id */

+ pdata = priv->dev->platform_data;

+ priv->csi_id = pdata->csi;
+
+ v4l2_subdev_init(&priv->sd, &csi_subdev_ops);

+ v4l2_set_subdevdata(&priv->sd, priv);

+ priv->sd.internal_ops = &csi_internal_ops;
+ priv->sd.entity.ops = &csi_entity_ops;

+ /* FIXME: this the right function? */

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;

+ priv->sd.grp_id = priv->csi_id ?
+ IMX_MEDIA_GRP_ID_CSI1 : IMX_MEDIA_GRP_ID_CSI0;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;
+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ imx_media_grp_id_to_sd_name(priv->sd.name, sizeof(priv->sd.name),
+ priv->sd.grp_id, ipu_get_num(priv->ipu));

+
+ v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);

+ priv->sd.ctrl_handler = &priv->ctrl_hdlr;

+
+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ goto free_ctrls;
+
+ return 0;
+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ return ret;
+}
+

+static int imx_csi_remove(struct platform_device *pdev)

+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct csi_priv *priv = sd_to_dev(sd);
+
+ imx_media_fim_free(priv->fim);

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);
+ v4l2_device_unregister_subdev(sd);
+

+ if (!IS_ERR_OR_NULL(priv->csi))
+ ipu_csi_put(priv->csi);

+
+ return 0;
+}
+

+static const struct platform_device_id imx_csi_ids[] = {

+ { .name = "imx-ipuv3-csi" },
+ { },
+};

+MODULE_DEVICE_TABLE(platform, imx_csi_ids);
+
+static struct platform_driver imx_csi_driver = {
+ .probe = imx_csi_probe,
+ .remove = imx_csi_remove,
+ .id_table = imx_csi_ids,

+ .driver = {
+ .name = "imx-ipuv3-csi",

+ .owner = THIS_MODULE,
+ },
+};

+module_platform_driver(imx_csi_driver);
+
+MODULE_DESCRIPTION("i.MX CSI subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-csi");
--
2.7.4

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:05 PM1/3/17

In version 2 (no functional changes):

- removed patch "gpio: pca953x: Add optional reset gpio control", it
has been submitted separately.
- fixed some whitespace errors.
- added a few missing Signed-off-by's.

Philipp Zabel (2):
ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
connections
media: imx: Add video switch subdev driver

Steve Longerbeam (17):
ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
ARM: dts: imx6qdl: add media device
ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabreauto: create i2cmux for i2c3
ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
media: Add i.MX media core driver
media: imx: Add CSI subdev driver
media: imx: Add SMFC subdev driver
media: imx: Add IC subdev drivers
media: imx: Add Camera Interface subdev driver
media: imx: Add MIPI CSI-2 Receiver subdev driver
media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
media: imx: Add Parallel OV5642 sensor subdev driver
ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

Documentation/devicetree/bindings/media/imx.txt | 205 +
Documentation/media/v4l-drivers/imx.rst | 430 ++
arch/arm/boot/dts/imx6dl-sabrelite.dts | 5 +
arch/arm/boot/dts/imx6dl-sabresd.dts | 5 +
arch/arm/boot/dts/imx6dl.dtsi | 183 +
arch/arm/boot/dts/imx6q-sabrelite.dts | 6 +
arch/arm/boot/dts/imx6q-sabresd.dts | 5 +
arch/arm/boot/dts/imx6q.dtsi | 123 +
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 142 +-
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 122 +-
arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 +-
arch/arm/boot/dts/imx6qdl.dtsi | 25 +-
arch/arm/configs/imx_v6_v7_defconfig | 10 +-
drivers/staging/media/Kconfig | 2 +
drivers/staging/media/Makefile | 1 +
drivers/staging/media/imx/Kconfig | 36 +
drivers/staging/media/imx/Makefile | 16 +
drivers/staging/media/imx/TODO | 18 +
drivers/staging/media/imx/imx-camif.c | 1010 +++++
drivers/staging/media/imx/imx-csi.c | 638 +++
drivers/staging/media/imx/imx-ic-common.c | 113 +
drivers/staging/media/imx/imx-ic-pp.c | 636 +++
drivers/staging/media/imx/imx-ic-prpenc.c | 1037 +++++
drivers/staging/media/imx/imx-ic-prpvf.c | 1180 ++++++
drivers/staging/media/imx/imx-ic.h | 36 +
drivers/staging/media/imx/imx-media-common.c | 985 +++++
drivers/staging/media/imx/imx-media-dev.c | 479 +++
drivers/staging/media/imx/imx-media-fim.c | 509 +++
drivers/staging/media/imx/imx-media-internal-sd.c | 457 +++
drivers/staging/media/imx/imx-media-of.c | 291 ++
drivers/staging/media/imx/imx-media-of.h | 25 +
drivers/staging/media/imx/imx-media.h | 299 ++
drivers/staging/media/imx/imx-mipi-csi2.c | 509 +++
drivers/staging/media/imx/imx-smfc.c | 739 ++++
drivers/staging/media/imx/imx-video-switch.c | 351 ++
drivers/staging/media/imx/ov5640-mipi.c | 2349 +++++++++++
drivers/staging/media/imx/ov5642.c | 4364 +++++++++++++++++++++
include/media/imx.h | 15 +
include/uapi/Kbuild | 1 +
include/uapi/linux/v4l2-controls.h | 4 +
include/uapi/media/Kbuild | 2 +
include/uapi/media/imx.h | 30 +
42 files changed, 17479 insertions(+), 28 deletions(-)
create mode 100644 Documentation/devicetree/bindings/media/imx.txt
create mode 100644 Documentation/media/v4l-drivers/imx.rst
create mode 100644 drivers/staging/media/imx/Kconfig
create mode 100644 drivers/staging/media/imx/Makefile
create mode 100644 drivers/staging/media/imx/TODO
create mode 100644 drivers/staging/media/imx/imx-camif.c
create mode 100644 drivers/staging/media/imx/imx-csi.c

create mode 100644 drivers/staging/media/imx/imx-media-common.c
create mode 100644 drivers/staging/media/imx/imx-media-dev.c
create mode 100644 drivers/staging/media/imx/imx-media-fim.c
create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
create mode 100644 drivers/staging/media/imx/imx-media-of.c
create mode 100644 drivers/staging/media/imx/imx-media-of.h
create mode 100644 drivers/staging/media/imx/imx-media.h
create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
create mode 100644 drivers/staging/media/imx/imx-smfc.c
create mode 100644 drivers/staging/media/imx/imx-video-switch.c
create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
create mode 100644 drivers/staging/media/imx/ov5642.c
create mode 100644 include/media/imx.h
create mode 100644 include/uapi/media/Kbuild
create mode 100644 include/uapi/media/imx.h

--
2.7.4

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:06 PM1/3/17

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

Until the OV5652 sensor module compatible with the SabreSD becomes
available for testing, the ov5642 node is currently disabled.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

compatible = "fsl,imx6dl-sabresd", "fsl,imx6dl";

};
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

diff --git a/arch/arm/boot/dts/imx6q-sabresd.dts b/arch/arm/boot/dts/imx6q-sabresd.dts
index 9cbdfe7..8c1d7ad 100644
--- a/arch/arm/boot/dts/imx6q-sabresd.dts
+++ b/arch/arm/boot/dts/imx6q-sabresd.dts

@@ -23,3 +23,8 @@

&sata {
status = "okay";
};
+
+&ipu1_csi1_from_mipi_vc1 {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
index 55ef535..39b4228 100644
--- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
@@ -10,6 +10,7 @@
* http://www.gnu.org/copyleft/gpl.html
*/

+#include <dt-bindings/clock/imx6qdl-clock.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/input/input.h>

@@ -146,6 +147,33 @@

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;

+ data-shift = <12>; /* Lines 19:12 used */
+ hsync-active = <1>;

+ vsync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;

+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+

+&mipi_csi {

+ status = "okay";
+};

+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+ remote-endpoint = <&ov5640_to_mipi_csi>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
+

&audmux {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_audmux>;

@@ -214,7 +242,33 @@
0x8014 /* 4:FN_DMICCDAT */
0x0000 /* 5:Default */
>;
- };
+ };
+
+ camera: ov5642@3c {
+ compatible = "ovti,ov5642";

+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5642>;
+ clocks = <&clks IMX6QDL_CLK_CKO>;
+ clock-names = "xclk";
+ reg = <0x3c>;
+ xclk = <24000000>;

+ DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+ AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
+ rev B board is VGEN5 */
+ DVDD-supply = <&vgen2_reg>; /* 1.5v*/
+ pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>; /* SD1_DAT0 */
+ reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>; /* SD1_DAT1 */
+ status = "disabled";

+
+ port {
+ ov5642_to_ipu1_csi0_mux: endpoint {

+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;

+ hsync-active = <1>;
+ vsync-active = <1>;
+ };
+ };
+ };

};

&i2c2 {
@@ -322,6 +376,34 @@
};
};
};
+
+ mipi_camera: ov5640@3c {
+ compatible = "ovti,ov5640_mipi";

+ pinctrl-names = "default";

+ pinctrl-0 = <&pinctrl_ov5640>;
+ reg = <0x3c>;
+ clocks = <&clks IMX6QDL_CLK_CKO>;

+ clock-names = "xclk";

+ xclk = <24000000>;
+ DOVDD-supply = <&vgen4_reg>; /* 1.8v */
+ AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
+ rev B board is VGEN5 */
+ DVDD-supply = <&vgen2_reg>; /* 1.5v*/
+ pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>; /* SD1_DAT2 */
+ reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>; /* SD1_CLK */
+
+ port {

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ ov5640_to_mipi_csi: endpoint@1 {
+ reg = <1>;

+ remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+ };
+ };
+ };
};

&i2c3 {

@@ -426,6 +508,36 @@
>;
};

+ pinctrl_ov5640: ov5640grp {
+ fsl,pins = <

+ MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x80000000
+ MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x80000000
+ >;
+ };
+

+ pinctrl_ov5642: ov5642grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x80000000

+ MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x80000000
+ >;
+ };
+

+ pinctrl_ipu1_csi0: ipu1grp-csi0 {
+ fsl,pins = <

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:06 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:07 PM1/3/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

+ #address-cells = <1>;
+ #size-cells = <0>;

+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_i2c3mux>;
+ mux-gpios = <&gpio5 4 0>;
+ i2c-parent = <&i2c3>;
+ idle-state = <0>;
+
+ i2c@1 {

+ #address-cells = <1>;
+ #size-cells = <0>;

+ reg = <1>;
+

Steve Longerbeam

unread,

Jan 3, 2017, 4:10:09 PM1/3/17

This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

A video4linux camera capture driver for i.MX5/6.

+config IMX_OV5640_MIPI
+ tristate "OmniVision OV5640 MIPI CSI-2 camera support"
+ depends on GPIOLIB && VIDEO_IMX_CAMERA
+ select IMX_MIPI_CSI2

+ default y
+ ---help---

+ MIPI CSI-2 OV5640 Camera support.
+
endmenu

endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile

index e3d6d8d..f96e623 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -11,3 +11,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-video-switch.o
+
+obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
new file mode 100644
index 0000000..a83b581
--- /dev/null
+++ b/drivers/staging/media/imx/ov5640-mipi.c
@@ -0,0 +1,2349 @@
+/*
+ * Copyright (c) 2014 Mentor Graphics Inc.
+ * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.

+ return ret;
+ }
+
+ return 0;
+}
+

+ return 0;
+}
+

+#define OV5640_READ_REG(s, r, v) { \
+ ret = ov5640_read_reg((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }

+#define OV5640_WRITE_REG(s, r, v) { \
+ ret = ov5640_write_reg((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)

+{
+ int ret;
+

+ OV5640_WRITE_REG(sensor, reg, val >> 8);
+ OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
+ return 0;
+}
+#define OV5640_WRITE_REG16(s, r, v) { \
+ ret = ov5640_write_reg16((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+ int i, ret;
+

+ }
+
+ return 0;
+}
+

+static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)

+{
+ int ret;
+

+ OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);

+ return 0;
+}
+

+{
+ int ret;
+

+ OV5640_WRITE_REG16(sensor, 0x380e, VTS);

+ return 0;
+}
+

+ if (ret < 0)
+ return ret;

+ sensor->prev_sysclk = ret;
+ /* read preview HTS */
+ ret = ov5640_get_HTS(sensor);

+ if (ret < 0)
+ return ret;

+ sensor->prev_hts = ret;
+
+ /* read preview VTS */
+ ret = ov5640_get_VTS(sensor);

+ if (ret < 0)
+ return ret;

+ prev_vts = ret;
+
+ /* calculate banding filter */
+ /* 60Hz */
+ band_step60 = sensor->prev_sysclk * 100 / sensor->prev_hts * 100/120;
+ OV5640_WRITE_REG16(sensor, 0x3a0a, band_step60);
+
+ max_band60 = (int)((prev_vts-4)/band_step60);
+ OV5640_WRITE_REG(sensor, 0x3a0d, max_band60);
+
+ /* 50Hz */
+ band_step50 = sensor->prev_sysclk * 100 / sensor->prev_hts;
+ OV5640_WRITE_REG16(sensor, 0x3a08, band_step50);
+
+ max_band50 = (int)((prev_vts-4)/band_step50);
+ OV5640_WRITE_REG(sensor, 0x3a0e, max_band50);

+
+ return 0;
+}
+

+ int ret = 0;
+

+ if (ret < 0)
+ return ret;

+ prev_shutter = ret;
+ ret = ov5640_binning_on(sensor);
+ if (ret < 0)
+ return ret;
+ if (ret && mode != ov5640_mode_720P_1280_720 &&
+ mode != ov5640_mode_1080P_1920_1080)
+ prev_shutter *= 2;
+
+ /* read preview gain */
+ ret = ov5640_get_gain(sensor);

+ if (ret < 0)
+ return ret;

+ prev_gain16 = ret;
+
+ /* get average */
+ OV5640_READ_REG(sensor, 0x56a1, &average);
+
+ /* turn off night mode for capture */
+ ret = ov5640_set_night_mode(sensor);
+ if (ret < 0)
+ return ret;
+
+ /* turn off overlay */
+ /* OV5640_WRITE_REG(0x3022, 0x06); //if no af function,
+ just skip it */
+
+ ret = ov5640_set_stream(sensor, false);
+ if (ret < 0)
+ return ret;
+
+ /* Write capture setting */
+ ret = ov5640_load_regs(sensor, mode_data, mode_size);
+ if (ret < 0)
+ return ret;
+
+ /* read capture VTS */
+ ret = ov5640_get_VTS(sensor);

+ if (ret < 0)
+ return ret;

+ cap_vts = ret;
+ ret = ov5640_get_HTS(sensor);

+ if (ret < 0)
+ return ret;

+ cap_hts = ret;
+ ret = ov5640_get_sysclk(sensor);

+ if (ret < 0)
+ return ret;

+ cap_sysclk = ret;
+
+ /* calculate capture banding filter */
+ ret = ov5640_get_light_freq(sensor);

+ if (ret < 0)
+ return ret;

+ light_freq = ret;
+
+ if (light_freq == 60) {
+ /* 60Hz */
+ cap_bandfilt = cap_sysclk * 100 / cap_hts * 100 / 120;
+ } else {
+ /* 50Hz */
+ cap_bandfilt = cap_sysclk * 100 / cap_hts;
+ }
+ cap_maxband = (int)((cap_vts - 4) / cap_bandfilt);
+
+ /* calculate capture shutter/gain16 */
+ if (average > sensor->ae_low && average < sensor->ae_high) {
+ /* in stable range */
+ cap_gain16_shutter =
+ prev_gain16 * prev_shutter *
+ cap_sysclk / sensor->prev_sysclk *
+ sensor->prev_hts / cap_hts *
+ sensor->ae_target / average;
+ } else {
+ cap_gain16_shutter =
+ prev_gain16 * prev_shutter *
+ cap_sysclk / sensor->prev_sysclk *
+ sensor->prev_hts / cap_hts;
+ }
+
+ /* gain to shutter */
+ if (cap_gain16_shutter < (cap_bandfilt * 16)) {
+ /* shutter < 1/100 */
+ cap_shutter = cap_gain16_shutter / 16;
+ if (cap_shutter < 1)
+ cap_shutter = 1;
+
+ cap_gain16 = cap_gain16_shutter / cap_shutter;
+ if (cap_gain16 < 16)
+ cap_gain16 = 16;
+ } else {
+ if (cap_gain16_shutter > (cap_bandfilt * cap_maxband * 16)) {
+ /* exposure reach max */
+ cap_shutter = cap_bandfilt * cap_maxband;
+ cap_gain16 = cap_gain16_shutter / cap_shutter;
+ } else {
+ /* 1/100 < (cap_shutter = n/100) =< max */
+ cap_shutter =
+ ((int)(cap_gain16_shutter / 16 / cap_bandfilt))
+ * cap_bandfilt;
+ cap_gain16 = cap_gain16_shutter / cap_shutter;
+ }
+ }
+
+ /* write capture gain */
+ ret = ov5640_set_gain(sensor, cap_gain16);
+ if (ret < 0)
+ return ret;
+
+ /* write capture shutter */
+ if (cap_shutter > (cap_vts - 4)) {
+ cap_vts = cap_shutter + 4;
+ ret = ov5640_set_VTS(sensor, cap_vts);
+ if (ret < 0)

+ return ret;
+ }
+

+ int ret = 0;
+

+ if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
+ mode != ov5640_mode_INIT) {
+ v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");

+ return -EINVAL;
+ }
+

+ if (ret < 0)
+ return ret;

+ ret = ov5640_get_light_freq(sensor);

+ if (ret < 0)
+ return ret;

+ ret = ov5640_set_bandingfilter(sensor);

+ if (ret < 0)
+ return ret;

+ ret = ov5640_set_virtual_channel(sensor);
+ if (ret < 0)
+ return ret;
+
+ /* restore controls */
+ ov5640_restore_ctrls(sensor);
+
+ if (ret >= 0 && mode != ov5640_mode_INIT) {
+ sensor->current_mode = mode;
+ sensor->current_fr = frame_rate;

+ }
+
+ return 0;
+}
+

+/* restore the last set video mode after chip power-on */
+static int ov5640_restore_mode(struct ov5640_dev *sensor)
+{

+ int ret = 0;
+

+{
+ int ret;
+

+ if (sensor->io_regulator) {
+ ret = regulator_enable(sensor->io_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "io reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->core_regulator) {
+ ret = regulator_enable(sensor->core_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "core reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->gpo_regulator) {
+ ret = regulator_enable(sensor->gpo_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "gpo reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->analog_regulator) {
+ ret = regulator_enable(sensor->analog_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "analog reg enable failed\n");

+ return ret;
+ }
+ }
+
+ return 0;
+}
+

+ v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+

+ if (on && !sensor->on) {
+ if (sensor->xclk)
+ clk_prepare_enable(sensor->xclk);
+
+ ret = ov5640_regulators_on(sensor);

+ if (ret)
+ return ret;
+

+ ov5640_reset(sensor);
+ ov5640_power(sensor, true);
+
+ ret = ov5640_init_slave_id(sensor);

+ if (ret)
+ return ret;
+

+ ret = ov5640_restore_mode(sensor);

+ if (ret)
+ return ret;
+
+ /*

+ * NOTE: Freescale adds a long delay (600 msec) after
+ * powering up and programming a mode on the ov5640-mipi
+ * camera (search for "msec_wait4stable" in FSL's
+ * ov5640_mipi.c), which equivalently would need to go
+ * right here. If we run into MIPI CSI-2 receiver dphy
+ * ready timeouts, it might be a clue to add that delay
+ * here.
+ */
+ } else if (!on && sensor->on) {
+ ov5640_power(sensor, false);
+
+ ov5640_regulators_off(sensor);
+
+ if (sensor->xclk)
+ clk_disable_unprepare(sensor->xclk);
+ }
+
+ sensor->on = on;

+
+ return 0;
+}
+

+static int ov5640_g_parm(struct v4l2_subdev *sd, struct v4l2_streamparm *a)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+ struct v4l2_captureparm *cparm = &a->parm.capture;

+
+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)

+ return -EINVAL;
+
+ /* This is the only case currently handled. */
+ memset(a, 0, sizeof(*a));
+ a->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ cparm->capability = sensor->streamcap.capability;
+ cparm->timeperframe = sensor->streamcap.timeperframe;
+ cparm->capturemode = sensor->streamcap.capturemode;

+
+ return 0;
+}
+

+ int ret = 0;
+

+ return -EINVAL;
+ }
+

+ ret = ov5640_change_mode(sensor, frame_rate,
+ sensor->current_mode,
+ sensor->current_mode);
+ if (ret < 0)
+ return ret;
+
+ sensor->streamcap.timeperframe = *timeperframe;

+
+ return 0;
+}
+

+ return 0;
+}
+

+ return 0;
+ }
+

+ ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);

+ if (ret)
+ return ret;
+

+ ret = ov5640_change_mode(sensor, sensor->current_fr,
+ new_mode, sensor->current_mode);
+ if (ret >= 0)
+ sensor->fmt = format->format;
+

+ return ret;
+}
+
+

+/*
+ * Sensor Controls.
+ */
+
+static int ov5640_set_hue(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+ OV5640_WRITE_REG16(sensor, 0x5581, value);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);

+
+ return 0;
+}
+

+static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+ OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);

+
+ return 0;
+}
+

+static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 1 << 1);
+ OV5640_WRITE_REG(sensor, 0x5583, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x5584, value & 0xff);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 1, 0);

+
+ return 0;
+}
+

+static int ov5640_set_awb(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ sensor->awb_on = value ? true : false;
+ OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);

+ return 0;
+}
+

+static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);

+ return 0;
+}
+

+#if 0
+static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3403, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3402, (value & 0xf00) >> 8);
+ return 0;
+}
+#endif
+
+static int ov5640_set_blue_balance(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);

+ return 0;
+}
+

+ }
+
+ return 0;
+}
+

+{
+ int ret;
+

+ /* this enables/disables both AEC and AGC */
+ sensor->agc_on = value ? true : false;
+ OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);

+
+ return 0;
+}
+

+static int ov5640_set_gain(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->agc_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);

+ return 0;
+}
+

+{
+ int ret;
+

+ OV5640_MOD_REG(sensor, 0x503d, 0xa4, value ? 0xa4 : 0);
+ return 0;
+}
+#endif
+
+static struct ov5640_control ov5640_ctrls[] = {
+ {
+ .set = ov5640_set_agc,
+ .ctrl = {
+ .id = V4L2_CID_AUTOGAIN,
+ .name = "Auto Gain/Exposure Control",
+ .minimum = 0,
+ .maximum = 1,
+ .step = 1,
+ .default_value = 1,
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ },
+ }, {
+ .set = ov5640_set_exposure,
+ .ctrl = {
+ .id = V4L2_CID_EXPOSURE,
+ .name = "Exposure",
+ .minimum = 0,
+ .maximum = 65535,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_gain,
+ .ctrl = {
+ .id = V4L2_CID_GAIN,
+ .name = "Gain",
+ .minimum = 0,
+ .maximum = 1023,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_hue,
+ .ctrl = {
+ .id = V4L2_CID_HUE,
+ .name = "Hue",
+ .minimum = 0,
+ .maximum = 359,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_contrast,
+ .ctrl = {
+ .id = V4L2_CID_CONTRAST,
+ .name = "Contrast",
+ .minimum = 0,
+ .maximum = 255,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_saturation,
+ .ctrl = {
+ .id = V4L2_CID_SATURATION,
+ .name = "Saturation",
+ .minimum = 0,
+ .maximum = 255,
+ .step = 1,
+ .default_value = 64,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_awb,
+ .ctrl = {
+ .id = V4L2_CID_AUTO_WHITE_BALANCE,
+ .name = "Auto White Balance",
+ .minimum = 0,
+ .maximum = 1,
+ .step = 1,
+ .default_value = 1,
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ },
+ }, {
+ .set = ov5640_set_red_balance,
+ .ctrl = {
+ .id = V4L2_CID_RED_BALANCE,
+ .name = "Red Balance",
+ .minimum = 0,
+ .maximum = 4095,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ }, {
+ .set = ov5640_set_blue_balance,
+ .ctrl = {
+ .id = V4L2_CID_BLUE_BALANCE,
+ .name = "Blue Balance",
+ .minimum = 0,
+ .maximum = 4095,
+ .step = 1,
+ .default_value = 0,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ },
+ },
+};
+#define OV5640_NUM_CONTROLS ARRAY_SIZE(ov5640_ctrls)
+
+static struct ov5640_control *ov5640_get_ctrl(int id, int *index)
+{
+ struct ov5640_control *ret = NULL;
+ int i;
+
+ for (i = 0; i < OV5640_NUM_CONTROLS; i++) {
+ if (id == ov5640_ctrls[i].ctrl.id) {
+ ret = &ov5640_ctrls[i];

+ break;
+ }
+ }
+

+ if (ret && index)
+ *index = i;

+ return ret;
+}
+

+ }
+
+ return 0;
+}
+

+static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
+ struct ov5640_control *c;

+ int ret = 0;

+ return ret;
+}
+

+
+ return 0;
+}
+

+ return 0;
+}
+

+static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ *status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;

+
+ return 0;
+}
+

+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ if (code->pad != 0)
+ return -EINVAL;
+ if (code->index != 0)
+ return -EINVAL;
+
+ code->code = sensor->fmt.code;

+
+ return 0;
+}
+

+static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);

+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ return -EINVAL;
+ }
+

+ if (ret)
+ goto free_ctrls;
+

+ ret = v4l2_async_register_subdev(&sensor->sd);

+ if (ret)
+ goto free_ctrls;
+
+ return 0;
+

+free_ctrls:
+ v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+ ov5640_s_power(&sensor->sd, 0);
+entity_cleanup:
+ media_entity_cleanup(&sensor->sd.entity);
+ ov5640_regulators_off(sensor);

+ return ret;
+}
+

+
+ return 0;
+}
+

+ { /* sentinel */ }
+};

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");

+MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
--
2.7.4

Fabio Estevam

unread,

Jan 3, 2017, 4:20:06 PM1/3/17

On Tue, Jan 3, 2017 at 6:57 PM, Steve Longerbeam <slong...@gmail.com> wrote:
> In version 2 (no functional changes):
>
> - removed patch "gpio: pca953x: Add optional reset gpio control", it
> has been submitted separately.
> - fixed some whitespace errors.
> - added a few missing Signed-off-by's.

Tested the series on a mx6qsabresd, so:

Tested-by: Fabio Estevam <fabio....@nxp.com>

Rob Herring

unread,

Jan 3, 2017, 4:20:07 PM1/3/17

On Thu, Dec 29, 2016 at 02:27:26PM -0800, Steve Longerbeam wrote:
> Add the core media driver for i.MX SOC.

>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

> Documentation/devicetree/bindings/media/imx.txt | 205 +++++
> Documentation/media/v4l-drivers/imx.rst | 429 ++++++++++

> drivers/staging/media/Kconfig | 2 +
> drivers/staging/media/Makefile | 1 +

> drivers/staging/media/imx/Kconfig | 8 +
> drivers/staging/media/imx/Makefile | 6 +
> drivers/staging/media/imx/TODO | 18 +
> drivers/staging/media/imx/imx-media-common.c | 981 ++++++++++++++++++++++
> drivers/staging/media/imx/imx-media-dev.c | 479 +++++++++++
> drivers/staging/media/imx/imx-media-fim.c | 508 +++++++++++
> drivers/staging/media/imx/imx-media-internal-sd.c | 456 ++++++++++
> drivers/staging/media/imx/imx-media-of.c | 291 +++++++
> drivers/staging/media/imx/imx-media-of.h | 25 +
> drivers/staging/media/imx/imx-media.h | 290 +++++++

> 19 files changed, 3751 insertions(+)

> create mode 100644 Documentation/devicetree/bindings/media/imx.txt
> create mode 100644 Documentation/media/v4l-drivers/imx.rst
> create mode 100644 drivers/staging/media/imx/Kconfig
> create mode 100644 drivers/staging/media/imx/Makefile
> create mode 100644 drivers/staging/media/imx/TODO

> create mode 100644 drivers/staging/media/imx/imx-media-common.c
> create mode 100644 drivers/staging/media/imx/imx-media-dev.c
> create mode 100644 drivers/staging/media/imx/imx-media-fim.c
> create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
> create mode 100644 drivers/staging/media/imx/imx-media-of.c
> create mode 100644 drivers/staging/media/imx/imx-media-of.h
> create mode 100644 drivers/staging/media/imx/imx-media.h

> create mode 100644 include/media/imx.h
> create mode 100644 include/uapi/media/Kbuild
> create mode 100644 include/uapi/media/imx.h
>

> diff --git a/Documentation/devicetree/bindings/media/imx.txt b/Documentation/devicetree/bindings/media/imx.txt
> new file mode 100644
> index 0000000..3593354
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/media/imx.txt
> @@ -0,0 +1,205 @@
> +Freescale i.MX Media Video Devices
> +
> +Video Media Controller node
> +---------------------------
> +
> +This is the parent media controller node for video capture support.
> +
> +Required properties:
> +- compatible : "fsl,imx-media";
> +- ports : Should contain a list of phandles pointing to camera
> + sensor interface ports of IPU devices
> +
> +
> +fim child node
> +--------------
> +
> +This is an optional child node of the ipu_csi port nodes. It can be used
> +to modify the default control values for the video capture Frame
> +Interval Monitor. Refer to Documentation/media/v4l-drivers/imx.rst for
> +more info on the Frame Interval Monitor.
> +
> +Optional properties:
> +- enable : enable (1) or disable (0) the FIM;

"status" property doesn't work for you?

> +- num-avg : how many frame intervals the FIM will average;
> +- num-skip : how many frames the FIM will skip after a video
> + capture restart before beginning to sample frame
> + intervals;
> +- tolerance-range : a range of tolerances for the averaged frame
> + interval error, specified as <min max>, in usec.
> + The FIM will signal a frame interval error if
> + min < error < max. If the max is <= min, then
> + tolerance range is disabled (interval error if
> + error > min).

Needs a unit suffix (see property-units.txt).

> +- input-capture-channel: an input capture channel and channel flags,

These all need vendor prefix.

> + specified as <chan flags>. The channel number
> + must be 0 or 1. The flags can be
> + IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
> + IRQ_TYPE_EDGE_BOTH, and specify which input
> + capture signal edge will trigger the event. If
> + an input capture channel is specified, the FIM
> + will use this method to measure frame intervals
> + instead of via the EOF interrupt. The input capture
> + method is much preferred over EOF as it is not
> + subject to interrupt latency errors. However it
> + requires routing the VSYNC or FIELD output
> + signals of the camera sensor to one of the
> + i.MX input capture pads (SD1_DAT0, SD1_DAT1),
> + which also gives up support for SD1.
> +
> +
> +mipi_csi2 node
> +--------------
> +
> +This is the device node for the MIPI CSI-2 Receiver, required for MIPI
> +CSI-2 sensors.
> +
> +Required properties:
> +- compatible : "fsl,imx-mipi-csi2";

This needs to be an SoC specific compatible string.

> +- reg : physical base address and length of the register set;
> +- clocks : the MIPI CSI-2 receiver requires three clocks: hsi_tx
> + (the DPHY clock), video_27m, and eim_sel;
> +- clock-names : must contain "dphy_clk", "cfg_clk", "pix_clk";

"_clk" is redundant.

> +
> +Optional properties:
> +- interrupts : must contain two level-triggered interrupts,
> + in order: 100 and 101;
> +
> +
> +video mux node
> +--------------
> +
> +This is the device node for the video multiplexer. It can control
> +either the i.MX internal video mux that selects between parallel image
> +sensors and MIPI CSI-2 virtual channels, or an external mux controlled
> +by a GPIO. It must be a child device of the syscon GPR device.

These sound like 2 completely separate things and should not both be
"imx-video-mux".

> +
> +Required properties:
> +- compatible : "imx-video-mux";

Needs to be SoC specific.

> +- sink-ports : the number of sink (input) ports that follow
> +- ports : at least 2 sink ports must be specified that define

OF graph already provides a way to handle a mux. Multiple endpoints for
an input port is a mux (or some kind of mixer). It depends on the
definition of the port which depends on the parent compatible string.

> + the endpoint inputs to the video mux, and there must
> + be exactly one output port endpoint which must be the
> + last port endpoint defined;
> +
> +Optional properties:
> +- reg : the GPR iomuxc register offset and bitmask of the
> + internal mux bits;

This should not be optional.

> +- mux-gpios : if reg is not specified, this must exist to define
> + a GPIO to control an external mux;
> +
> +
> +SabreLite Quad with OV5642 and OV5640
> +-------------------------------------

Don't put board specifics in here. I assume these sensors are already
documented?

> +
> +On the Sabrelite, the OV5642 module is connected to the parallel bus
> +input on the i.MX internal video mux to IPU1 CSI0. It's i2c bus connects
> +to i2c bus 2, so the ov5642 sensor node must be a child of i2c2.
> +
> +The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
> +receiver, and the four virtual channel outputs from the receiver are
> +routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
> +vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
> +also connected to i2c bus 2 on the SabreLite, so it also must be a child
> +of i2c2. Therefore the OV5642 and OV5640 must not share the same i2c slave
> +address.
> +
> +OV5642 Required properties:
> +- compatible : "ovti,ov5642";
> +- clocks : the OV5642 system clock (cko2, 200 on Sabrelite);
> +- clock-names : must be "xclk";
> +- reg : i2c slave address (must not be default 0x3c on Sabrelite);
> +- xclk : the system clock frequency (24000000 on Sabrelite);
> +- reset-gpios : gpio for the reset pin to OV5642
> +- pwdn-gpios : gpio for the powewr-down pin to OV5642
> +
> +OV5642 Endpoint Required properties:
> +- remote-endpoint : must connect to parallel sensor interface input endpoint
> + on ipu1_csi0 video mux (ipu1_csi0_mux_from_parallel_sensor).
> +- bus-width : must be 8;
> +- hsync-active : must be 1;
> +- vsync-active : must be 1;
> +
> +OV5640 Required properties:
> +- compatible : "ovti,ov5640_mipi";

The "mipi" part is implied by the type of sensor or the graph
connection.

> +- clocks : the OV5640 system clock (pwm3 on Sabrelite);
> +- clock-names : must be "xclk";
> +- reg : i2c slave address (must not be default 0x3c on Sabrelite);
> +- xclk : the system clock frequency (22000000 on Sabrelite);
> +- reset-gpios : gpio for the reset pin to OV5640
> +- pwdn-gpios : gpio for the power-down pin to OV5640
> +
> +OV5640 MIPI CSI-2 Endpoint Required properties:
> +- remote-endpoint : must connect to mipi_csi receiver input endpoint
> + (mipi_csi_from_mipi_sensor).
> +- reg : the MIPI CSI-2 virtual channel to transmit over;
> +- data-lanes : must be <0 1>;
> +- clock-lanes : must be <2>;
> +
> +OV5640/OV5642 Optional properties:
> +- DOVDD-supply : DOVDD regulator supply;
> +- AVDD-supply : AVDD regulator supply;
> +- DVDD-supply : DVDD regulator supply;
> +
> +
> +SabreAuto Quad with ADV7180
> +---------------------------
> +
> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
> +parallel bus input on the internal video mux to IPU1 CSI0.
> +
> +Two analog video inputs are routed to the ADV7180 on the SabreAuto,
> +composite on Ain1, and composite on Ain3. Those inputs are defined
> +via inputs and input-names properties in the ADV7180 device node.
> +
> +Regulators and port expanders are required for the ADV7180 (power pin
> +is via port expander gpio on i2c3). The reset pin to the port expander
> +chip (MAX7310) is controlled by a gpio, so a reset-gpios property must
> +be defined under the port expander node to control it.
> +
> +The sabreauto uses a steering pin to select between the SDA signal on
> +i2c3 bus, and a data-in pin for an SPI NOR chip. i2cmux can be used to
> +control this steering pin. Idle state of the i2cmux selects SPI NOR.
> +This is not classic way to use i2cmux, since one side of the mux selects
> +something other than an i2c bus, but it works and is probably the cleanest
> +solution. Note that if one thread is attempting to access SPI NOR while
> +another thread is accessing i2c3, the SPI NOR access will fail since the
> +i2cmux has selected the SDA pin rather than SPI NOR data-in. This couldn't
> +be avoided in any case, the board is not designed to allow concurrent
> +i2c3 and SPI NOR functions (and the default device-tree does not enable
> +SPI NOR anyway).
> +
> +ADV7180 Required properties:
> +- compatible : "adi,adv7180";
> +- reg : must be 0x21;
> +
> +ADV7180 Optional properties:
> +- DOVDD-supply : DOVDD regulator supply;
> +- AVDD-supply : AVDD regulator supply;
> +- DVDD-supply : DVDD regulator supply;
> +- PVDD-supply : PVDD regulator supply;
> +- pwdn-gpios : gpio to control ADV7180 power pin, must be
> + <&port_exp_b 2 GPIO_ACTIVE_LOW> on SabreAuto;
> +- interrupts : interrupt from ADV7180, must be <27 0x8> on SabreAuto;
> +- interrupt-parent : must be <&gpio1> on SabreAuto;
> +- inputs : list of input mux values, must be 0x00 followed by
> + 0x02 on SabreAuto;
> +- input-names : names of the inputs;
> +
> +ADV7180 Endpoint Required properties:
> +- remote-endpoint : must connect to parallel sensor interface input endpoint
> + on ipu1_csi0 video mux (ipu1_csi0_mux_from_parallel_sensor).
> +- bus-width : must be 8;
> +
> +
> +SabreSD Quad with OV5642 and MIPI CSI-2 OV5640
> +----------------------------------------------
> +
> +Similarly to SabreLite, the SabreSD supports a parallel interface
> +OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640 module. The OV5642
> +connects to i2c bus 1 (i2c1) and the OV5640 to i2c bus 2 (i2c2).
> +
> +OV5640 and OV5642 properties are as described above on SabreLite.
> +
> +The OV5642 support has not been tested yet due to lack of hardware,
> +so only OV5640 is enabled in the device tree at this time.

Vladimir Zapolskiy

unread,

Jan 4, 2017, 7:30:05 AM1/4/17

Hi Steve,

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> Both hang off the same i2c2 bus, so they require different (and non-
> default) i2c slave addresses.
>
> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
>
> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> mipi_csi. It is set to transmit over MIPI virtual channel 1.
>
> Note there is a pin conflict with GPIO6. This pin functions as a power
> input pin to the OV5642, but ENET uses it as the h/w workaround for
> erratum ERR006687, to wake-up the ARM cores on normal RX and TX packet
> done events (see 6261c4c8). So workaround 6261c4c8 is reverted here to
> support the OV5642, and the "fsl,err006687-workaround-present" boolean
> also must be removed. The result is that the CPUidle driver will no longer
> allow entering the deep idle states on the sabrelite.

For me it sounds like a candidate of its own separate change.

>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

[snip]

> &audmux {
> pinctrl-names = "default";
> pinctrl-0 = <&pinctrl_audmux>;
> @@ -271,9 +298,6 @@
> txd1-skew-ps = <0>;
> txd2-skew-ps = <0>;
> txd3-skew-ps = <0>;
> - interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
> - <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;

Like you say in the commit message this is a partial revert of 6261c4c8f13e
("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC interrupt.")

> - fsl,err006687-workaround-present;

This is a partial revert of a28eeb43ee57 ("ARM: dts: imx6: tag boards that
have the HW workaround for ERR006687").

The change should be split and reviewed separately in my opinion, also
cc Gary Bisson <gary....@boundarydevices.com> for SabreLite changes.

Please reorder device nodes by address value, also according to ePAPR
node names should be generic, labels can be specific:

ov5640: camera@40 {
...
};

ov5642: camera@42 {
...
};

Yep.

> >;
> };
>
> @@ -449,6 +518,39 @@
> >;
> };
>
> + pinctrl_ov5642: ov5642grp {
> + fsl,pins = <
> + MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x80000000
> + MX6QDL_PAD_GPIO_6__GPIO1_IO06 0x80000000
> + MX6QDL_PAD_GPIO_8__GPIO1_IO08 0x80000000
> + MX6QDL_PAD_GPIO_3__CCM_CLKO2 0x80000000
> + >;
> + };
> +
> + pinctrl_ipu1_csi0: ipu1grp-csi0 {

Please rename node name to ipu1csi0grp.

> + fsl,pins = <
> + MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x80000000
> + MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x80000000
> + MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x80000000
> + MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x80000000
> + MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x80000000
> + MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x80000000
> + MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x80000000
> + MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x80000000
> + MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x80000000
> + MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x80000000
> + MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x80000000
> + MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x80000000
> + >;
> + };
> +
> + pinctrl_ov5640: ov5640grp {
> + fsl,pins = <
> + MX6QDL_PAD_NANDF_D5__GPIO2_IO05 0x000b0
> + MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
> + >;
> + };
> +

Indentation issues above, please use tabs instead of spaces.

Also please add new pin control groups preserving the alphanimerical order.

--
With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 7:40:05 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
>
> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
>
> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> mipi_csi. It is set to transmit over MIPI virtual channel 1.
>
> Until the OV5652 sensor module compatible with the SabreSD becomes
> available for testing, the ov5642 node is currently disabled.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

[snip]

> +
> + camera: ov5642@3c {

ov5642: camera@3c

> + compatible = "ovti,ov5642";
> + pinctrl-names = "default";
> + pinctrl-0 = <&pinctrl_ov5642>;
> + clocks = <&clks IMX6QDL_CLK_CKO>;
> + clock-names = "xclk";
> + reg = <0x3c>;
> + xclk = <24000000>;
> + DOVDD-supply = <&vgen4_reg>; /* 1.8v */
> + AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
> + rev B board is VGEN5 */
> + DVDD-supply = <&vgen2_reg>; /* 1.5v*/
> + pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>; /* SD1_DAT0 */
> + reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>; /* SD1_DAT1 */

Comments about SD1_* pad names are redundant.

> + status = "disabled";

Why is it disabled here?

> +
> + port {
> + ov5642_to_ipu1_csi0_mux: endpoint {
> + remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
> + bus-width = <8>;
> + hsync-active = <1>;
> + vsync-active = <1>;
> + };
> + };
> + };
> };
>
> &i2c2 {
> @@ -322,6 +376,34 @@
> };
> };
> };
> +
> + mipi_camera: ov5640@3c {

ov5640: camera@3c

> + compatible = "ovti,ov5640_mipi";
> + pinctrl-names = "default";
> + pinctrl-0 = <&pinctrl_ov5640>;
> + reg = <0x3c>;
> + clocks = <&clks IMX6QDL_CLK_CKO>;
> + clock-names = "xclk";
> + xclk = <24000000>;
> + DOVDD-supply = <&vgen4_reg>; /* 1.8v */
> + AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
> + rev B board is VGEN5 */
> + DVDD-supply = <&vgen2_reg>; /* 1.5v*/
> + pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>; /* SD1_DAT2 */
> + reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>; /* SD1_CLK */

Comments about SD1_* pad names are redundant.

Please rename the node name to ipu1csi0grp.

Please add new pin control groups preserving the alphanimerical order.

> + fsl,pins = <
> + MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x80000000
> + MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x80000000
> + MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x80000000
> + MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x80000000
> + MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x80000000
> + MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x80000000
> + MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x80000000
> + MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x80000000
> + MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x80000000
> + MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x80000000
> + MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x80000000
> + >;
> + };
> +
> pinctrl_pcie: pciegrp {
> fsl,pins = <
> MX6QDL_PAD_GPIO_17__GPIO7_IO12 0x1b0b0
>

--

With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 7:50:06 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> Enables the ADV7180 decoder sensor. The ADV7180 connects to the
> parallel-bus mux input on ipu1_csi0_mux.
>
> On the sabreauto, two analog video inputs are routed to the ADV7180,
> composite on Ain1, and composite on Ain3. Those inputs are defined
> via inputs and input-names under the ADV7180 node. The ADV7180 power
> pin is via max7310_b port expander.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---
> arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 56 ++++++++++++++++++++++++++++++++
> 1 file changed, 56 insertions(+)
>
> diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> index 83ac2ff..30ee378 100644
> --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
> @@ -147,10 +147,42 @@
> gpio-controller;
> #gpio-cells = <2>;
> };
> +
> + camera: adv7180@21 {

adv7180: camera@21

Please rename node name to ipu1csi0grp.

With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 8:50:08 AM1/4/17

Hi Steve,

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> Add the core media driver for i.MX SOC.
>

> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

> Documentation/devicetree/bindings/media/imx.txt | 205 +++++

v2 was sent before getting Rob's review comments, but still they
should be addressed in v3.

Also I would suggest to separate device tree binding documentation
change and place it as the first patch in the series, this should
make the following DTS changes valid.

> Documentation/media/v4l-drivers/imx.rst | 430 ++++++++++

> drivers/staging/media/Kconfig | 2 +
> drivers/staging/media/Makefile | 1 +

> drivers/staging/media/imx/Kconfig | 8 +
> drivers/staging/media/imx/Makefile | 6 +
> drivers/staging/media/imx/TODO | 18 +

> drivers/staging/media/imx/imx-media-common.c | 985 ++++++++++++++++++++++
> drivers/staging/media/imx/imx-media-dev.c | 479 +++++++++++
> drivers/staging/media/imx/imx-media-fim.c | 509 +++++++++++
> drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
> drivers/staging/media/imx/imx-media-of.c | 291 +++++++
> drivers/staging/media/imx/imx-media-of.h | 25 +
> drivers/staging/media/imx/imx-media.h | 299 +++++++

Probably Greg should ack the UAPI changes, you may consider
to split them into a separate patch.

> 19 files changed, 3767 insertions(+)

> create mode 100644 include/media/imx.h
> create mode 100644 include/uapi/media/Kbuild
> create mode 100644 include/uapi/media/imx.h
>

[snip]

> +
> +struct imx_media_subdev *
> +imx_media_find_subdev_by_sd(struct imx_media_dev *imxmd,
> + struct v4l2_subdev *sd)
> +{
> + struct imx_media_subdev *imxsd;
> + int i, ret = -ENODEV;
> +
> + for (i = 0; i < imxmd->num_subdevs; i++) {
> + imxsd = &imxmd->subdev[i];
> + if (sd == imxsd->sd) {

This can be simplifed:

...

if (sd == imxsd->sd)
return imxsd;
}

return ERR_PTR(-ENODEV);

> + ret = 0;

> + break;
> + }
> + }
> +

> + return ret ? ERR_PTR(ret) : imxsd;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_find_subdev_by_sd);
> +
> +struct imx_media_subdev *
> +imx_media_find_subdev_by_id(struct imx_media_dev *imxmd, u32 grp_id)
> +{
> + struct imx_media_subdev *imxsd;
> + int i, ret = -ENODEV;
> +
> + for (i = 0; i < imxmd->num_subdevs; i++) {
> + imxsd = &imxmd->subdev[i];
> + if (imxsd->sd && imxsd->sd->grp_id == grp_id) {
> + ret = 0;
> + break;

This can be simplifed:

...

if (imxsd->sd && imxsd->sd->grp_id == grp_i)
return imxsd;
}

return ERR_PTR(-ENODEV);

> + }
> + }
> +
> + return ret ? ERR_PTR(ret) : imxsd;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_find_subdev_by_id);
> +

[snip]

> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
> new file mode 100644
> index 0000000..8d22730
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-dev.c
> @@ -0,0 +1,479 @@
> +/*
> + * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *

> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */

> +#include <linux/module.h>
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/timer.h>
> +#include <linux/sched.h>
> +#include <linux/slab.h>

> +#include <linux/spinlock.h>
> +#include <linux/platform_device.h>
> +#include <linux/pinctrl/consumer.h>

> +#include <linux/of_platform.h>
> +#include <media/v4l2-ioctl.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-event.h>
> +#include <media/v4l2-mc.h>

Please sort out the list of headers alphabetically.

> +#include <video/imx-ipu-v3.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-media-of.h"
> +
> +#define DEVICE_NAME "imx-media"

I suppose you don't need this macro.

[snip]

> + */
> +static int imx_media_create_links(struct imx_media_dev *imxmd)
> +{
> + struct imx_media_subdev *local_sd;
> + struct imx_media_subdev *remote_sd;
> + struct v4l2_subdev *source, *sink;
> + struct imx_media_link *link;
> + struct imx_media_pad *pad;
> + u16 source_pad, sink_pad;
> + int num_pads, i, j, k;

> + int ret = 0;
> +

> + for (i = 0; i < imxmd->num_subdevs; i++) {
> + local_sd = &imxmd->subdev[i];
> + num_pads = local_sd->num_sink_pads + local_sd->num_src_pads;
> +
> + for (j = 0; j < num_pads; j++) {
> + pad = &local_sd->pad[j];
> +
> + for (k = 0; k < pad->num_links; k++) {
> + link = &pad->link[k];
> +
> + remote_sd = imx_media_find_async_subdev(
> + imxmd, link->remote_sd_node,
> + link->remote_devname);
> + if (!remote_sd) {
> + v4l2_warn(&imxmd->v4l2_dev,
> + "%s: no remote for %s:%d\n",
> + __func__, local_sd->sd->name,
> + link->local_pad);
> + continue;
> + }
> +
> + /* only create the source->sink links */
> + if (pad->pad.flags & MEDIA_PAD_FL_SINK)
> + continue;
> +
> + source = local_sd->sd;
> + sink = remote_sd->sd;
> + source_pad = link->local_pad;
> + sink_pad = link->remote_pad;
> +
> + v4l2_info(&imxmd->v4l2_dev,
> + "%s: %s:%d -> %s:%d\n", __func__,
> + source->name, source_pad,
> + sink->name, sink_pad);
> +
> + ret = media_create_pad_link(&source->entity,
> + source_pad,
> + &sink->entity,
> + sink_pad,
> + 0);
> + if (ret) {
> + v4l2_err(&imxmd->v4l2_dev,
> + "create_pad_link failed: %d\n",
> + ret);
> + goto out;

Indentation depth is quite terrific.

> + }
> + }
> + }
> + }
> +

> +out:
> + return ret;
>
[snip]

> +
> +static const struct of_device_id imx_media_dt_ids[] = {
> + { .compatible = "fsl,imx-media" },

> + { /* sentinel */ }
> +};

> +MODULE_DEVICE_TABLE(of, imx_media_dt_ids);
> +
> +static struct platform_driver imx_media_pdrv = {
> + .probe = imx_media_probe,
> + .remove = imx_media_remove,
> + .driver = {

> + .name = DEVICE_NAME,
> + .owner = THIS_MODULE,

Setting of .owner is not needed nowadays IIRC.

> + .of_match_table = imx_media_dt_ids,
> + },
> +};
> +
> +module_platform_driver(imx_media_pdrv);
> +
> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");

> +MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");

> +MODULE_LICENSE("GPL");
> diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
> new file mode 100644
> index 0000000..52bfa8d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-fim.c

> @@ -0,0 +1,509 @@
> +/*

> + * Frame Interval Monitor.
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *

> +#include <linux/module.h>
> +#include <linux/delay.h>
> +#include <linux/slab.h>
> +#include <linux/platform_device.h>
> +#ifdef CONFIG_IMX_GPT_ICAP
> +#include <linux/mxc_icap.h>
> +#endif

This looks clumsy. Include it unconditionally, if needed
do #ifdef's inside the header file.

> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>

Please sort out the list alphabetically.

> +#include <media/imx.h>
> +#include "imx-media.h"
> +

[snip]

> +
> +static int of_parse_fim(struct imx_media_fim *fim, struct device_node *np)
> +{
> + struct device_node *fim_np;
> + u32 val, tol[2], icap[2];
> + int ret;
> +
> + fim_np = of_get_child_by_name(np, "fim");
> + if (!fim_np) {
> + /* set to the default defaults */
> + fim->of_defaults[FIM_CL_ENABLE] = FIM_CL_ENABLE_DEF;
> + fim->of_defaults[FIM_CL_NUM] = FIM_CL_NUM_DEF;
> + fim->of_defaults[FIM_CL_NUM_SKIP] = FIM_CL_NUM_SKIP_DEF;
> + fim->of_defaults[FIM_CL_TOLERANCE_MIN] =
> + FIM_CL_TOLERANCE_MIN_DEF;
> + fim->of_defaults[FIM_CL_TOLERANCE_MAX] =
> + FIM_CL_TOLERANCE_MAX_DEF;
> + fim->icap_channel = -1;

> + return 0;
> + }
> +

> + ret = of_property_read_u32(fim_np, "enable", &val);
> + if (ret)
> + val = FIM_CL_ENABLE_DEF;
> + fim->of_defaults[FIM_CL_ENABLE] = val;
> +
> + ret = of_property_read_u32(fim_np, "num-avg", &val);
> + if (ret)
> + val = FIM_CL_NUM_DEF;
> + fim->of_defaults[FIM_CL_NUM] = val;
> +
> + ret = of_property_read_u32(fim_np, "num-skip", &val);
> + if (ret)
> + val = FIM_CL_NUM_SKIP_DEF;
> + fim->of_defaults[FIM_CL_NUM_SKIP] = val;
> +
> + ret = of_property_read_u32_array(fim_np, "tolerance-range", tol, 2);
> + if (ret) {
> + tol[0] = FIM_CL_TOLERANCE_MIN_DEF;
> + tol[1] = FIM_CL_TOLERANCE_MAX_DEF;
> + }
> + fim->of_defaults[FIM_CL_TOLERANCE_MIN] = tol[0];
> + fim->of_defaults[FIM_CL_TOLERANCE_MAX] = tol[1];
> +
> + fim->icap_channel = -1;
> + if (IS_ENABLED(CONFIG_IMX_GPT_ICAP)) {
> + ret = of_property_read_u32_array(fim_np,
> + "input-capture-channel",
> + icap, 2);
> + if (!ret) {
> + fim->icap_channel = icap[0];
> + fim->icap_flags = icap[1];
> + }

Should you return error otherwise?

> + }
> +
> + of_node_put(fim_np);
> + return 0;
> +}

[snip]

> diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c
> new file mode 100644
> index 0000000..018d05a
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-of.c
> @@ -0,0 +1,291 @@
> +/*
> + * Media driver for Freescale i.MX5/6 SOC
> + *
> + * Open Firmware parsing.
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *

> +#include <linux/of_platform.h>
> +#include <media/v4l2-device.h>

> +#include <media/videobuf2-dma-contig.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>

Please sort out the list alphabetically.

> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +

> +static int of_add_pad_link(struct imx_media_dev *imxmd,
> + struct imx_media_pad *pad,
> + struct device_node *local_sd_node,
> + struct device_node *remote_sd_node,
> + int local_pad, int remote_pad)
> +{
> + dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
> + local_sd_node->name, local_pad,
> + remote_sd_node->name, remote_pad);
> +
> + return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
> + local_pad, remote_pad);
> +}
> +
> +/* parse inputs property from a sensor node */
> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> + struct imx_media_subdev *sensor,
> + struct device_node *sensor_np)
> +{
> + struct imx_media_sensor_input *sinput = &sensor->input;
> + int ret, i;
> +
> + for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> + const char *input_name;
> + u32 val;
> +
> + ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> + if (ret)
> + break;
> +
> + sinput->value[i] = val;
> +
> + ret = of_property_read_string_index(sensor_np, "input-names",
> + i, &input_name);
> + /*
> + * if input-names not provided, they will be set using
> + * the subdev name once the sensor is known during
> + * async bind
> + */
> + if (!ret)
> + strncpy(sinput->name[i], input_name,
> + sizeof(sinput->name[i]));
> + }
> +
> + sinput->num = i;
> +
> + /* if no inputs provided just assume a single input */
> + if (sinput->num == 0)
> + sinput->num = 1;
> +}
> +
> +static void of_parse_sensor(struct imx_media_dev *imxmd,
> + struct imx_media_subdev *sensor,
> + struct device_node *sensor_np)
> +{

> + struct device_node *endpoint;
> +

> + of_parse_sensor_inputs(imxmd, sensor, sensor_np);
> +
> + endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
> + if (endpoint) {
> + v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
> + of_node_put(endpoint);
> + }
> +}
> +
> +static int of_get_port_count(const struct device_node *np)
> +{
> + struct device_node *child;
> + int num = 0;
> +
> + /* if this node is itself a port, return 1 */
> + if (of_node_cmp(np->name, "port") == 0)
> + return 1;
> +
> + for_each_child_of_node(np, child) {
> + if (of_node_cmp(child->name, "port") == 0)
> + num++;
> + }

Unneeded bracers.

> + return num;
> +}
> +
> +/*
> + * find the remote device node and remote port id (remote pad #)
> + * given local endpoint node
> + */
> +static void of_get_remote_pad(struct device_node *epnode,
> + struct device_node **remote_node,
> + int *remote_pad)
> +{
> + struct device_node *rp, *rpp;
> + struct device_node *remote;
> +
> + rp = of_graph_get_remote_port(epnode);
> + rpp = of_graph_get_remote_port_parent(epnode);
> +
> + if (of_device_is_compatible(rpp, "fsl,imx6q-ipu")) {
> + /* the remote is one of the CSI ports */
> + remote = rp;
> + *remote_pad = 0;
> + of_node_put(rpp);
> + } else {
> + remote = rpp;
> + of_property_read_u32(rp, "reg", remote_pad);
> + of_node_put(rp);
> + }
> +
> + if (!remote || !of_device_is_available(remote)) {
> + of_node_put(remote);
> + *remote_node = NULL;
> + } else {
> + *remote_node = remote;
> + }
> +}
> +
> +static struct imx_media_subdev *
> +of_parse_subdev(struct imx_media_dev *imxmd, struct device_node *sd_np,
> + bool is_csi_port)
> +{
> + struct imx_media_subdev *imxsd;
> + int i, num_pads, ret;
> +
> + if (!of_device_is_available(sd_np)) {
> + dev_dbg(imxmd->dev, "%s: %s not enabled\n", __func__,
> + sd_np->name);
> + return NULL;
> + }
> +
> + /* register this subdev with async notifier */
> + imxsd = imx_media_add_async_subdev(imxmd, sd_np, NULL);
> + if (!imxsd)
> + return NULL;
> + if (IS_ERR(imxsd))
> + return imxsd;

if (IS_ERR_OR_NULL(imxsd))
return imxsd;

> +
> + if (is_csi_port) {
> + /*
> + * the ipu-csi has one sink port and one source port.
> + * The source port is not represented in the device tree,
> + * but is described by the internal pads and links later.
> + */
> + num_pads = 2;
> + imxsd->num_sink_pads = 1;
> + } else if (of_device_is_compatible(sd_np, "fsl,imx-mipi-csi2")) {
> + num_pads = of_get_port_count(sd_np);
> + /* the mipi csi2 receiver has only one sink port */
> + imxsd->num_sink_pads = 1;
> + } else if (of_device_is_compatible(sd_np, "imx-video-mux")) {
> + num_pads = of_get_port_count(sd_np);
> + /* for the video mux, all but the last port are sinks */
> + imxsd->num_sink_pads = num_pads - 1;
> + } else {
> + /* must be a sensor */
> + num_pads = 1;
> + imxsd->num_sink_pads = 0;
> + }
> +
> + if (imxsd->num_sink_pads >= num_pads)
> + return ERR_PTR(-EINVAL);
> +
> + imxsd->num_src_pads = num_pads - imxsd->num_sink_pads;
> +
> + dev_dbg(imxmd->dev, "%s: %s has %d pads (%d sink, %d src)\n",
> + __func__, sd_np->name, num_pads,

> + imxsd->num_sink_pads, imxsd->num_src_pads);

> +
> + if (imxsd->num_sink_pads == 0) {
> + /* this might be a sensor */
> + of_parse_sensor(imxmd, imxsd, sd_np);
> + }

Unneeded bracers.

> +
> + for (i = 0; i < num_pads; i++) {
> + struct device_node *epnode = NULL, *port, *remote_np;
> + struct imx_media_subdev *remote_imxsd;
> + struct imx_media_pad *pad;
> + int remote_pad;

Too deep indentation, may be move the cycle body into a separate function?

> +
> + /* init this pad */

> + pad = &imxsd->pad[i];

> + pad->pad.flags = (i < imxsd->num_sink_pads) ?
> + MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
> +
> + if (is_csi_port)
> + port = (i < imxsd->num_sink_pads) ? sd_np : NULL;
> + else
> + port = of_graph_get_port_by_id(sd_np, i);
> + if (!port)
> + continue;
> +
> + while ((epnode = of_get_next_child(port, epnode))) {

Please reuse for_each_child_of_node() here.

> + of_get_remote_pad(epnode, &remote_np, &remote_pad);
> + if (!remote_np) {
> + of_node_put(epnode);

Please remove of_node_put() here, of_get_next_child() does it.

> + continue;
> + }
> +
> + ret = of_add_pad_link(imxmd, pad, sd_np, remote_np,
> + i, remote_pad);
> + if (ret) {
> + imxsd = ERR_PTR(ret);
> + break;
> + }
> +
> + if (i < imxsd->num_sink_pads) {
> + /* follow sink endpoints upstream */
> + remote_imxsd = of_parse_subdev(imxmd,
> + remote_np,
> + false);
> + if (IS_ERR(remote_imxsd)) {
> + imxsd = remote_imxsd;

> + break;
> + }
> + }
> +

> + of_node_put(remote_np);
> + of_node_put(epnode);
> + }
> +
> + if (port != sd_np)
> + of_node_put(port);
> + if (IS_ERR(imxsd)) {
> + of_node_put(remote_np);
> + of_node_put(epnode);

> + break;
> + }
> + }
> +

> + return imxsd;
> +}
> +
> +int imx_media_of_parse(struct imx_media_dev *imxmd,
> + struct imx_media_subdev *(*csi)[4],
> + struct device_node *np)
> +{
> + struct device_node *csi_np;
> + struct imx_media_subdev *lcsi;

Please swap two lines above to get the reverse christmas tree ordering.

> + u32 ipu_id, csi_id;

> + int i, ret;
> +

> + for (i = 0; ; i++) {
> + csi_np = of_parse_phandle(np, "ports", i);
> + if (!csi_np)
> + break;
> +
> + lcsi = of_parse_subdev(imxmd, csi_np, true);
> + if (IS_ERR(lcsi)) {
> + ret = PTR_ERR(lcsi);
> + goto err_put;
> + }
> +
> + of_property_read_u32(csi_np, "reg", &csi_id);

Not sure if it is safe enough to ignore return value and potentially
left csi_id uninitialized.

> + ipu_id = of_alias_get_id(csi_np->parent, "ipu");
> +
> + if (ipu_id > 1 || csi_id > 1) {
> + dev_err(imxmd->dev, "%s: invalid ipu/csi id (%u/%u)\n",
> + __func__, ipu_id, csi_id);
> + ret = -EINVAL;
> + goto err_put;
> + }
> +
> + of_node_put(csi_np);

You can put the node right after of_alias_get_id() call, then in case
of error return right from the if block and remove the goto label.

> +
> + (*csi)[ipu_id * 2 + csi_id] = lcsi;

> + }
> +
> + return 0;

> +err_put:
> + of_node_put(csi_np);
> + return ret;
> +}
> diff --git a/drivers/staging/media/imx/imx-media-of.h b/drivers/staging/media/imx/imx-media-of.h
> new file mode 100644
> index 0000000..0c61b05
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-of.h
> @@ -0,0 +1,25 @@
> +/*
> + * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
> + *
> + * Open Firmware parsing.
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *

> +#ifndef _IMX_MEDIA_OF_H
> +#define _IMX_MEDIA_OF_H
> +

I do believe you should include some headers or add declarations
of "struct imx_media_dev", "struct imx_media_subdev", "struct device_node".

> +struct imx_media_subdev *
> +imx_media_of_find_subdev(struct imx_media_dev *imxmd,
> + struct device_node *np,
> + const char *name);
> +
> +int imx_media_of_parse(struct imx_media_dev *dev,
> + struct imx_media_subdev *(*csi)[4],
> + struct device_node *np);
> +
> +#endif
> diff --git a/drivers/staging/media/imx/imx-media.h b/drivers/staging/media/imx/imx-media.h
> new file mode 100644
> index 0000000..6a018a9
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media.h
> @@ -0,0 +1,299 @@
> +/*
> + * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *

> +#ifndef _IMX_MEDIA_H
> +#define _IMX_MEDIA_H

Please insert here an empty line to improve readability.

> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-of.h>

Please sort out the list alphabetically.

> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +
> +/*
> + * This is somewhat arbitrary, but we need at least:
> + * - 2 camera interface subdevs
> + * - 3 IC subdevs
> + * - 2 CSI subdevs
> + * - 1 mipi-csi2 receiver subdev
> + * - 2 video-mux subdevs
> + * - 3 camera sensor subdevs (2 parallel, 1 mipi-csi2)
> + *
> + * And double the above numbers for quad i.mx!
> + */

[snip]

Vladimir Zapolskiy

unread,

Jan 4, 2017, 8:50:10 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

[snip]

> diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
> new file mode 100644
> index 0000000..975eafb
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-csi.c
> @@ -0,0 +1,638 @@
> +/*
> + * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>
> +#include <linux/platform_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-ctrls.h>

Please add the headers alphabetically ordered.

if (ret)

v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);

You can remove the return above.

> + }
> +
> + return 0;
> + }
> +

[snip]

This is a functionally equal and simplified version:

if (ret)
v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

Please drop .owner.

> + },
> +};
> +module_platform_driver(imx_csi_driver);
> +
> +MODULE_DESCRIPTION("i.MX CSI subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:imx-ipuv3-csi");
>

--

With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 9:30:05 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO
> Controller module. Video frames are received from the CSI and can
> be routed to various sinks including the i.MX Image Converter for
> scaling, color-space conversion, motion compensated deinterlacing,
> and image rotation.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---
> drivers/staging/media/imx/Makefile | 1 +
> drivers/staging/media/imx/imx-smfc.c | 739 +++++++++++++++++++++++++++++++++++
> 2 files changed, 740 insertions(+)
> create mode 100644 drivers/staging/media/imx/imx-smfc.c
>
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index 133672a..3559d7b 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
> obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
>
> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

May be

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o imx-smfc.o

Please sort the list of headers alphabetically.

> +#include <media/imx.h>
> +#include "imx-media.h"
> +

[snip]

> +static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
> +{
> + struct imx_smfc_priv *priv = dev_id;
> + struct imx_media_dma_buf *done, *next;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&priv->irqlock, flags);

spin_lock(&priv->irqlock) should be sufficient.

[snip]

> +
> +static const struct platform_device_id imx_smfc_ids[] = {
> + { .name = "imx-ipuv3-smfc" },
> + { },
> +};
> +MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
> +
> +static struct platform_driver imx_smfc_driver = {
> + .probe = imx_smfc_probe,
> + .remove = imx_smfc_remove,
> + .id_table = imx_smfc_ids,
> + .driver = {
> + .name = "imx-ipuv3-smfc",
> + .owner = THIS_MODULE,

You can drop owner assignment.

> + },
> +};
> +module_platform_driver(imx_smfc_driver);
> +
> +MODULE_DESCRIPTION("i.MX SMFC subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:imx-ipuv3-smfc");
>

--

With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 10:00:07 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> This is a set of three media entity subdevice drivers for the i.MX
> Image Converter. The i.MX IC module contains three independent
> "tasks":
>
> - Pre-processing Encode task: video frames are routed directly from
> the CSI and can be scaled, color-space converted, and rotated.
> Scaled output is limited to 1024x1024 resolution. Output frames
> are routed to the camera interface entities (camif).
>
> - Pre-processing Viewfinder task: this task can perform the same
> conversions as the pre-process encode task, but in addition can
> be used for hardware motion compensated deinterlacing. Frames can
> come either directly from the CSI or from the SMFC entities (memory
> buffers via IDMAC channels). Scaled output is limited to 1024x1024
> resolution. Output frames can be routed to various sinks including
> the post-processing task entities.
>
> - Post-processing task: same conversions as pre-process encode. However
> this entity sends frames to the i.MX IPU image converter which supports
> image tiling, which allows scaled output up to 4096x4096 resolution.
> Output frames can be routed to the camera interfaces.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

[snip]

if (ret)

ic_ops[priv->task_id]->remove(priv);

return ret;

as an alternative.

[snip]

> +static const struct platform_device_id imx_ic_ids[] = {
> + { .name = "imx-ipuv3-ic" },
> + { },
> +};
> +MODULE_DEVICE_TABLE(platform, imx_ic_ids);
> +
> +static struct platform_driver imx_ic_driver = {
> + .probe = imx_ic_probe,
> + .remove = imx_ic_remove,
> + .id_table = imx_ic_ids,
> + .driver = {
> + .name = "imx-ipuv3-ic",
> + .owner = THIS_MODULE,

Please drop .owner assignment.

Please sort the list of headers alphabetically.

> +#include <media/imx.h>
> +#include <video/imx-ipu-image-convert.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +

[snip]

In OOM situation the core will report it, probably you can drop the message.

> + ret = -ENOMEM;
> + goto out_free_ring;
> + }
> +
> + priv->stop = false;
> +
> + return 0;
> +
> +out_free_ring:
> + imx_media_free_dma_buf_ring(priv->in_ring);
> + priv->in_ring = NULL;
> +out_unprep:
> + ipu_image_convert_unprepare(priv->ic_ctx);
> + return ret;
> +}
> +

[snip]

Please sort the list of headers alphabetically.

> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +

[snip]

> +static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
> +{
> + struct prpenc_priv *priv = dev_id;
> + struct imx_media_dma_buf *done, *next;
> + struct ipuv3_channel *channel;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&priv->irqlock, flags);

Here spin_lock(&priv->irqlock) should be sufficient.

[snip]

if (ret)
v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

return ret;

version is shorter.

> +}

[snip]

Please sort the list of headers alphabetically.

> +#include <media/imx.h>
> +#include "imx-media.h"
> +#include "imx-ic.h"
> +
> +/*

[snip]

> +/* prpvf_out_ch EOF interrupt (progressive frame ready) */
> +static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
> +{
> + struct prpvf_priv *priv = dev_id;
> + struct imx_media_dma_buf *done;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&priv->irqlock, flags);

Here spin_lock(&priv->irqlock) should be sufficient.

[snip]

> diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
> new file mode 100644
> index 0000000..9aed5f5
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-ic.h
> @@ -0,0 +1,36 @@
> +/*
> + * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#ifndef _IMX_IC_H
> +#define _IMX_IC_H
> +

Please add header files or declarations of all used structs.

> +struct imx_ic_priv {
> + struct device *dev;
> + struct v4l2_subdev sd;
> + int ipu_id;
> + int task_id;
> + void *task_priv;
> +};
> +
> +struct imx_ic_ops {
> + struct v4l2_subdev_ops *subdev_ops;
> + struct v4l2_subdev_internal_ops *internal_ops;
> + struct media_entity_operations *entity_ops;
> +
> + int (*init)(struct imx_ic_priv *ic_priv);
> + void (*remove)(struct imx_ic_priv *ic_priv);
> +};
> +
> +extern struct imx_ic_ops imx_ic_prpenc_ops;
> +extern struct imx_ic_ops imx_ic_prpvf_ops;
> +extern struct imx_ic_ops imx_ic_pp_ops;
> +
> +#endif
> +
>

--

With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 10:00:08 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> This is the camera interface driver that provides the v4l2
> user interface. Frames can be received from various sources:
>
> - directly from SMFC for capturing unconverted images directly from
> camera sensors.
>
> - from the IC pre-process encode task.
>
> - from the IC pre-process viewfinder task.
>
> - from the IC post-process task.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---
> drivers/staging/media/imx/Makefile | 2 +-
> drivers/staging/media/imx/imx-camif.c | 1010 +++++++++++++++++++++++++++++++++
> 2 files changed, 1011 insertions(+), 1 deletion(-)
> create mode 100644 drivers/staging/media/imx/imx-camif.c
>
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index d2a962c..fe9e992 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>
> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> -
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o imx-csi.o imx-smfc.o

as an option.

Please sort the list of headers alphabetically.

> +#include <video/imx-ipu-v3.h>
> +#include <media/imx.h>
> +#include "imx-media.h"
> +
> +#define DEVICE_NAME "imx-media-camif"

I would propose to drop this macro.

> +
> +#define CAMIF_NUM_PADS 2
> +
> +#define CAMIF_DQ_TIMEOUT 5000

Add a comment about time unit?

> +
> +struct camif_priv;
> +

This is a leftover apparently.

> +struct camif_priv {
> + struct device *dev;
> + struct video_device vfd;
> + struct media_pipeline mp;
> + struct imx_media_dev *md;

[snip]

A shorter version:

if (ret)
v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

Please drop the owner assignment.

> + },
> +};
> +
> +module_platform_driver(imx_camif_driver);
> +
> +MODULE_DESCRIPTION("i.MX camera interface subdev driver");
> +MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
> +MODULE_LICENSE("GPL");
>

--

With best wishes,
Vladimir

Vladimir Zapolskiy

unread,

Jan 4, 2017, 10:10:05 AM1/4/17

On 01/03/2017 10:57 PM, Steve Longerbeam wrote:

> Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
> for sensors with a MIPI CSI2 interface.

>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

> drivers/staging/media/imx/Makefile | 1 +
> drivers/staging/media/imx/imx-mipi-csi2.c | 509 ++++++++++++++++++++++++++++++
> 2 files changed, 510 insertions(+)
> create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
>
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index fe9e992..0decef7 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
> diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c
> new file mode 100644
> index 0000000..84df16e
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-mipi-csi2.c

> @@ -0,0 +1,509 @@
> +/*

> + * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.
> + *
> + * Copyright (c) 2012-2014 Mentor Graphics Inc.

> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +#include <linux/module.h>

> +#include <linux/export.h>
> +#include <linux/types.h>
> +#include <linux/init.h>
> +#include <linux/platform_device.h>
> +#include <linux/err.h>
> +#include <linux/delay.h>
> +#include <linux/interrupt.h>
> +#include <linux/io.h>
> +#include <linux/clk.h>
> +#include <linux/list.h>
> +#include <linux/irq.h>
> +#include <linux/of_device.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/v4l2-async.h>

Please sort the list of headers alphabetically.

> +#include <asm/mach/irq.h>

Why do you need to include this header?

> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +

[snip]

> +static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> + struct imxcsi2_dev *csi2 = sd_to_dev(sd);
> + int i, ret = 0;
> +
> + if (!csi2->src_sd)
> + return -EPIPE;
> + for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
> + if (csi2->sink_sd[i])
> + break;
> + }
> + if (i >= CSI2_NUM_SRC_PADS)

> + return -EPIPE;
> +
> + v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> +

> + if (enable && !csi2->stream_on) {
> + clk_prepare_enable(csi2->pix_clk);

It can complicate the design for you, but in general clk_prepare_enable()
can return an error.

> + ret = imxcsi2_dphy_wait(csi2);
> + if (ret)
> + clk_disable_unprepare(csi2->pix_clk);
> + } else if (!enable && csi2->stream_on) {
> + clk_disable_unprepare(csi2->pix_clk);
> + }
> +
> + if (!ret)
> + csi2->stream_on = enable;

> + return ret;
> +}
> +

[snip]

> +
> +static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
> +{
> + struct device_node *node = csi2->dev->of_node;
> + struct device_node *epnode;
> + struct v4l2_of_endpoint ep;

> + int ret = 0;
> +

> + epnode = of_graph_get_next_endpoint(node, NULL);
> + if (!epnode) {
> + v4l2_err(&csi2->sd, "failed to get endpoint node\n");

> + return -EINVAL;
> + }
> +

> + v4l2_of_parse_endpoint(epnode, &ep);

Do of_node_put(epnode) here and remove 'out' goto label.

> + if (ep.bus_type != V4L2_MBUS_CSI2) {
> + v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");
> + ret = -EINVAL;

> + goto out;
> + }
> +

> + csi2->bus = ep.bus.mipi_csi2;
> +
> + v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
> + v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);
> +out:
> + of_node_put(epnode);

> + return ret;
> +}
> +

[snip]

> +static const struct of_device_id imxcsi2_dt_ids[] = {
> + { .compatible = "fsl,imx-mipi-csi2", },

> + { /* sentinel */ }
> +};

> +MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
> +
> +static struct platform_driver imxcsi2_driver = {

> + .driver = {
> + .name = DEVICE_NAME,
> + .owner = THIS_MODULE,

Please drop .owner assignment.

> + .of_match_table = imxcsi2_dt_ids,
> + },
> + .probe = imxcsi2_probe,
> + .remove = imxcsi2_remove,
> +};
> +
> +module_platform_driver(imxcsi2_driver);
> +
> +MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");

> +MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
> +MODULE_LICENSE("GPL");

> +

Fabio Estevam

unread,

Jan 4, 2017, 10:30:05 AM1/4/17

On Tue, Jan 3, 2017 at 6:57 PM, Steve Longerbeam <slong...@gmail.com> wrote:

> + camera: ov5642@3c {
> + compatible = "ovti,ov5642";
> + pinctrl-names = "default";
> + pinctrl-0 = <&pinctrl_ov5642>;
> + clocks = <&clks IMX6QDL_CLK_CKO>;
> + clock-names = "xclk";
> + reg = <0x3c>;
> + xclk = <24000000>;
> + DOVDD-supply = <&vgen4_reg>; /* 1.8v */
> + AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
> + rev B board is VGEN5 */

Please use vgen3 so that by default we have the valid AVDD-supply for
revC boards which is more recent and more the users have access to.

> + mipi_camera: ov5640@3c {
> + compatible = "ovti,ov5640_mipi";
> + pinctrl-names = "default";
> + pinctrl-0 = <&pinctrl_ov5640>;
> + reg = <0x3c>;
> + clocks = <&clks IMX6QDL_CLK_CKO>;
> + clock-names = "xclk";
> + xclk = <24000000>;
> + DOVDD-supply = <&vgen4_reg>; /* 1.8v */
> + AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
> + rev B board is VGEN5 */

Same here.

> + pinctrl_ov5640: ov5640grp {
> + fsl,pins = <
> + MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x80000000
> + MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x80000000

Please avoid all the 0x80000000 IOMUX settings and replace them by
their real values.

Steve Longerbeam

unread,

Jan 5, 2017, 2:30:05 PM1/5/17

Hi Vladimir,

On 01/04/2017 04:25 AM, Vladimir Zapolskiy wrote:
> Hi Steve,
>
> On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
>> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
>> Both hang off the same i2c2 bus, so they require different (and non-
>> default) i2c slave addresses.
>>
>> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
>>
>> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
>> mipi_csi. It is set to transmit over MIPI virtual channel 1.
>>
>> Note there is a pin conflict with GPIO6. This pin functions as a power
>> input pin to the OV5642, but ENET uses it as the h/w workaround for
>> erratum ERR006687, to wake-up the ARM cores on normal RX and TX packet
>> done events (see 6261c4c8). So workaround 6261c4c8 is reverted here to
>> support the OV5642, and the "fsl,err006687-workaround-present" boolean
>> also must be removed. The result is that the CPUidle driver will no longer
>> allow entering the deep idle states on the sabrelite.
> For me it sounds like a candidate of its own separate change.

Yes, I split out the two partial reverts into a separate commit
("ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687
workaround").

>
>
>
>> +
>> + mipi_camera: ov5640@40 {
> Please reorder device nodes by address value,

done.

> also according to ePAPR
> node names should be generic, labels can be specific:
>
> ov5640: camera@40 {
> ...
> };
>
> ov5642: camera@42 {
> ...
> };

fixed.

>
>> + pinctrl_ipu1_csi0: ipu1grp-csi0 {
> Please rename node name to ipu1csi0grp.

done.

>
>> +
>> + pinctrl_ov5640: ov5640grp {
>> + fsl,pins = <
>> + MX6QDL_PAD_NANDF_D5__GPIO2_IO05 0x000b0
>> + MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
>> + >;
>> + };
>> +
> Indentation issues above, please use tabs instead of spaces.

fixed.

>
> Also please add new pin control groups preserving the alphanimerical order.

done.

Steve

Steve Longerbeam

unread,

Jan 5, 2017, 5:40:04 PM1/5/17

On 01/04/2017 04:33 AM, Vladimir Zapolskiy wrote:
>
>> +
>> + camera: ov5642@3c {
> ov5642: camera@3c

done.

>> + pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>; /* SD1_DAT0 */
>> + reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>; /* SD1_DAT1 */
> Comments about SD1_* pad names are redundant.

sure, removed.

>> + status = "disabled";
> Why is it disabled here?

It's explained in the header. I don't yet have the OV5642 module for
the sabresd for testing, so it is disabled for now.

>> +
>> + mipi_camera: ov5640@3c {
> ov5640: camera@3c

done.

>
>> + pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>; /* SD1_DAT2 */
>> + reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>; /* SD1_CLK */
> Comments about SD1_* pad names are redundant.

removed.

>> +
>> + pinctrl_ipu1_csi0: ipu1grp-csi0 {
> Please rename the node name to ipu1csi0grp.
>
> Please add new pin control groups preserving the alphanimerical order.

done and done.

Steve

Steve Longerbeam

unread,

Jan 5, 2017, 5:40:05 PM1/5/17

On 01/04/2017 04:41 AM, Vladimir Zapolskiy wrote:
> On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
>> +
>> + camera: adv7180@21 {
> adv7180: camera@21

done.

>>
>> + pinctrl_ipu1_csi0: ipu1grp-csi0 {
> Please rename node name to ipu1csi0grp.

done.

Steve

Steve Longerbeam

unread,

Jan 5, 2017, 8:30:04 PM1/5/17

On 01/04/2017 05:33 AM, Vladimir Zapolskiy wrote:
> Hi Steve,
>
> On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
>> Add the core media driver for i.MX SOC.
>>
>> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
>> ---
>> Documentation/devicetree/bindings/media/imx.txt | 205 +++++
> v2 was sent before getting Rob's review comments, but still they
> should be addressed in v3.

yes, those changes will be part of v3 as well.

>
> Also I would suggest to separate device tree binding documentation
> change and place it as the first patch in the series, this should
> make the following DTS changes valid.

done.

>
>> Documentation/media/v4l-drivers/imx.rst | 430 ++++++++++
>> drivers/staging/media/Kconfig | 2 +
>> drivers/staging/media/Makefile | 1 +
>> drivers/staging/media/imx/Kconfig | 8 +
>> drivers/staging/media/imx/Makefile | 6 +
>> drivers/staging/media/imx/TODO | 18 +
>> drivers/staging/media/imx/imx-media-common.c | 985 ++++++++++++++++++++++
>> drivers/staging/media/imx/imx-media-dev.c | 479 +++++++++++
>> drivers/staging/media/imx/imx-media-fim.c | 509 +++++++++++
>> drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
>> drivers/staging/media/imx/imx-media-of.c | 291 +++++++
>> drivers/staging/media/imx/imx-media-of.h | 25 +
>> drivers/staging/media/imx/imx-media.h | 299 +++++++
>> include/media/imx.h | 15 +
>> include/uapi/Kbuild | 1 +
>> include/uapi/linux/v4l2-controls.h | 4 +
>> include/uapi/media/Kbuild | 2 +
>> include/uapi/media/imx.h | 30 +
> Probably Greg should ack the UAPI changes, you may consider
> to split them into a separate patch.

I split out the one-line addition to include/uapi/Kbuild with an empty
include/uapi/media/Kbuild into a new patch "UAPI: Add media UAPI Kbuild
file".
Also added a patch "media: Add userspace header file for i.MX".

>
>
>> +
>> +struct imx_media_subdev *
>> +imx_media_find_subdev_by_sd(struct imx_media_dev *imxmd,
>> + struct v4l2_subdev *sd)
>> +{
>> + struct imx_media_subdev *imxsd;
>> + int i, ret = -ENODEV;
>> +
>> + for (i = 0; i < imxmd->num_subdevs; i++) {
>> + imxsd = &imxmd->subdev[i];
>> + if (sd == imxsd->sd) {
> This can be simplifed:
>
> ...
>
> if (sd == imxsd->sd)
> return imxsd;
> }
>
> return ERR_PTR(-ENODEV);

yep, done.

>
>> +struct imx_media_subdev *
>> +imx_media_find_subdev_by_id(struct imx_media_dev *imxmd, u32 grp_id)
>> +{
>> + struct imx_media_subdev *imxsd;
>> + int i, ret = -ENODEV;
>> +
>> + for (i = 0; i < imxmd->num_subdevs; i++) {
>> + imxsd = &imxmd->subdev[i];
>> + if (imxsd->sd && imxsd->sd->grp_id == grp_id) {
>> + ret = 0;
>> + break;
> This can be simplifed:
>
> ...
>
> if (imxsd->sd && imxsd->sd->grp_id == grp_i)
> return imxsd;
> }
>
> return ERR_PTR(-ENODEV);

done.

>> +#include <video/imx-ipu-v3.h>
>> +#include <media/imx.h>
>> +#include "imx-media.h"
>> +#include "imx-media-of.h"
>> +
>> +#define DEVICE_NAME "imx-media"
> I suppose you don't need this macro.

sure why not, removed.

>
> [snip]
>
>> + */
>> +static int imx_media_create_links(struct imx_media_dev *imxmd)
>> +{
>> + struct imx_media_subdev *local_sd;
>> + struct imx_media_subdev *remote_sd;
>> + struct v4l2_subdev *source, *sink;
>> + struct imx_media_link *link;
>> + struct imx_media_pad *pad;
>> + u16 source_pad, sink_pad;
>> + int num_pads, i, j, k;
>> + int ret = 0;
>> +
>> + for (i = 0; i < imxmd->num_subdevs; i++) {
>> + local_sd = &imxmd->subdev[i];
>> + num_pads = local_sd->num_sink_pads + local_sd->num_src_pads;
>> +
>> + for (j = 0; j < num_pads; j++) {
>> + pad = &local_sd->pad[j];
>> +
>> + for (k = 0; k < pad->num_links; k++) {
>> + link = &pad->link[k];
>> +

>> <snip>

> Indentation depth is quite terrific.

yes, it needs to iterate by subdev, pads in the subdev, then links
in the pad. But I moved the code under the innermost loop into a
function imx_media_create_link(), which creates a single media
link. So it's not so much an eye-sore now.

>
>
>> +static struct platform_driver imx_media_pdrv = {
>> + .probe = imx_media_probe,
>> + .remove = imx_media_remove,
>> + .driver = {
>> + .name = DEVICE_NAME,
>> + .owner = THIS_MODULE,
> Setting of .owner is not needed nowadays IIRC.

a quick look at struct device_driver definition didn't mention that
.owner member is deprecated or not needed, so for now I'll leave
it in place. We can revisit that later.

The i.MX input capture patch will come later. These are just
placeholders until then, the config name could even change.
For now I just removed the above conditional include.

>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/v4l2-of.h>
>> +#include <media/v4l2-ctrls.h>
> Please sort out the list alphabetically.

done.

>> + if (IS_ENABLED(CONFIG_IMX_GPT_ICAP)) {
>> + ret = of_property_read_u32_array(fim_np,
>> + "input-capture-channel",
>> + icap, 2);
>> + if (!ret) {
>> + fim->icap_channel = icap[0];
>> + fim->icap_flags = icap[1];
>> + }
> Should you return error otherwise?

nope, it's an optional property.

done.

>>
>> +static int of_get_port_count(const struct device_node *np)
>> +{
>> + struct device_node *child;
>> + int num = 0;
>> +
>> + /* if this node is itself a port, return 1 */
>> + if (of_node_cmp(np->name, "port") == 0)
>> + return 1;
>> +
>> + for_each_child_of_node(np, child) {
>> + if (of_node_cmp(child->name, "port") == 0)
>> + num++;
>> + }
> Unneeded bracers.

fixed.

>
>> +static struct imx_media_subdev *
>> +of_parse_subdev(struct imx_media_dev *imxmd, struct device_node *sd_np,
>> + bool is_csi_port)
>> +{
>> + struct imx_media_subdev *imxsd;
>> + int i, num_pads, ret;
>> +
>> + if (!of_device_is_available(sd_np)) {
>> + dev_dbg(imxmd->dev, "%s: %s not enabled\n", __func__,
>> + sd_np->name);
>> + return NULL;
>> + }
>> +
>> + /* register this subdev with async notifier */
>> + imxsd = imx_media_add_async_subdev(imxmd, sd_np, NULL);
>> + if (!imxsd)
>> + return NULL;
>> + if (IS_ERR(imxsd))
>> + return imxsd;
> if (IS_ERR_OR_NULL(imxsd))
> return imxsd;

yep, done.

>> +
>> + if (imxsd->num_sink_pads == 0) {
>> + /* this might be a sensor */
>> + of_parse_sensor(imxmd, imxsd, sd_np);
>> + }
> Unneeded bracers.

ok removed.

>
>> +
>> + for (i = 0; i < num_pads; i++) {
>> + struct device_node *epnode = NULL, *port, *remote_np;
>> + struct imx_media_subdev *remote_imxsd;
>> + struct imx_media_pad *pad;
>> + int remote_pad;
> Too deep indentation, may be move the cycle body into a separate function?

in this case I prefer not to, of_parse_subdev() is called recursively, and I
think it's bad form to hide that fact by moving the recursive call into a
separate function. The indentation is not that deep, only two loops deep.

>
>> +
>> + /* init this pad */
>> + pad = &imxsd->pad[i];
>> + pad->pad.flags = (i < imxsd->num_sink_pads) ?
>> + MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
>> +
>> + if (is_csi_port)
>> + port = (i < imxsd->num_sink_pads) ? sd_np : NULL;
>> + else
>> + port = of_graph_get_port_by_id(sd_np, i);
>> + if (!port)
>> + continue;
>> +
>> + while ((epnode = of_get_next_child(port, epnode))) {
> Please reuse for_each_child_of_node() here.

done.

>
>> + of_get_remote_pad(epnode, &remote_np, &remote_pad);
>> + if (!remote_np) {
>> + of_node_put(epnode);
> Please remove of_node_put() here, of_get_next_child() does it.

oops, good catch, fixed.

>
>> + continue;
>> + }
>> +
>> + ret = of_add_pad_link(imxmd, pad, sd_np, remote_np,
>> + i, remote_pad);
>> + if (ret) {
>> + imxsd = ERR_PTR(ret);
>> + break;
>> + }
>> +
>> + if (i < imxsd->num_sink_pads) {
>> + /* follow sink endpoints upstream */
>> + remote_imxsd = of_parse_subdev(imxmd,
>> + remote_np,
>> + false);
>> + if (IS_ERR(remote_imxsd)) {
>> + imxsd = remote_imxsd;
>> + break;
>> + }
>> + }
>> +
>> + of_node_put(remote_np);
>> + of_node_put(epnode);

also removed this put of epnode.

>>
>> +
>> +int imx_media_of_parse(struct imx_media_dev *imxmd,
>> + struct imx_media_subdev *(*csi)[4],
>> + struct device_node *np)
>> +{
>> + struct device_node *csi_np;
>> + struct imx_media_subdev *lcsi;
> Please swap two lines above to get the reverse christmas tree ordering.

done.

>
>> + u32 ipu_id, csi_id;
>> + int i, ret;
>> +
>> + for (i = 0; ; i++) {
>> + csi_np = of_parse_phandle(np, "ports", i);
>> + if (!csi_np)
>> + break;
>> +
>> + lcsi = of_parse_subdev(imxmd, csi_np, true);
>> + if (IS_ERR(lcsi)) {
>> + ret = PTR_ERR(lcsi);
>> + goto err_put;
>> + }
>> +
>> + of_property_read_u32(csi_np, "reg", &csi_id);
> Not sure if it is safe enough to ignore return value and potentially
> left csi_id uninitialized.

The CSI nodes are port nodes, and the reg property is required,
so I added a check and error return here.

>
>> + ipu_id = of_alias_get_id(csi_np->parent, "ipu");
>> +
>> + if (ipu_id > 1 || csi_id > 1) {
>> + dev_err(imxmd->dev, "%s: invalid ipu/csi id (%u/%u)\n",
>> + __func__, ipu_id, csi_id);
>> + ret = -EINVAL;
>> + goto err_put;
>> + }
>> +
>> + of_node_put(csi_np);
> You can put the node right after of_alias_get_id() call, then in case
> of error return right from the if block and remove the goto label.

done.

actually imx-media-of.h isn't really needed, I just moved those prototypes
into imx-media.h and removed imx-media-of.h.

>> diff --git a/drivers/staging/media/imx/imx-media.h b/drivers/staging/media/imx/imx-media.h
>> new file mode 100644
>> index 0000000..6a018a9
>> --- /dev/null
>> +++ b/drivers/staging/media/imx/imx-media.h
>> @@ -0,0 +1,299 @@
>> +/*
>> + * V4L2 Media Controller Driver for Freescale i.MX5/6 SOC
>> + *
>> + * Copyright (c) 2016 Mentor Graphics Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + */
>> +#ifndef _IMX_MEDIA_H
>> +#define _IMX_MEDIA_H
> Please insert here an empty line to improve readability.

done.

>
>> +#include <media/v4l2-device.h>
>> +#include <media/v4l2-subdev.h>
>> +#include <media/v4l2-ctrls.h>
>> +#include <media/v4l2-of.h>
> Please sort out the list alphabetically.

done.

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 1:10:05 PM1/6/17

done.

I failed to cleanup in this path, so it is now:

ret = ipu_csi_enable(priv->csi);

if (ret) {
v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);

goto fim_off;
}

return 0;
fim_off:
if (priv->fim)

imx_media_fim_set_stream(priv->fim, priv->sensor, false);

return ret;

>> +
>> +static int csi_link_setup(struct media_entity *entity,
>> + const struct media_pad *local,
>> + const struct media_pad *remote, u32 flags)
>> +{
>> + struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
>> + struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> + struct v4l2_subdev *remote_sd;
>> +
>> + dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
>> + local->entity->name);
>> +
>> + remote_sd = media_entity_to_v4l2_subdev(remote->entity);
>> +
>> + if (local->flags & MEDIA_PAD_FL_SINK) {
>> + if (flags & MEDIA_LNK_FL_ENABLED) {
>> + if (priv->src_sd)
>> + return -EBUSY;
>> + priv->src_sd = remote_sd;
>> + } else {
>> + priv->src_sd = NULL;
>> + return 0;
> You can remove the return above.

right, fixed.

>
>> +
>> + ret = v4l2_async_register_subdev(&priv->sd);
>> + if (ret)
>> + goto free_ctrls;
>> +
>> + return 0;
>> +free_ctrls:
>> + v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
>> + return ret;
> This is a functionally equal and simplified version:
>
> if (ret)
> v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
>
> return ret;

thanks, done.

>> +
>> +static struct platform_driver imx_csi_driver = {
>> + .probe = imx_csi_probe,
>> + .remove = imx_csi_remove,
>> + .id_table = imx_csi_ids,
>> + .driver = {
>> + .name = "imx-ipuv3-csi",
>> + .owner = THIS_MODULE,
> Please drop .owner.

ok, I tested this and there are no regressions, done
for all modules.

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 1:20:07 PM1/6/17

On 01/04/2017 06:23 AM, Vladimir Zapolskiy wrote:
> On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
>> This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO
>> Controller module. Video frames are received from the CSI and can
>> be routed to various sinks including the i.MX Image Converter for
>> scaling, color-space conversion, motion compensated deinterlacing,
>> and image rotation.
>>
>> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
>> ---
>> drivers/staging/media/imx/Makefile | 1 +
>> drivers/staging/media/imx/imx-smfc.c | 739 +++++++++++++++++++++++++++++++++++
>> 2 files changed, 740 insertions(+)
>> create mode 100644 drivers/staging/media/imx/imx-smfc.c
>>
>> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
>> index 133672a..3559d7b 100644
>> --- a/drivers/staging/media/imx/Makefile
>> +++ b/drivers/staging/media/imx/Makefile
>> @@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
>> obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
>>
>> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
> May be
>
> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o imx-smfc.o

I'd prefer to keep them on separate lines, to indicate they
are all built as separate modules.

done.

>
>> +static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
>> +{
>> + struct imx_smfc_priv *priv = dev_id;
>> + struct imx_media_dma_buf *done, *next;
>> + unsigned long flags;
>> +
>> + spin_lock_irqsave(&priv->irqlock, flags);
> spin_lock(&priv->irqlock) should be sufficient.

yes thanks.

>
>> +
>> +static const struct platform_device_id imx_smfc_ids[] = {
>> + { .name = "imx-ipuv3-smfc" },
>> + { },
>> +};
>> +MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
>> +
>> +static struct platform_driver imx_smfc_driver = {
>> + .probe = imx_smfc_probe,
>> + .remove = imx_smfc_remove,
>> + .id_table = imx_smfc_ids,
>> + .driver = {
>> + .name = "imx-ipuv3-smfc",
>> + .owner = THIS_MODULE,
> You can drop owner assignment.

done.

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 1:40:07 PM1/6/17

On 01/04/2017 06:48 AM, Vladimir Zapolskiy wrote:
> On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
>
>> +
>> + ret = ic_ops[priv->task_id]->init(priv);
>> + if (ret)
>> + return ret;
>> +
>> + ret = v4l2_async_register_subdev(&priv->sd);
>> + if (ret)
>> + goto remove;
>> +
>> + return 0;
>> +remove:
>> + ic_ops[priv->task_id]->remove(priv);
>> + return ret;
> if (ret)
> ic_ops[priv->task_id]->remove(priv);
>
> return ret;
>
> as an alternative.

done.

>
>
> +
> +static struct platform_driver imx_ic_driver = {
> + .probe = imx_ic_probe,
> + .remove = imx_ic_remove,
> + .id_table = imx_ic_ids,
> + .driver = {
> + .name = "imx-ipuv3-ic",
> + .owner = THIS_MODULE,
> Please drop .owner assignment.

done.

>
>>
>> +
>> + priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
>> + sizeof(*priv->out_run), GFP_KERNEL);
>> + if (!priv->out_run) {
>> + v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
> In OOM situation the core will report it, probably you can drop the message.

with a backtrace? I can't remember, if not I'd like to keep this.

done.

>
>> +static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
>> +{
>> + struct prpenc_priv *priv = dev_id;
>> + struct imx_media_dma_buf *done, *next;
>> + struct ipuv3_channel *channel;
>> + unsigned long flags;
>> +
>> + spin_lock_irqsave(&priv->irqlock, flags);
> Here spin_lock(&priv->irqlock) should be sufficient.

done.

>> +
>> + ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);
>> + if (ret)
>> + goto free_ctrls;
>> +
>> + return 0;
>> +free_ctrls:
>> + v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
>> + return ret;
> if (ret)
> v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
>
> return ret;
>
> version is shorter.

done.

>
>> +/* prpvf_out_ch EOF interrupt (progressive frame ready) */
>> +static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
>> +{
>> + struct prpvf_priv *priv = dev_id;
>> + struct imx_media_dma_buf *done;
>> + unsigned long flags;
>> +
>> + spin_lock_irqsave(&priv->irqlock, flags);
>
> Here spin_lock(&priv->irqlock) should be sufficient.

done.

>
>> diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
>> new file mode 100644
>> index 0000000..9aed5f5
>> --- /dev/null
>> +++ b/drivers/staging/media/imx/imx-ic.h
>> @@ -0,0 +1,36 @@
>> +/*
>> + * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
>> + *
>> + * Copyright (c) 2016 Mentor Graphics Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + */
>> +#ifndef _IMX_IC_H
>> +#define _IMX_IC_H
>> +
> Please add header files or declarations of all used structs.

done (it only needs v4l2-subdev.h).

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 1:50:06 PM1/6/17

On 01/04/2017 06:55 AM, Vladimir Zapolskiy wrote:
> On 01/03/2017 10:57 PM, Steve Longerbeam wrote:
>>
>> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
>> index d2a962c..fe9e992 100644
>> --- a/drivers/staging/media/imx/Makefile
>> +++ b/drivers/staging/media/imx/Makefile
>> @@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
>>
>> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
>> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
>> -
>> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
> obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o imx-csi.o imx-smfc.o
>
> as an option.

I prefer to keep on separate lines as explained earlier.

done.

>> +#include <video/imx-ipu-v3.h>
>> +#include <media/imx.h>
>> +#include "imx-media.h"
>> +
>> +#define DEVICE_NAME "imx-media-camif"
> I would propose to drop this macro.

done.

>> +
>> +#define CAMIF_DQ_TIMEOUT 5000
> Add a comment about time unit?

actually that was ancient and no longer used, removed.

>
>> +
>> +struct camif_priv;
>> +
> This is a leftover apparently.

ditto, removed.

>> +
>> + ret = v4l2_async_register_subdev(&priv->sd);
>> + if (ret)
>> + goto free_ctrls;
>> +
>> + return 0;
>> +free_ctrls:
>> + v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
>> + return ret;
> A shorter version:
>
> if (ret)
> v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
>
> return ret;

done.

>> +
>> +static struct platform_driver imx_camif_driver = {
>> + .probe = camif_probe,
>> + .remove = camif_remove,
>> + .driver = {
>> + .name = DEVICE_NAME,
>> + .owner = THIS_MODULE,
> Please drop the owner assignment.

done.

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 2:00:05 PM1/6/17

done.

>
>> +#include <asm/mach/irq.h>
> Why do you need to include this header?

good question. In fact this include list was in need of a lot of pruning,
so I paired it down to the essentials.

>> +static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> + struct imxcsi2_dev *csi2 = sd_to_dev(sd);
>> + int i, ret = 0;
>> +
>> + if (!csi2->src_sd)
>> + return -EPIPE;
>> + for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
>> + if (csi2->sink_sd[i])
>> + break;
>> + }
>> + if (i >= CSI2_NUM_SRC_PADS)
>> + return -EPIPE;
>> +
>> + v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
>> +
>> + if (enable && !csi2->stream_on) {
>> + clk_prepare_enable(csi2->pix_clk);
> It can complicate the design for you, but in general clk_prepare_enable()
> can return an error.

I added an error check and reorganized a little.

>> +
>> +static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
>> +{
>> + struct device_node *node = csi2->dev->of_node;
>> + struct device_node *epnode;
>> + struct v4l2_of_endpoint ep;
>> + int ret = 0;
>> +
>> + epnode = of_graph_get_next_endpoint(node, NULL);
>> + if (!epnode) {
>> + v4l2_err(&csi2->sd, "failed to get endpoint node\n");
>> + return -EINVAL;
>> + }
>> +
>> + v4l2_of_parse_endpoint(epnode, &ep);
> Do of_node_put(epnode) here and remove 'out' goto label.

done.

>> +static const struct of_device_id imxcsi2_dt_ids[] = {
>> + { .compatible = "fsl,imx-mipi-csi2", },
>> + { /* sentinel */ }
>> +};
>> +MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
>> +
>> +static struct platform_driver imxcsi2_driver = {
>> + .driver = {
>> + .name = DEVICE_NAME,
>> + .owner = THIS_MODULE,
> Please drop .owner assignment.

done.

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 7:30:05 PM1/6/17

On 01/04/2017 07:26 AM, Fabio Estevam wrote:
> On Tue, Jan 3, 2017 at 6:57 PM, Steve Longerbeam <slong...@gmail.com> wrote:
>
>> + camera: ov5642@3c {
>> + compatible = "ovti,ov5642";
>> + pinctrl-names = "default";
>> + pinctrl-0 = <&pinctrl_ov5642>;
>> + clocks = <&clks IMX6QDL_CLK_CKO>;
>> + clock-names = "xclk";
>> + reg = <0x3c>;
>> + xclk = <24000000>;
>> + DOVDD-supply = <&vgen4_reg>; /* 1.8v */
>> + AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
>> + rev B board is VGEN5 */
> Please use vgen3 so that by default we have the valid AVDD-supply for
> revC boards which is more recent and more the users have access to.

done.

>
>> + mipi_camera: ov5640@3c {
>> + compatible = "ovti,ov5640_mipi";
>> + pinctrl-names = "default";
>> + pinctrl-0 = <&pinctrl_ov5640>;
>> + reg = <0x3c>;
>> + clocks = <&clks IMX6QDL_CLK_CKO>;
>> + clock-names = "xclk";
>> + xclk = <24000000>;
>> + DOVDD-supply = <&vgen4_reg>; /* 1.8v */
>> + AVDD-supply = <&vgen5_reg>; /* 2.8v, rev C board is VGEN3
>> + rev B board is VGEN5 */
> Same here.

done.

>
>> + pinctrl_ov5640: ov5640grp {
>> + fsl,pins = <
>> + MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x80000000
>> + MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x80000000
> Please avoid all the 0x80000000 IOMUX settings and replace them by
> their real values.

yeah, finally got around to this, done!

Steve

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:04 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

@@ -136,6 +136,9 @@
reg = <0x32>;
gpio-controller;
#gpio-cells = <2>;

+ pinctrl-names = "default";

+ pinctrl-0 = <&pinctrl_max7310>;
+ reset-gpios = <&gpio1 15 GPIO_ACTIVE_LOW>;
};

max7310_c: gpio@34 {
@@ -442,6 +445,12 @@
>;
};

+ pinctrl_max7310: max7310grp {
+ fsl,pins = <

+ MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0

+ >;
+ };
+
pinctrl_pwm3: pwm1grp {
fsl,pins = <
MX6QDL_PAD_SD4_DAT1__PWM3_OUT 0x1b0b1
--

2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:04 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 61 ++++++++++++++++++++++++++++++++
1 file changed, 61 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 495709f..7999857 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -124,6 +124,24 @@
#size-cells = <0>;
reg = <1>;

+ adv7180: camera@21 {

+ compatible = "adi,adv7180";
+ reg = <0x21>;
+ powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+ interrupt-parent = <&gpio1>;
+ interrupts = <27 0x8>;
+ inputs = <0x00 0x02>;
+ input-names = "ADV7180 Composite on Ain1",
+ "ADV7180 Composite on Ain3";
+
+ port {
+ adv7180_to_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;
+ };
+ };
+ };

+
max7310_a: gpio@30 {
compatible = "maxim,max7310";
reg = <0x30>;
@@ -151,6 +169,25 @@

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+ bus-width = <8>;
+};
+
+&ipu1_csi0 {

+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+ /* enable frame interval monitor on this port */
+ fim {
+ status = "okay";
+ };

+};
+
&clks {
assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
<&clks IMX6QDL_PLL4_BYPASS>,

@@ -445,6 +482,30 @@
>;
};

+ pinctrl_ipu1_csi0: ipu1csi0grp {
+ fsl,pins = <
+ MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x1b0b0
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x1b0b0
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x1b0b0
+ >;
+ };
+
pinctrl_max7310: max7310grp {
fsl,pins = <
MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:04 PM1/6/17

From: Philipp Zabel <p.z...@pengutronix.de>

This patch adds the device tree graph connecting the input multiplexers
to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU
multiplexers are added as children of the iomuxc-gpr syscon device node.
On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2
CSI1 allow to select between CSI0/1 parallel input pads and the MIPI
CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1
CSI1 allow to select between CSI0/1 parallel input pads and any of the
four MIPI CSI-2 virtual channels.

Signed-off-by: Philipp Zabel <p.z...@pengutronix.de>

--

- Removed some dangling/unused endpoints (ipu2_csi0_from_csi2ipu)
- Renamed the mipi virtual channel endpoint labels, from "mipi_csiX_..."
to "mipi_vcX...".
- Added input endpoint anchors to the video muxes for the connections
from parallel sensors.
- Added input endpoint anchors to the mipi_csi for the connections from
mipi csi-2 sensors.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6dl.dtsi | 187 +++++++++++++++++++++++++++++++++++++++++
arch/arm/boot/dts/imx6q.dtsi | 123 +++++++++++++++++++++++++++
arch/arm/boot/dts/imx6qdl.dtsi | 10 ++-
3 files changed, 319 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi
index 1ade195..0849e85 100644
--- a/arch/arm/boot/dts/imx6dl.dtsi
+++ b/arch/arm/boot/dts/imx6dl.dtsi
@@ -181,6 +181,193 @@
"di0", "di1";
};

+&gpr {
+ ipu1_csi0_mux: ipu1_csi0_mux@34 {
+ compatible = "video-multiplexer";
+ reg = <0x34>;
+ bit-mask = <0x7>;
+ bit-shift = <0>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu1_csi0_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ ipu1_csi0_mux_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi0_mux_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+
+ ipu1_csi0_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+
+ ipu1_csi0_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@5 {
+ reg = <5>;
+
+ ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+ remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+ };
+ };
+ };
+
+ ipu1_csi1_mux: ipu1_csi1_mux@34 {
+ compatible = "video-multiplexer";
+ reg = <0x34>;
+ bit-mask = <0x7>;
+ bit-shift = <3>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu1_csi1_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ ipu1_csi1_mux_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi1_mux_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+
+ ipu1_csi1_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu1_csi1_mux>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+
+ ipu1_csi1_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@5 {
+ reg = <5>;
+
+ ipu1_csi1_mux_to_ipu1_csi1: endpoint {
+ remote-endpoint = <&ipu1_csi1_from_ipu1_csi1_mux>;

+ };
+ };
+ };
+};
+

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ mipi_vc3_to_ipu1_csi0_mux: endpoint@0 {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc3>;
+ };
+
+ mipi_vc3_to_ipu1_csi1_mux: endpoint@1 {
+ remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc3>;
+ };
+ };
+};
+
&vpu {
compatible = "fsl,imx6dl-vpu", "cnm,coda960";
};
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..9b2ca32 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -143,10 +143,18 @@

ipu2_csi0: port@0 {
reg = <0>;
+
+ ipu2_csi0_from_mipi_vc2: endpoint {
+ remote-endpoint = <&mipi_vc2_to_ipu2_csi0>;
+ };
};

ipu2_csi1: port@1 {
reg = <1>;
+
+ ipu2_csi1_from_ipu2_csi1_mux: endpoint {
+ remote-endpoint = <&ipu2_csi1_mux_to_ipu2_csi1>;
+ };
};

ipu2_di0: port@2 {
@@ -266,6 +274,80 @@
};
};

+&gpr {
+ ipu1_csi0_mux: ipu1_csi0_mux@4 {
+ compatible = "video-multiplexer";
+ reg = <0x04>;
+ bit-mask = <1>;
+ bit-shift = <19>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu1_csi0_mux_from_mipi_vc0: endpoint {
+ remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ ipu1_csi0_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+ remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+ };
+ };
+ };
+
+ ipu2_csi1_mux: ipu2_csi1_mux@4 {
+ compatible = "video-multiplexer";
+ reg = <0x04>;
+ bit-mask = <1>;
+ bit-shift = <20>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+

+ port@0 {
+ reg = <0>;
+
+ ipu2_csi1_mux_from_mipi_vc3: endpoint {
+ remote-endpoint = <&mipi_vc3_to_ipu2_csi1_mux>;
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ ipu2_csi1_mux_from_parallel_sensor: endpoint {
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ ipu2_csi1_mux_to_ipu2_csi1: endpoint {
+ remote-endpoint = <&ipu2_csi1_from_ipu2_csi1_mux>;

+ };
+ };
+ };
+};
+

+&ipu1_csi1 {
+ ipu1_csi1_from_mipi_vc1: endpoint {
+ remote-endpoint = <&mipi_vc1_to_ipu1_csi1>;
+ };
+};
+
&ldb {
clocks = <&clks IMX6QDL_CLK_LDB_DI0_SEL>, <&clks IMX6QDL_CLK_LDB_DI1_SEL>,
<&clks IMX6QDL_CLK_IPU1_DI0_SEL>, <&clks IMX6QDL_CLK_IPU1_DI1_SEL>,
@@ -312,6 +394,47 @@
};
};

+&mipi_csi {
+ port@0 {
+ reg = <0>;
+
+ mipi_csi_from_mipi_sensor: endpoint {
+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ mipi_vc0_to_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ mipi_vc1_to_ipu1_csi1: endpoint {
+ remote-endpoint = <&ipu1_csi1_from_mipi_vc1>;
+ };
+ };
+
+ port@3 {
+ reg = <3>;
+
+ mipi_vc2_to_ipu2_csi0: endpoint {
+ remote-endpoint = <&ipu2_csi0_from_mipi_vc2>;
+ };
+ };
+
+ port@4 {
+ reg = <4>;
+
+ mipi_vc3_to_ipu2_csi1_mux: endpoint {
+ remote-endpoint = <&ipu2_csi1_mux_from_mipi_vc3>;
+ };
+ };
+};
+
&mipi_dsi {
ports {
port@2 {
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 42926e9..010388c 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -799,8 +799,10 @@
};

gpr: iomuxc-gpr@020e0000 {
- compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+ compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
reg = <0x020e0000 0x38>;

+ #address-cells = <1>;
+ #size-cells = <0>;

};

iomuxc: iomuxc@020e0000 {
@@ -1127,6 +1129,8 @@
mipi_csi: mipi@021dc000 {
compatible = "fsl,imx6-mipi-csi2";
reg = <0x021dc000 0x4000>;

+ #address-cells = <1>;
+ #size-cells = <0>;

interrupts = <0 100 0x04>, <0 101 0x04>;
clocks = <&clks IMX6QDL_CLK_HSI_TX>,
<&clks IMX6QDL_CLK_VIDEO_27M>,
@@ -1232,6 +1236,10 @@

ipu1_csi0: port@0 {
reg = <0>;
+
+ ipu1_csi0_from_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_to_ipu1_csi0>;
+ };
};

ipu1_csi1: port@1 {
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6dl-sabrelite.dts | 5 ++
arch/arm/boot/dts/imx6q-sabrelite.dts | 6 ++
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
3 files changed, 129 insertions(+)

diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
index 0f06ca5..fec2524 100644
--- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
+++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts
@@ -48,3 +48,8 @@
model = "Freescale i.MX6 DualLite SABRE Lite Board";
compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
};
+
+&ipu1_csi1_from_ipu1_csi1_mux {

+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
+

diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
index 795b5a5..bca9fed 100644
--- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi
@@ -39,6 +39,8 @@
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
+
+#include <dt-bindings/clock/imx6qdl-clock.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/input/input.h>

@@ -96,6 +98,15 @@
};
};

+ mipi_xclk: mipi_xclk {
+ compatible = "pwm-clock";
+ #clock-cells = <0>;
+ clock-frequency = <22000000>;
+ clock-output-names = "mipi_pwm3";
+ pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */

+ status = "okay";
+ };
+

gpio-keys {
compatible = "gpio-keys";
pinctrl-names = "default";
@@ -220,6 +231,22 @@

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;

+ data-shift = <12>; /* Lines 19:12 used */
+ hsync-active = <1>;
+ vync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;

+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+

&audmux {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_audmux>;

@@ -299,6 +326,52 @@

pinctrl-names = "default";
pinctrl-0 = <&pinctrl_i2c2>;

status = "okay";
+
+ ov5640: camera@40 {
+ compatible = "ovti,ov5640";
+ pinctrl-names = "default";

+ pinctrl-0 = <&pinctrl_ov5640>;
+ clocks = <&mipi_xclk>;

+ clock-names = "xclk";

+ reg = <0x40>;
+ xclk = <22000000>;
+ reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+ pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
+
+ port {

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ ov5640_to_mipi_csi: endpoint@1 {
+ reg = <1>;

+ remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;

+ };
+ };
+ };
+
+ ov5642: camera@42 {
+ compatible = "ovti,ov5642";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5642>;
+ clocks = <&clks IMX6QDL_CLK_CKO2>;
+ clock-names = "xclk";
+ reg = <0x42>;
+ xclk = <24000000>;

+ reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+ pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
+ gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;

+
+ port {
+ ov5642_to_ipu1_csi0_mux: endpoint {

+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;

+ hsync-active = <1>;
+ vsync-active = <1>;
+ };
+ };
+ };
};

&i2c3 {
@@ -412,6 +485,23 @@

>;
};

+ pinctrl_ipu1_csi0: ipu1csi0grp {
+ fsl,pins = <

+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x1b0b0
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x1b0b0
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x1b0b0

+ MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
+ >;
+ };
+
pinctrl_j15: j15grp {
fsl,pins = <
MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
@@ -445,6 +535,22 @@
>;
};

+ pinctrl_ov5640: ov5640grp {
+ fsl,pins = <

+ MX6QDL_PAD_NANDF_D5__GPIO2_IO05 0x000b0
+ MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
+ >;
+ };
+

+ pinctrl_ov5642: ov5642grp {
+ fsl,pins = <

+ MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+ MX6QDL_PAD_GPIO_6__GPIO1_IO06 0x1b0b0
+ MX6QDL_PAD_GPIO_8__GPIO1_IO08 0x130b0
+ MX6QDL_PAD_GPIO_3__CCM_CLKO2 0x000b0
+ >;
+ };
+
pinctrl_pwm1: pwm1grp {
fsl,pins = <
MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
@@ -601,3 +707,15 @@
vmmc-supply = <&reg_3p3v>;
status = "okay";
};
+
+&mipi_csi {

+ status = "okay";
+};
+

+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+ remote-endpoint = <&ov5640_to_mipi_csi>;

+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
+

--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

This is a media entity subdevice for the i.MX Camera
Serial Interface module.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Kconfig | 13 +
drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
3 files changed, 659 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-csi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index bfde58d..ce2d2c8 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
Say yes here to enable support for video4linux media controller
driver for the i.MX5/6 SOC.

+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA
+ tristate "i.MX5/6 Camera driver"
+ depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+ select VIDEOBUF2_DMA_CONTIG
+ default y
+ ---help---
+ A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index ef9f11b..133672a 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \

obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
new file mode 100644
index 0000000..64ef862
--- /dev/null
+++ b/drivers/staging/media/imx/imx-csi.c
@@ -0,0 +1,644 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>

+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>

+#include <media/videobuf2-dma-contig.h>

+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+

+#define CSI_NUM_PADS 2
+
+struct csi_priv {

+ struct device *dev;

+ struct ipu_soc *ipu;
+ struct imx_media_dev *md;

+ struct v4l2_subdev sd;

+
+ return 0;
+}

+
+static int csi_start(struct csi_priv *priv)
+{
+ int ret;
+
+ if (!priv->sensor) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return -EINVAL;
+ }
+

+ ret = csi_setup(priv);

+ if (ret)
+ return ret;
+

+ /* start the frame interval monitor */

+ if (priv->fim) {

+ ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);

+ if (ret)
+ return ret;
+ }
+

+ ret = ipu_csi_enable(priv->csi);
+ if (ret) {
+ v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);

+ goto fim_off;

+ }
+
+ return 0;

+
+fim_off:
+ if (priv->fim)
+ imx_media_fim_set_stream(priv->fim, priv->sensor, false);

+ return ret;
+}
+

+static void csi_stop(struct csi_priv *priv)
+{

+ /* stop the frame interval monitor */
+ if (priv->fim)

+ imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+ ipu_csi_disable(priv->csi);
+}
+

+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+

+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !priv->stream_on)
+ ret = csi_start(priv);

+ else if (!enable && priv->stream_on)
+ csi_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+

+ v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+ if (priv->fim && on != priv->power_on)

+ ret = imx_media_fim_set_power(priv->fim, on);

+
+ if (!ret)
+ priv->power_on = on;

+ return ret;
+}
+

+static int csi_link_setup(struct media_entity *entity,
+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ struct v4l2_subdev *remote_sd;
+
+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SINK) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;
+ }

+
+ return 0;
+ }
+

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ return 0;
+ }
+
+ /* set CSI destination */
+ switch (remote_sd->grp_id) {
+ case IMX_MEDIA_GRP_ID_SMFC0:
+ case IMX_MEDIA_GRP_ID_SMFC1:
+ case IMX_MEDIA_GRP_ID_SMFC2:
+ case IMX_MEDIA_GRP_ID_SMFC3:
+ priv->dest = IPU_CSI_DEST_IDMAC;

+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPVF:

+ priv->dest = IPU_CSI_DEST_VDIC;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPENC:
+ priv->dest = IPU_CSI_DEST_IC;
+ break;
+ default:

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int csi_link_validate(struct v4l2_subdev *sd,
+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ bool is_csi2;
+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);
+ priv->sensor = NULL;

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ &priv->format_mbus[priv->input_pad]);
+ }

+
+ /* select either parallel or MIPI-CSI2 as input to CSI */
+ ipu_set_csi_src_mux(priv->ipu, priv->csi_id, is_csi2);

+
+ return 0;
+}
+

+static int csi_eof_isr(struct v4l2_subdev *sd, u32 status, bool *handled)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+

+ if (priv->fim) {
+ struct timespec cur_ts;
+
+ ktime_get_ts(&cur_ts);
+ /* call frame interval monitor */
+ imx_media_fim_eof_monitor(priv->fim, &cur_ts);

+ }
+
+ return 0;

+}
+
+static int csi_try_crop(struct csi_priv *priv, struct v4l2_rect *crop)
+{
+ struct v4l2_mbus_framefmt *infmt;
+ struct imx_media_subdev *sensor;
+ v4l2_std_id std;
+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ ret = v4l2_subdev_call(sensor->sd, video, g_mbus_config,
+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+

+ if (ret)
+ return ret;

+ }
+
+ return 0;

+}
+
+static int csi_get_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);
+

+ if (sdformat->pad >= CSI_NUM_PADS)
+ return -EINVAL;
+
+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int csi_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;
+ struct v4l2_rect crop;
+ int ret;
+
+ if (sdformat->pad >= CSI_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;

+
+ infmt = &priv->format_mbus[priv->input_pad];

+ outfmt = &priv->format_mbus[priv->output_pad];
+
+ if (sdformat->pad == priv->output_pad) {
+ sdformat->format.code = infmt->code;
+ sdformat->format.field = infmt->field;
+ crop.left = priv->crop.left;
+ crop.top = priv->crop.top;
+ crop.width = sdformat->format.width;
+ crop.height = sdformat->format.height;
+ ret = csi_try_crop(priv, &crop);

+ if (ret)
+ return ret;

+ sdformat->format.width = crop.width;
+ sdformat->format.height = crop.height;
+ }
+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ /* Update the crop window if this is output pad */
+ if (sdformat->pad == priv->output_pad)
+ priv->crop = crop;

+ }
+
+ return 0;

+}
+
+static int csi_get_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_selection *sel)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt;
+
+ if (sel->pad != priv->output_pad)
+ return -EINVAL;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int csi_set_selection(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_selection *sel)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *outfmt;
+ int ret;
+
+ if (sel->pad != priv->output_pad ||
+ sel->target != V4L2_SEL_TGT_CROP)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+
+ /*
+ * Modifying the crop rectangle always changes the format on the source
+ * pad. If the KEEP_CONFIG flag is set, just return the current crop
+ * rectangle.
+ */
+ if (sel->flags & V4L2_SEL_FLAG_KEEP_CONFIG) {
+ sel->r = priv->crop;
+ if (sel->which == V4L2_SUBDEV_FORMAT_TRY)
+ cfg->try_crop = sel->r;

+ return 0;
+ }
+

+ outfmt = &priv->format_mbus[priv->output_pad];
+
+ ret = csi_try_crop(priv, &sel->r);

+ if (ret)
+ return ret;
+

+ }
+
+ return 0;

+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int csi_registered(struct v4l2_subdev *sd)
+{

+ struct csi_priv *priv = v4l2_get_subdevdata(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i, ret;
+
+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+ if (IS_ERR(imxsd)) {

+ ret = PTR_ERR(imxsd);
+ goto put_csi;
+ }
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+ ret = -EINVAL;
+ goto put_csi;
+ }
+
+ for (i = 0; i < CSI_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;
+
+ /* set a default mbus format */
+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+ 640, 480, 0, V4L2_FIELD_NONE,

+ NULL);
+ if (ret)
+ goto put_csi;
+ }
+
+ priv->fim = imx_media_fim_init(&priv->sd);
+ if (IS_ERR(priv->fim)) {
+ ret = PTR_ERR(priv->fim);
+ goto put_csi;
+ }
+
+ ret = media_entity_pads_init(&sd->entity, CSI_NUM_PADS, priv->pad);
+ if (ret)
+ goto free_fim;
+
+ return 0;
+free_fim:
+ if (priv->fim)
+ imx_media_fim_free(priv->fim);
+put_csi:
+ ipu_csi_put(priv->csi);

+ return ret;
+}
+

+static struct media_entity_operations csi_entity_ops = {
+ .link_setup = csi_link_setup,
+ .link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_core_ops csi_core_ops = {
+ .s_power = csi_s_power,
+ .interrupt_service_routine = csi_eof_isr,
+};
+
+static struct v4l2_subdev_video_ops csi_video_ops = {
+ .s_stream = csi_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops csi_pad_ops = {
+ .get_fmt = csi_get_fmt,
+ .set_fmt = csi_set_fmt,
+ .get_selection = csi_get_selection,
+ .set_selection = csi_set_selection,
+ .link_validate = csi_link_validate,
+};
+
+static struct v4l2_subdev_ops csi_subdev_ops = {
+ .core = &csi_core_ops,
+ .video = &csi_video_ops,
+ .pad = &csi_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops csi_internal_ops = {
+ .registered = csi_registered,
+};
+
+static int imx_csi_probe(struct platform_device *pdev)
+{

+ struct ipu_client_platformdata *pdata;
+ struct csi_priv *priv;

+ int ret;
+

+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+

+ platform_set_drvdata(pdev, &priv->sd);

+ priv->dev = &pdev->dev;
+

+ /* get parent IPU */
+ priv->ipu = dev_get_drvdata(priv->dev->parent);

+
+ /* get our CSI id */

+ pdata = priv->dev->platform_data;

+ priv->csi_id = pdata->csi;
+
+ v4l2_subdev_init(&priv->sd, &csi_subdev_ops);

+ v4l2_set_subdevdata(&priv->sd, priv);

+ priv->sd.internal_ops = &csi_internal_ops;
+ priv->sd.entity.ops = &csi_entity_ops;

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;

+ priv->sd.grp_id = priv->csi_id ?
+ IMX_MEDIA_GRP_ID_CSI1 : IMX_MEDIA_GRP_ID_CSI0;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ imx_media_grp_id_to_sd_name(priv->sd.name, sizeof(priv->sd.name),
+ priv->sd.grp_id, ipu_get_num(priv->ipu));

+
+ v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);

+ priv->sd.ctrl_handler = &priv->ctrl_hdlr;

+
+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+

+ return ret;
+}
+

+static int imx_csi_remove(struct platform_device *pdev)

+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+ struct csi_priv *priv = sd_to_dev(sd);
+
+ if (priv->fim)
+ imx_media_fim_free(priv->fim);

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(sd);
+
+ if (!IS_ERR_OR_NULL(priv->csi))
+ ipu_csi_put(priv->csi);

+
+ return 0;
+}

+
+static const struct platform_device_id imx_csi_ids[] = {
+ { .name = "imx-ipuv3-csi" },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, imx_csi_ids);

+
+static struct platform_driver imx_csi_driver = {
+ .probe = imx_csi_probe,
+ .remove = imx_csi_remove,
+ .id_table = imx_csi_ids,

+ .driver = {
+ .name = "imx-ipuv3-csi",
+ },
+};
+module_platform_driver(imx_csi_driver);
+
+MODULE_DESCRIPTION("i.MX CSI subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-csi");
--

2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

From: Philipp Zabel <p.z...@pengutronix.de>

This driver can handle SoC internal and external video bus multiplexers,
controlled either by register bit fields or by a GPIO. The subdevice
passes through frame interval and mbus configuration of the active input
to the output side.

Signed-off-by: Sascha Hauer <s.h...@pengutronix.de>

Signed-off-by: Philipp Zabel <p.z...@pengutronix.de>

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev()
should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev()
to vidsw_remove().

- there was a line left over from a previous iteration that negated
the new way of determining the pad count just before it which
has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding
to the subdev async notifier waiting list using a chaining method
from the async registered callbacks (v4l2_of_subdev_registered()
and the prep patches for that). For now, I've removed the use of
v4l2_of_subdev_registered() for the vidmux driver's registered
callback. This doesn't affect the functionality of this driver,
but allows for it to be merged now, before adding the chaining
support.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

.../bindings/media/video-multiplexer.txt | 59 +++
drivers/media/platform/Kconfig | 8 +
drivers/media/platform/Makefile | 2 +
drivers/media/platform/video-multiplexer.c | 472 +++++++++++++++++++++
4 files changed, 541 insertions(+)
create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
create mode 100644 drivers/media/platform/video-multiplexer.c

diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt
new file mode 100644
index 0000000..9d133d9
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/video-multiplexer.txt
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+ may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+ source and sink devices according to of_graph bindings. The last port is
+ the output port, all others are inputs.
+
+Example:
+
+syscon {
+ compatible = "syscon", "simple-mfd";
+
+ mux {
+ compatible = "video-multiplexer";
+ /* Single bit (1 << 19) in syscon register 0x04: */

+ reg = <0x04>;
+ bit-mask = <1>;
+ bit-shift = <19>;

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ port@0 {
+ reg = <0>;
+

+ mux_in0: endpoint {
+ remote-endpoint = <&video_source0_out>;

+ };
+ };
+
+ port@1 {

+ reg = <1>;
+

+ mux_in1: endpoint {
+ remote-endpoint = <&video_source1_out>;

+ };
+ };
+
+ port@2 {
+ reg = <2>;
+

+ mux_out: endpoint {
+ remote-endpoint = <&capture_interface_in>;
+ };
+ };
+ };
+};
diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index d944421..65614b5 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
To compile this driver as a module, choose M here: the
module will be called arv.

+config VIDEO_MULTIPLEXER
+ tristate "Video Multiplexer"
+ depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+ help
+ This driver provides support for SoC internal N:1 video bus
+ multiplexers controlled by register bitfields as well as external
+ 2:1 video multiplexers controlled by a single GPIO.
+
config VIDEO_OMAP3
tristate "OMAP 3 Camera support"
depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
diff --git a/drivers/media/platform/Makefile b/drivers/media/platform/Makefile
index 5b3cb27..7cf0ee5 100644
--- a/drivers/media/platform/Makefile
+++ b/drivers/media/platform/Makefile
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU) += sh_veu.o

obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE) += m2m-deinterlace.o

+obj-$(CONFIG_VIDEO_MULTIPLEXER) += video-multiplexer.o
+
obj-$(CONFIG_VIDEO_S3C_CAMIF) += s3c-camif/
obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) += exynos4-is/
obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG) += s5p-jpeg/
diff --git a/drivers/media/platform/video-multiplexer.c b/drivers/media/platform/video-multiplexer.c
new file mode 100644
index 0000000..48980c4
--- /dev/null
+++ b/drivers/media/platform/video-multiplexer.c
@@ -0,0 +1,472 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <ker...@pengutronix.de>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <ker...@pengutronix.de>

+ *
+ * This program is free software; you can redistribute it and/or

+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+ struct v4l2_subdev subdev;
+ unsigned int num_pads;
+ struct media_pad *pads;
+ struct v4l2_mbus_framefmt *format_mbus;
+ struct v4l2_fract timeperframe;
+ struct v4l2_of_endpoint *endpoint;
+ struct regmap_field *field;
+ struct gpio_desc *gpio;
+ int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+ return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+ vidsw->active = active;
+ if (active < 0)
+ return;
+
+ dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+ if (vidsw->field)
+ regmap_field_write(vidsw->field, active);
+ else if (vidsw->gpio)
+ gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+ /* We have no limitations on enabling or disabling our output link */
+ if (local->index == vidsw->num_pads - 1)
+ return 0;
+
+ dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+ if (local->index == vidsw->active) {
+ dev_dbg(sd->dev, "going inactive\n");
+ vidsw->active = -1;

+ }
+ return 0;
+ }
+

+ if (vidsw->active >= 0) {
+ struct media_pad *pad;
+
+ if (vidsw->active == local->index)
+ return 0;
+
+ pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+ if (pad) {
+ struct media_link *link;
+ int ret;
+
+ link = media_entity_find_link(pad,
+ &vidsw->pads[vidsw->active]);
+ if (link) {
+ ret = __media_entity_setup_link(link, 0);

+ if (ret)
+ return ret;
+ }
+ }
+ }

+
+ vidsw_set_active(vidsw, local->index);

+
+ return 0;
+}
+

+static struct media_entity_operations vidsw_ops = {
+ .link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+ struct device_node *rpp;
+
+ if (!of_device_is_available(ep))
+ return true;
+
+ rpp = of_graph_get_remote_port_parent(ep);
+ if (!rpp)
+ return true;
+
+ return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+ struct device_node *ep;
+ u32 portno;
+ int numports;
+ int ret;
+ int i;
+ bool active_link = false;
+
+ numports = vidsw->num_pads;
+
+ for (i = 0; i < numports - 1; i++)
+ vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+ vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+ vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
+ ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+ vidsw->pads);
+ if (ret < 0)
+ return ret;
+
+ vidsw->subdev.entity.ops = &vidsw_ops;
+
+ for_each_endpoint_of_node(node, ep) {
+ struct v4l2_of_endpoint endpoint;
+
+ v4l2_of_parse_endpoint(ep, &endpoint);
+
+ portno = endpoint.base.port;
+ if (portno >= numports - 1)
+ continue;
+
+ if (vidsw_endpoint_disabled(ep)) {
+ dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+ continue;
+ }
+
+ vidsw->endpoint[portno] = endpoint;
+
+ if (portno == vidsw->active)
+ active_link = true;
+ }
+
+ for (portno = 0; portno < numports - 1; portno++) {
+ if (!vidsw->endpoint[portno].base.local_node)
+ continue;
+
+ /* If the active input is not connected, use another */
+ if (!active_link) {
+ vidsw_set_active(vidsw, portno);
+ active_link = true;
+ }
+ }
+
+ return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+ struct media_pad *pad;
+ int ret;
+
+ if (vidsw->active == -1) {
+ dev_err(sd->dev, "no configuration for inactive mux\n");

+ return -EINVAL;
+ }
+

+ /*
+ * Retrieve media bus configuration from the entity connected to the
+ * active input
+ */
+ pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+ if (pad) {
+ sd = media_entity_to_v4l2_subdev(pad->entity);
+ ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+ if (ret == -ENOIOCTLCMD)
+ pad = NULL;
+ else if (ret < 0) {
+ dev_err(sd->dev, "failed to get source configuration\n");

+ return ret;
+ }
+ }

+ if (!pad) {
+ /* Mirror the input side on the output side */
+ cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+ if (cfg->type == V4L2_MBUS_PARALLEL ||
+ cfg->type == V4L2_MBUS_BT656)
+ cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;

+ }
+
+ return 0;
+}
+

+static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+ struct v4l2_subdev *upstream_sd;
+ struct media_pad *pad;
+
+ if (vidsw->active == -1) {
+ dev_err(sd->dev, "Can not start streaming on inactive mux\n");

+ return -EINVAL;
+ }
+

+ pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
+ if (!pad) {
+ dev_err(sd->dev, "Failed to find remote source pad\n");
+ return -ENOLINK;
+ }
+
+ if (!is_media_entity_v4l2_subdev(pad->entity)) {
+ dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
+ return -ENODEV;
+ }
+
+ upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
+
+ return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
+}
+
+static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
+ struct v4l2_subdev_frame_interval *fi)
+{
+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+ fi->interval = vidsw->timeperframe;

+
+ return 0;
+}
+

+static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
+ struct v4l2_subdev_frame_interval *fi)
+{
+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+ vidsw->timeperframe = fi->interval;

+
+ return 0;
+}
+

+static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
+ .g_mbus_config = vidsw_g_mbus_config,
+ .s_stream = vidsw_s_stream,
+ .g_frame_interval = vidsw_g_frame_interval,
+ .s_frame_interval = vidsw_s_frame_interval,
+};
+
+static struct v4l2_mbus_framefmt *
+__vidsw_get_pad_format(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ unsigned int pad, u32 which)
+{
+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+ switch (which) {
+ case V4L2_SUBDEV_FORMAT_TRY:
+ return v4l2_subdev_get_try_format(sd, cfg, pad);
+ case V4L2_SUBDEV_FORMAT_ACTIVE:
+ return &vidsw->format_mbus[pad];
+ default:

+ return NULL;
+ }
+}
+

+static int vidsw_get_format(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
+ sdformat->which);

+ return 0;
+}
+

+static int vidsw_set_format(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+ struct v4l2_mbus_framefmt *mbusformat;
+
+ if (sdformat->pad >= vidsw->num_pads)
+ return -EINVAL;
+
+ mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
+ sdformat->which);
+ if (!mbusformat)
+ return -EINVAL;
+
+ /* Output pad mirrors active input pad, no limitations on input pads */
+ if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
+ sdformat->format = vidsw->format_mbus[vidsw->active];
+
+ *mbusformat = sdformat->format;

+
+ return 0;
+}
+

+static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
+ .get_fmt = vidsw_get_format,
+ .set_fmt = vidsw_set_format,
+};
+
+static struct v4l2_subdev_ops vidsw_subdev_ops = {
+ .pad = &vidsw_pad_ops,
+ .video = &vidsw_subdev_video_ops,
+};
+
+static int of_get_reg_field(struct device_node *node, struct reg_field *field)
+{
+ u32 bit_mask;
+ int ret;
+
+ ret = of_property_read_u32(node, "reg", &field->reg);
+ if (ret < 0)
+ return ret;
+
+ ret = of_property_read_u32(node, "bit-mask", &bit_mask);
+ if (ret < 0)
+ return ret;
+
+ ret = of_property_read_u32(node, "bit-shift", &field->lsb);
+ if (ret < 0)
+ return ret;
+
+ field->msb = field->lsb + fls(bit_mask) - 1;

+
+ return 0;
+}
+

+static int vidsw_probe(struct platform_device *pdev)
+{
+ struct device_node *np = pdev->dev.of_node;
+ struct of_endpoint endpoint;
+ struct device_node *ep;
+ struct reg_field field;
+ struct vidsw *vidsw;
+ struct regmap *map;
+ unsigned int num_pads;
+ int ret;
+
+ vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
+ if (!vidsw)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, vidsw);
+
+ v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
+ snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
+ np->name);
+ vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+ vidsw->subdev.dev = &pdev->dev;
+
+ /*
+ * The largest numbered port is the output port. It determines
+ * total number of pads
+ */
+ num_pads = 0;
+ for_each_endpoint_of_node(np, ep) {
+ of_graph_parse_endpoint(ep, &endpoint);
+ num_pads = max(num_pads, endpoint.port + 1);
+ }
+
+ if (num_pads < 2) {
+ dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);

+ return -EINVAL;
+ }
+

+ ret = of_get_reg_field(np, &field);
+ if (ret == 0) {
+ map = syscon_node_to_regmap(np->parent);
+ if (!map) {
+ dev_err(&pdev->dev, "Failed to get syscon register map\n");
+ return PTR_ERR(map);
+ }
+
+ vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
+ if (IS_ERR(vidsw->field)) {
+ dev_err(&pdev->dev, "Failed to allocate regmap field\n");
+ return PTR_ERR(vidsw->field);
+ }
+
+ regmap_field_read(vidsw->field, &vidsw->active);
+ } else {
+ if (num_pads > 3) {
+ dev_err(&pdev->dev, "Too many ports %d\n", num_pads);

+ return -EINVAL;
+ }
+

+ vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
+ if (IS_ERR(vidsw->gpio)) {
+ dev_warn(&pdev->dev,
+ "could not request control gpio: %d\n", ret);
+ vidsw->gpio = NULL;
+ }
+
+ vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
+ }
+
+ vidsw->num_pads = num_pads;
+ vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
+ GFP_KERNEL);
+ vidsw->format_mbus = devm_kzalloc(&pdev->dev,
+ sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
+ vidsw->endpoint = devm_kzalloc(&pdev->dev,
+ sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
+
+ ret = vidsw_async_init(vidsw, np);

+ if (ret)
+ return ret;
+

+ return 0;
+}
+

+static int vidsw_remove(struct platform_device *pdev)
+{
+ struct vidsw *vidsw = platform_get_drvdata(pdev);
+ struct v4l2_subdev *sd = &vidsw->subdev;
+
+ v4l2_async_unregister_subdev(sd);
+ media_entity_cleanup(&sd->entity);
+ v4l2_device_unregister_subdev(sd);

+
+ return 0;
+}
+

+static const struct of_device_id vidsw_dt_ids[] = {
+ { .compatible = "video-multiplexer", },

+ { /* sentinel */ }
+};
+

+static struct platform_driver vidsw_driver = {
+ .probe = vidsw_probe,
+ .remove = vidsw_remove,
+ .driver = {
+ .of_match_table = vidsw_dt_ids,
+ .name = "video-multiplexer",
+ },
+};
+
+module_platform_driver(vidsw_driver);
+
+MODULE_DESCRIPTION("video stream multiplexer");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 1 +
drivers/staging/media/imx/imx-smfc.c | 737 +++++++++++++++++++++++++++++++++++
2 files changed, 738 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-smfc.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 133672a..3559d7b 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

diff --git a/drivers/staging/media/imx/imx-smfc.c b/drivers/staging/media/imx/imx-smfc.c
new file mode 100644
index 0000000..614a4381
--- /dev/null
+++ b/drivers/staging/media/imx/imx-smfc.c
@@ -0,0 +1,737 @@
+/*
+ * V4L2 Capture SMFC Subdev for Freescale i.MX5/6 SOC
+ *

+ * This subdevice handles capture of raw/unconverted video frames
+ * from the CSI, directly to memory via the Sensor Multi-FIFO Controller.

+ *

+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>

+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>

+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>

+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>

+#include <media/imx.h>
+#include "imx-media.h"
+
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W 176
+#define MIN_H 144
+#define MAX_W 8192
+#define MAX_H 4096
+#define W_ALIGN 4 /* multiple of 16 pixels */
+#define H_ALIGN 1 /* multiple of 2 lines */
+#define S_ALIGN 1 /* multiple of 2 */
+
+#define SMFC_NUM_PADS 2
+
+struct imx_smfc_priv {

+ struct device *dev;
+ struct ipu_soc *ipu;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[SMFC_NUM_PADS];
+ int ipu_id;
+ int smfc_id;

+ int input_pad;
+ int output_pad;
+

+ struct ipuv3_channel *smfc_ch;
+ struct ipu_smfc *smfc;
+
+ struct v4l2_mbus_framefmt format_mbus[SMFC_NUM_PADS];
+ const struct imx_media_pixfmt *cc[SMFC_NUM_PADS];
+

+ struct v4l2_mbus_config sensor_mbus_cfg;
+

+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct imx_media_dma_buf *next;
+
+ int ipu_buf_num; /* ipu double buffer index: 0-1 */
+
+ /* the sink that will receive the dma buffers */
+ struct v4l2_subdev *sink_sd;
+ struct v4l2_subdev *src_sd;
+
+ /*
+ * the CSI id and mipi virtual channel number at
+ * link validate
+ */
+ int csi_id;
+ int vc_num;

+
+ /* the attached sensor at stream on */
+ struct imx_media_subdev *sensor;
+

+ spinlock_t irqlock;
+ struct timer_list eof_timeout_timer;
+ int eof_irq;
+ int nfb4eof_irq;
+

+ bool stream_on; /* streaming is on */

+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;
+};
+
+static void imx_smfc_put_ipu_resources(struct imx_smfc_priv *priv)
+{
+ if (!IS_ERR_OR_NULL(priv->smfc_ch))
+ ipu_idmac_put(priv->smfc_ch);
+ priv->smfc_ch = NULL;
+
+ if (!IS_ERR_OR_NULL(priv->smfc))
+ ipu_smfc_put(priv->smfc);
+ priv->smfc = NULL;
+}
+
+static int imx_smfc_get_ipu_resources(struct imx_smfc_priv *priv)
+{
+ int ch_num, ret;
+
+ priv->ipu = priv->md->ipu[priv->ipu_id];
+
+ ch_num = IPUV3_CHANNEL_CSI0 + priv->smfc_id;
+
+ priv->smfc = ipu_smfc_get(priv->ipu, ch_num);
+ if (IS_ERR(priv->smfc)) {
+ v4l2_err(&priv->sd, "failed to get SMFC\n");
+ ret = PTR_ERR(priv->smfc);

+ goto out;
+ }
+

+ priv->smfc_ch = ipu_idmac_get(priv->ipu, ch_num);
+ if (IS_ERR(priv->smfc_ch)) {
+ v4l2_err(&priv->sd, "could not get IDMAC channel %u\n", ch_num);
+ ret = PTR_ERR(priv->smfc_ch);
+ goto out;

+ }
+
+ return 0;

+out:
+ imx_smfc_put_ipu_resources(priv);

+ return ret;
+}
+

+static irqreturn_t imx_smfc_eof_interrupt(int irq, void *dev_id)
+{
+ struct imx_smfc_priv *priv = dev_id;

+ struct imx_media_dma_buf *done, *next;
+

+ spin_lock(&priv->irqlock);

+
+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;
+ goto unlock;
+ }
+
+ /* inform CSI of this EOF so it can monitor frame intervals */
+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+

+ if (ipu_idmac_buffer_is_ready(priv->smfc_ch, priv->ipu_buf_num))

+ ipu_idmac_clear_buffer(priv->smfc_ch, priv->ipu_buf_num);

+
+ /* get next queued buffer */
+ next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+

+ ipu_cpmem_set_buffer(priv->smfc_ch, priv->ipu_buf_num, next->phys);
+ ipu_idmac_select_buffer(priv->smfc_ch, priv->ipu_buf_num);

+
+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ priv->next = next;
+
+unlock:

+ spin_unlock(&priv->irqlock);

+ return IRQ_HANDLED;
+}
+

+static irqreturn_t imx_smfc_nfb4eof_interrupt(int irq, void *dev_id)
+{

+ struct imx_smfc_priv *priv = dev_id;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+
+ v4l2_err(&priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void imx_smfc_eof_timeout(unsigned long data)
+{
+ struct imx_smfc_priv *priv = (struct imx_smfc_priv *)data;
+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+
+ v4l2_err(&priv->sd, "EOF timeout\n");
+
+ v4l2_subdev_notify_event(&priv->sd, &ev);
+}
+
+/* init the SMFC IDMAC channel */
+static void imx_smfc_setup_channel(struct imx_smfc_priv *priv)
+{
+ struct v4l2_mbus_framefmt *infmt, *outfmt;
+ struct imx_media_dma_buf *buf0, *buf1;
+ unsigned int burst_size;
+ struct ipu_image image;
+ bool passthrough;

+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ ipu_cpmem_zero(priv->smfc_ch);
+
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+
+ image.phys0 = buf0->phys;
+ image.phys1 = buf1->phys;
+ ipu_cpmem_set_image(priv->smfc_ch, &image);
+
+ burst_size = (outfmt->width & 0xf) ? 8 : 16;
+
+ ipu_cpmem_set_burstsize(priv->smfc_ch, burst_size);
+
+ /*
+ * If the sensor uses 16-bit parallel CSI bus, we must handle
+ * the data internally in the IPU as 16-bit generic, aka
+ * passthrough mode.
+ */
+ passthrough = (priv->sensor_mbus_cfg.type != V4L2_MBUS_CSI2 &&
+ priv->sensor->sensor_ep.bus.parallel.bus_width >= 16);
+
+ if (passthrough)
+ ipu_cpmem_set_format_passthrough(priv->smfc_ch, 16);
+
+ /*
+ * Set the channel for the direct CSI-->memory via SMFC
+ * use-case to very high priority, by enabling the watermark
+ * signal in the SMFC, enabling WM in the channel, and setting
+ * the channel priority to high.
+ *
+ * Refer to the i.mx6 rev. D TRM Table 36-8: Calculated priority
+ * value.
+ *
+ * The WM's are set very low by intention here to ensure that
+ * the SMFC FIFOs do not overflow.
+ */
+ ipu_smfc_set_watermark(priv->smfc, 0x02, 0x01);
+ ipu_cpmem_set_high_priority(priv->smfc_ch);
+ ipu_idmac_enable_watermark(priv->smfc_ch, true);
+ ipu_cpmem_set_axi_id(priv->smfc_ch, 0);
+ ipu_idmac_lock_enable(priv->smfc_ch, 8);
+
+ burst_size = ipu_cpmem_get_burstsize(priv->smfc_ch);
+ burst_size = passthrough ?
+ (burst_size >> 3) - 1 : (burst_size >> 2) - 1;
+
+ ipu_smfc_set_burstsize(priv->smfc, burst_size);
+
+ if (outfmt->field == V4L2_FIELD_NONE &&
+ (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+ infmt->field == V4L2_FIELD_ALTERNATE))
+ ipu_cpmem_interlaced_scan(priv->smfc_ch,
+ image.pix.bytesperline);
+
+ ipu_idmac_set_double_buffer(priv->smfc_ch, true);
+}
+
+static void imx_smfc_unsetup(struct imx_smfc_priv *priv)
+{
+ ipu_idmac_disable_channel(priv->smfc_ch);
+ ipu_smfc_disable(priv->smfc);
+}
+
+static void imx_smfc_setup(struct imx_smfc_priv *priv)
+{
+ imx_smfc_setup_channel(priv);
+
+ ipu_cpmem_dump(priv->smfc_ch);
+ ipu_dump(priv->ipu);
+
+ ipu_smfc_enable(priv->smfc);
+
+ /* set buffers ready */
+ ipu_idmac_select_buffer(priv->smfc_ch, 0);
+ ipu_idmac_select_buffer(priv->smfc_ch, 1);
+
+ /* enable the channels */
+ ipu_idmac_enable_channel(priv->smfc_ch);
+}
+
+static int imx_smfc_start(struct imx_smfc_priv *priv)
+{
+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return -EINVAL;
+ }
+

+ ret = imx_smfc_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ ipu_smfc_map_channel(priv->smfc, priv->csi_id, priv->vc_num);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ goto out_put_ipu;
+
+ priv->ipu_buf_num = 0;
+
+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);

+ priv->last_eof = false;
+

+ imx_smfc_setup(priv);
+
+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+ priv->smfc_ch,
+ IPU_IRQ_NFB4EOF);
+ ret = devm_request_irq(priv->dev, priv->nfb4eof_irq,
+ imx_smfc_nfb4eof_interrupt, 0,
+ "imx-smfc-nfb4eof", priv);

+ if (ret) {
+ v4l2_err(&priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_unsetup;
+ }
+
+ priv->eof_irq = ipu_idmac_channel_irq(priv->ipu, priv->smfc_ch,
+ IPU_IRQ_EOF);
+
+ ret = devm_request_irq(priv->dev, priv->eof_irq,
+ imx_smfc_eof_interrupt, 0,
+ "imx-smfc-eof", priv);

+ if (ret) {
+ v4l2_err(&priv->sd,

+ "Error registering eof irq: %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ /* start the EOF timeout timer */

+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+out_free_nfb4eof_irq:
+ devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+ imx_smfc_unsetup(priv);
+out_put_ipu:
+ imx_smfc_put_ipu_resources(priv);

+ return ret;
+}
+

+ if (ret == 0)

+ v4l2_warn(&priv->sd, "wait last EOF timeout\n");
+
+ devm_free_irq(priv->dev, priv->eof_irq, priv);
+ devm_free_irq(priv->dev, priv->nfb4eof_irq, priv);
+
+ imx_smfc_unsetup(priv);
+
+ /* cancel the EOF timeout timer */
+ del_timer_sync(&priv->eof_timeout_timer);
+
+ priv->out_ring = NULL;
+
+ /* inform sink that the buffer ring can now be freed */
+ v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+
+ imx_smfc_put_ipu_resources(priv);
+}
+
+static int imx_smfc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ int ret = 0;
+
+ if (!priv->src_sd || !priv->sink_sd)
+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+ if (enable && !priv->stream_on)

+ ret = imx_smfc_start(priv);

+ else if (!enable && priv->stream_on)

+ imx_smfc_stop(priv);

+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int imx_smfc_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (code->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+
+ return imx_media_enum_format(&code->code, code->index,
+ true, code->pad == priv->output_pad);
+}
+
+static int imx_smfc_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+

+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int imx_smfc_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc, *incc;
+ bool allow_planar;
+ u32 code;
+
+ if (sdformat->pad >= SMFC_NUM_PADS)
+ return -EINVAL;
+

+ if (priv->stream_on)
+ return -EBUSY;
+
+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+
+ cc = imx_media_find_format(0, sdformat->format.code,
+ true, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, false);
+ cc = imx_media_find_format(0, code, true, false);
+ sdformat->format.code = cc->codes[0];
+ }
+
+ v4l_bound_align_image(&sdformat->format.width, MIN_W, MAX_W,
+ W_ALIGN, &sdformat->format.height,
+ MIN_H, MAX_H, H_ALIGN, S_ALIGN);
+
+ if (sdformat->pad == priv->output_pad) {

+ incc = priv->cc[priv->input_pad];

+ sdformat->format.width = infmt->width;
+ sdformat->format.height = infmt->height;
+ if (sdformat->format.field != V4L2_FIELD_NONE)

+ sdformat->format.field = infmt->field;

+ if (cc->cs != incc->cs) {

+ sdformat->format.code = infmt->code;

+ cc = imx_media_find_format(0, sdformat->format.code,
+ true, false);
+ }

+ }
+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;

+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int imx_smfc_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {

+ priv->sink_sd = NULL;
+ }
+

+ return 0;
+ }
+

+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;

+ return 0;
+ }
+

+ /* must attach to CSI source */
+ if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_CSI))
+ return -EINVAL;
+

+ return 0;
+}
+

+static int imx_smfc_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);
+ int ret;
+
+ ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);

+ if (ret)
+ return ret;
+

+ switch (priv->src_sd->grp_id) {
+ case IMX_MEDIA_GRP_ID_CSI0:
+ priv->csi_id = 0;
+ break;
+ case IMX_MEDIA_GRP_ID_CSI1:
+ priv->csi_id = 1;
+ break;
+ default:

+ return -EINVAL;
+ }
+

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+ &priv->sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ }
+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int imx_smfc_registered(struct v4l2_subdev *sd)
+{
+ struct imx_smfc_priv *priv = v4l2_get_subdevdata(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i;

+
+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+ if (IS_ERR(imxsd))

+ return PTR_ERR(imxsd);

+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)

+ return -EINVAL;
+
+ for (i = 0; i < SMFC_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];
+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;
+
+ /* set a default mbus format */

+ imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);
+ }

+
+ return media_entity_pads_init(&sd->entity, SMFC_NUM_PADS, priv->pad);
+}
+
+static struct media_entity_operations imx_smfc_entity_ops = {
+ .link_setup = imx_smfc_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_video_ops imx_smfc_video_ops = {
+ .s_stream = imx_smfc_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imx_smfc_pad_ops = {
+ .enum_mbus_code = imx_smfc_enum_mbus_code,
+ .get_fmt = imx_smfc_get_fmt,
+ .set_fmt = imx_smfc_set_fmt,
+ .link_validate = imx_smfc_link_validate,
+};
+
+static struct v4l2_subdev_ops imx_smfc_subdev_ops = {
+ .video = &imx_smfc_video_ops,
+ .pad = &imx_smfc_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imx_smfc_internal_ops = {
+ .registered = imx_smfc_registered,
+};
+
+static int imx_smfc_probe(struct platform_device *pdev)
+{
+ struct imx_media_internal_sd_platformdata *pdata;
+ struct imx_smfc_priv *priv;

+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)

+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, &priv->sd);
+ priv->dev = &pdev->dev;
+

+ pdata = priv->dev->platform_data;

+ priv->ipu_id = pdata->ipu_id;
+

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;
+ priv->eof_timeout_timer.function = imx_smfc_eof_timeout;
+ spin_lock_init(&priv->irqlock);
+
+ v4l2_subdev_init(&priv->sd, &imx_smfc_subdev_ops);
+ v4l2_set_subdevdata(&priv->sd, priv);
+ priv->sd.internal_ops = &imx_smfc_internal_ops;
+ priv->sd.entity.ops = &imx_smfc_entity_ops;

+ /* FIXME: this the right function? */

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;
+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ /* get our group id and SMFC id */

+ priv->sd.grp_id = pdata->grp_id;

+ priv->smfc_id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_SMFC_BIT) - 1;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+

+ return v4l2_async_register_subdev(&priv->sd);
+}
+
+static int imx_smfc_remove(struct platform_device *pdev)
+{

+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imx_smfc_priv *priv = container_of(sd, struct imx_smfc_priv, sd);
+

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(sd);
+
+ return 0;
+}
+

+static const struct platform_device_id imx_smfc_ids[] = {

+ { .name = "imx-ipuv3-smfc" },
+ { },
+};

+MODULE_DEVICE_TABLE(platform, imx_smfc_ids);
+
+static struct platform_driver imx_smfc_driver = {
+ .probe = imx_smfc_probe,
+ .remove = imx_smfc_remove,
+ .id_table = imx_smfc_ids,

+ .driver = {
+ .name = "imx-ipuv3-smfc",
+ },
+};
+module_platform_driver(imx_smfc_driver);
+
+MODULE_DESCRIPTION("i.MX SMFC subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-smfc");
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6q.dtsi | 4 ++++
arch/arm/boot/dts/imx6qdl.dtsi | 8 ++++++++
2 files changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index 9b2ca32..8867e78 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -458,3 +458,7 @@
&vpu {
compatible = "fsl,imx6q-vpu", "cnm,coda960";
};
+
+&media0 {
+ ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};
diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 010388c..cbb42ec 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1296,5 +1296,13 @@
};
};
};
+
+ media0: media@0 {
+ compatible = "fsl,imx-media";
+ ports = <&ipu1_csi0>, <&ipu1_csi1>;

+ #address-cells = <1>;
+ #size-cells = <0>;

+ status = "okay";
+ };
};
};
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

From: Philipp Zabel <p.z...@pengutronix.de>

Signed-off-by: Philipp Zabel <p.z...@pengutronix.de>
---
Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
include/uapi/linux/media.h | 6 ++++++
2 files changed, 28 insertions(+)

diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
index 3e03dc2..023be29 100644
--- a/Documentation/media/uapi/mediactl/media-types.rst
+++ b/Documentation/media/uapi/mediactl/media-types.rst
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
received on its sink pad and outputs the statistics data on
its source pad.

+ - .. row 29
+
+ .. _MEDIA-ENT-F-MUX:
+
+ - ``MEDIA_ENT_F_MUX``
+
+ - Video multiplexer. An entity capable of multiplexing must have at
+ least two sink pads and one source pad, and must pass the video
+ frame(s) received from the active sink pad to the source pad. Video
+ frame(s) from the inactive sink pads are discarded.
+
+ - .. row 30
+
+ .. _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+ - ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+ - Video interface bridge. A video interface bridge entity must have at
+ least one sink pad and one source pad. It receives video frame(s) on
+ its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+ converts them and outputs them on its source pad in another bus format
+ (eDP, MIPI CSI-2, parallel, ...).

.. tabularcolumns:: |p{5.5cm}|p{12.0cm}|

diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index 4890787..08a8bfa 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -105,6 +105,12 @@ struct media_device_info {
#define MEDIA_ENT_F_PROC_VIDEO_STATISTICS (MEDIA_ENT_F_BASE + 0x4006)

/*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX (MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE (MEDIA_ENT_F_BASE + 0x5002)
+
+/*
* Connectors
*/
/* It is a responsibility of the entity drivers to add connectors and links */
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

This driver is based on ov5640_mipi.c from Freescale imx_3.10.17_1.0.0_beta
branch, modified heavily to bring forward to latest interfaces and code
cleanup.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Kconfig | 8 +
drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++++++++++++++++++++++
3 files changed, 2358 insertions(+)
create mode 100644 drivers/staging/media/imx/ov5640-mipi.c

diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
index ce2d2c8..09f373d 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -17,5 +17,13 @@ config VIDEO_IMX_CAMERA
---help---

A video4linux camera capture driver for i.MX5/6.

+config IMX_OV5640_MIPI
+ tristate "OmniVision OV5640 MIPI CSI-2 camera support"
+ depends on GPIOLIB && VIDEO_IMX_CAMERA
+ select IMX_MIPI_CSI2

+ default y
+ ---help---

+ MIPI CSI-2 OV5640 Camera support.
+
endmenu
endif
diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index 0decef7..aa954c1 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -10,3 +10,5 @@ obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
+
+obj-$(CONFIG_IMX_OV5640_MIPI) += ov5640-mipi.o
diff --git a/drivers/staging/media/imx/ov5640-mipi.c b/drivers/staging/media/imx/ov5640-mipi.c
new file mode 100644
index 0000000..54647a7
--- /dev/null
+++ b/drivers/staging/media/imx/ov5640-mipi.c
@@ -0,0 +1,2348 @@
+/*
+ * Copyright (c) 2014 Mentor Graphics Inc.
+ * Copyright (C) 2011-2013 Freescale Semiconductor, Inc. All Rights Reserved.

+ return ret;
+ }
+

+ return 0;
+}
+

+ return ret;
+ }
+

+ return 0;
+}
+

+#define OV5640_READ_REG(s, r, v) { \
+ ret = ov5640_read_reg((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }

+#define OV5640_WRITE_REG(s, r, v) { \
+ ret = ov5640_write_reg((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+static int ov5640_write_reg16(struct ov5640_dev *sensor, u16 reg, u16 val)

+{
+ int ret;
+

+ OV5640_WRITE_REG(sensor, reg, val >> 8);
+ OV5640_WRITE_REG(sensor, reg+1, val & 0xff);
+ return 0;
+}
+#define OV5640_WRITE_REG16(s, r, v) { \
+ ret = ov5640_write_reg16((s), (r), (v)); \

+ if (ret) \
+ return ret; \
+ }
+

+/* download ov5640 settings to sensor through i2c */
+static int ov5640_load_regs(struct ov5640_dev *sensor,
+ struct reg_value *regs,
+ int size)
+{
+ register u32 delay_ms = 0;
+ register u16 reg_addr = 0;
+ register u8 mask = 0;
+ register u8 val = 0;
+ int i, ret;
+
+ for (i = 0; i < size; ++i, ++regs) {
+ delay_ms = regs->delay_ms;
+ reg_addr = regs->reg_addr;
+ val = regs->val;
+ mask = regs->mask;
+
+ if (mask) {
+ OV5640_MOD_REG(sensor, reg_addr, mask, val);
+ } else {
+ OV5640_WRITE_REG(sensor, reg_addr, val);
+ }
+ if (delay_ms)
+ usleep_range(1000*delay_ms, 1000*delay_ms+100);

+ }
+
+ return 0;
+}
+

+static int ov5640_set_stream(struct ov5640_dev *sensor, bool on)

+{
+ int ret;
+

+ OV5640_WRITE_REG(sensor, 0x4202, on ? 0x00 : 0x0f);

+ return 0;
+}
+

+{
+ int ret;
+

+ OV5640_WRITE_REG16(sensor, 0x380e, VTS);

+ return 0;
+}
+

+ if (ret < 0)
+ return ret;

+ sensor->prev_sysclk = ret;
+ /* read preview HTS */
+ ret = ov5640_get_HTS(sensor);

+ if (ret < 0)
+ return ret;

+ sensor->prev_hts = ret;
+
+ /* read preview VTS */
+ ret = ov5640_get_VTS(sensor);

+ if (ret < 0)
+ return ret;

+
+ return 0;
+}
+

+ int ret = 0;
+

+ if (ret < 0)
+ return ret;

+ prev_shutter = ret;
+ ret = ov5640_binning_on(sensor);

+ if (ret < 0)
+ return ret;

+ if (ret && mode != ov5640_mode_720P_1280_720 &&
+ mode != ov5640_mode_1080P_1920_1080)
+ prev_shutter *= 2;
+
+ /* read preview gain */
+ ret = ov5640_get_gain(sensor);

+ if (ret < 0)
+ return ret;

+ cap_vts = ret;
+ ret = ov5640_get_HTS(sensor);

+ if (ret < 0)
+ return ret;

+ cap_hts = ret;
+ ret = ov5640_get_sysclk(sensor);

+ if (ret < 0)
+ return ret;

+ cap_sysclk = ret;
+
+ /* calculate capture banding filter */
+ ret = ov5640_get_light_freq(sensor);

+ if (ret < 0)
+ return ret;

+ return ret;
+ }
+

+ int ret = 0;
+

+ if ((mode >= ov5640_num_modes || mode < ov5640_mode_MIN) &&
+ mode != ov5640_mode_INIT) {
+ v4l2_err(&sensor->sd, "Wrong ov5640 mode detected!\n");

+ return -EINVAL;
+ }
+

+ if (ret < 0)
+ return ret;

+ ret = ov5640_get_light_freq(sensor);

+ if (ret < 0)
+ return ret;

+ ret = ov5640_set_bandingfilter(sensor);

+ if (ret < 0)
+ return ret;

+ }
+
+ return 0;
+}
+

+/* restore the last set video mode after chip power-on */
+static int ov5640_restore_mode(struct ov5640_dev *sensor)
+{

+ int ret = 0;
+

+{
+ int ret;
+

+ if (sensor->io_regulator) {
+ ret = regulator_enable(sensor->io_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "io reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->core_regulator) {
+ ret = regulator_enable(sensor->core_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "core reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->gpo_regulator) {
+ ret = regulator_enable(sensor->gpo_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "gpo reg enable failed\n");

+ return ret;
+ }
+ }

+ if (sensor->analog_regulator) {
+ ret = regulator_enable(sensor->analog_regulator);
+ if (ret) {
+ v4l2_err(&sensor->sd, "analog reg enable failed\n");

+ return ret;
+ }
+ }
+

+ return 0;
+}
+

+ if (ret)
+ return ret;
+

+ ov5640_reset(sensor);
+ ov5640_power(sensor, true);
+
+ ret = ov5640_init_slave_id(sensor);

+ if (ret)
+ return ret;
+

+ ret = ov5640_restore_mode(sensor);

+ if (ret)
+ return ret;
+

+
+ return 0;
+}
+

+ int ret = 0;
+

+ return -EINVAL;
+ }
+

+ ret = ov5640_change_mode(sensor, frame_rate,
+ sensor->current_mode,
+ sensor->current_mode);
+ if (ret < 0)
+ return ret;
+
+ sensor->streamcap.timeperframe = *timeperframe;

+
+ return 0;
+}
+

+ return 0;
+}
+

+ if (ret)
+ return ret;

+ cfg->try_fmt = format->format;

+ return 0;
+ }
+

+ ret = ov5640_try_fmt_internal(sd, &format->format, &new_mode);

+ if (ret)
+ return ret;
+

+ ret = ov5640_change_mode(sensor, sensor->current_fr,
+ new_mode, sensor->current_mode);
+ if (ret >= 0)
+ sensor->fmt = format->format;
+

+ return ret;
+}
+
+

+/*
+ * Sensor Controls.
+ */
+
+static int ov5640_set_hue(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 1 << 0);
+ OV5640_WRITE_REG16(sensor, 0x5581, value);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 0, 0);

+
+ return 0;
+}
+

+static int ov5640_set_contrast(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (value) {
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 1 << 2);
+ OV5640_WRITE_REG(sensor, 0x5585, value & 0xff);
+ } else
+ OV5640_MOD_REG(sensor, 0x5580, 1 << 2, 0);

+
+ return 0;
+}
+

+static int ov5640_set_saturation(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+
+ return 0;
+}
+

+static int ov5640_set_awb(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ sensor->awb_on = value ? true : false;
+ OV5640_MOD_REG(sensor, 0x3406, 1 << 0, sensor->awb_on ? 0 : 1);

+ return 0;
+}
+

+static int ov5640_set_red_balance(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3401, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3400, (value & 0xf00) >> 8);

+ return 0;
+}
+

+#if 0
+static int ov5640_set_green_balance(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->awb_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG(sensor, 0x3405, value & 0xff);
+ OV5640_WRITE_REG(sensor, 0x3404, (value & 0xf00) >> 8);

+ return 0;
+}
+

+ }
+
+ return 0;
+}
+

+{
+ int ret;
+

+ /* this enables/disables both AEC and AGC */
+ sensor->agc_on = value ? true : false;
+ OV5640_MOD_REG(sensor, 0x3503, 0x3, sensor->agc_on ? 0 : 0x3);

+
+ return 0;
+}
+

+static int ov5640_set_gain(struct ov5640_dev *sensor, int value)

+{
+ int ret;
+

+ if (sensor->agc_on)
+ return -EINVAL;
+
+ OV5640_WRITE_REG16(sensor, 0x350a, value & 0x3ff);

+ return 0;
+}
+

+{
+ int ret;
+

+ return ret;
+}
+

+ return 0;
+}
+

+static int ov5640_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct ov5640_dev *sensor = ctrl_to_ov5640_dev(ctrl);
+ struct ov5640_control *c;

+ int ret = 0;

+ return ret;
+}
+

+
+ return 0;
+}
+

+ return 0;
+}
+

+static int ov5640_g_input_status(struct v4l2_subdev *sd, u32 *status)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ *status = !sensor->on ? V4L2_IN_ST_NO_POWER : 0;

+
+ return 0;
+}
+

+static int ov5640_s_routing(struct v4l2_subdev *sd, u32 input,
+ u32 output, u32 config)
+{
+ return (input != 0) ? -EINVAL : 0;
+}
+
+static int ov5640_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct ov5640_dev *sensor = to_ov5640_dev(sd);
+
+ if (code->pad != 0)
+ return -EINVAL;
+ if (code->index != 0)
+ return -EINVAL;
+
+ code->code = sensor->fmt.code;

+
+ return 0;
+}
+

+static int ov5640_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct ov5640_dev *sensor = to_ov5640_dev(sd);

+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ return -EINVAL;
+ }
+

+ if (ret)
+ return ret;
+

+
+ return 0;
+

+free_ctrls:
+ v4l2_ctrl_handler_free(&sensor->ctrl_hdl);
+power_off:
+ ov5640_s_power(&sensor->sd, 0);
+entity_cleanup:
+ media_entity_cleanup(&sensor->sd.entity);
+ ov5640_regulators_off(sensor);

+ return ret;
+}
+

+
+ return 0;
+}
+

+static const struct i2c_device_id ov5640_id[] = {
+ {"ov5640", 0},
+ {},
+};
+MODULE_DEVICE_TABLE(i2c, ov5640_id);
+
+static const struct of_device_id ov5640_dt_ids[] = {
+ { .compatible = "ovti,ov5640" },

+ { /* sentinel */ }
+};

+MODULE_DEVICE_TABLE(of, ov5640_dt_ids);
+
+static struct i2c_driver ov5640_i2c_driver = {
+ .driver = {
+ .name = "ov5640",
+ .of_match_table = ov5640_dt_ids,
+ },
+ .id_table = ov5640_id,
+ .probe = ov5640_probe,
+ .remove = ov5640_remove,
+};
+
+module_i2c_driver(ov5640_i2c_driver);
+
+MODULE_AUTHOR("Freescale Semiconductor, Inc.");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");

+MODULE_DESCRIPTION("OV5640 MIPI Camera Subdev Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

Until the OV5652 sensor module compatible with the SabreSD becomes
available for testing, the ov5642 node is currently disabled.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

};
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

&sata {
status = "okay";
};
+
+&ipu1_csi1_from_mipi_vc1 {
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};

diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
index 55ef535..7c5dc99 100644
--- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi
@@ -10,6 +10,7 @@
* http://www.gnu.org/copyleft/gpl.html
*/

+#include <dt-bindings/clock/imx6qdl-clock.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/input/input.h>

@@ -146,6 +147,33 @@

};
};

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;
+ data-shift = <12>; /* Lines 19:12 used */
+ hsync-active = <1>;

+ vsync-active = <1>;
+};
+

+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+

+&mipi_csi {

+ status = "okay";
+};

+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+ remote-endpoint = <&ov5640_to_mipi_csi>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+};
+

&audmux {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_audmux>;

@@ -214,7 +242,33 @@
0x8014 /* 4:FN_DMICCDAT */
0x0000 /* 5:Default */
>;
- };
+ };
+
+ ov5642: camera@3c {

+ compatible = "ovti,ov5642";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5642>;

+ clocks = <&clks IMX6QDL_CLK_CKO>;
+ clock-names = "xclk";
+ reg = <0x3c>;
+ xclk = <24000000>;

+ DOVDD-supply = <&vgen4_reg>; /* 1.8v */

+ AVDD-supply = <&vgen3_reg>; /* 2.8v, rev C board is VGEN3
+ rev B board is VGEN5 */
+ DVDD-supply = <&vgen2_reg>; /* 1.5v*/
+ pwdn-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;

+ reset-gpios = <&gpio1 17 GPIO_ACTIVE_LOW>;

+ status = "disabled";

+
+ port {
+ ov5642_to_ipu1_csi0_mux: endpoint {
+ remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;
+ hsync-active = <1>;

+ vsync-active = <1>;
+ };
+ };
+ };
};

&i2c2 {
@@ -223,6 +277,34 @@

pinctrl-0 = <&pinctrl_i2c2>;
status = "okay";

+ ov5640: camera@3c {

+ compatible = "ovti,ov5640";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_ov5640>;

+ reg = <0x3c>;
+ clocks = <&clks IMX6QDL_CLK_CKO>;

+ clock-names = "xclk";

+ xclk = <24000000>;
+ DOVDD-supply = <&vgen4_reg>; /* 1.8v */

+ AVDD-supply = <&vgen3_reg>; /* 2.8v, rev C board is VGEN3
+ rev B board is VGEN5 */
+ DVDD-supply = <&vgen2_reg>; /* 1.5v*/

+ pwdn-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>;

+ reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>;

+
+ port {

+ #address-cells = <1>;
+ #size-cells = <0>;
+

+ ov5640_to_mipi_csi: endpoint@1 {
+ reg = <1>;

+ remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+ data-lanes = <0 1>;
+ clock-lanes = <2>;
+ };
+ };
+ };
+

pmic: pfuze100@08 {
compatible = "fsl,pfuze100";
reg = <0x08>;
@@ -426,6 +508,36 @@

>;
};

+ pinctrl_ipu1_csi0: ipu1csi0grp {
+ fsl,pins = <
+ MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x1b0b0
+ MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x1b0b0
+ MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC 0x1b0b0
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC 0x1b0b0

+ >;
+ };
+

+ pinctrl_ov5640: ov5640grp {
+ fsl,pins = <

+ MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x1b0b0
+ MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x1b0b0

+ >;
+ };
+
+ pinctrl_ov5642: ov5642grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0

+ MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x1b0b0
+ >;
+ };
+
pinctrl_pcie: pciegrp {
fsl,pins = <
MX6QDL_PAD_GPIO_17__GPIO7_IO12 0x1b0b0
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 65 +++++++++++++++++++++-----------
1 file changed, 44 insertions(+), 21 deletions(-)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 52390ba..cace88c 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -108,6 +108,44 @@
default-brightness-level = <7>;
status = "okay";
};
+
+ i2cmux {
+ compatible = "i2c-mux-gpio";

+ #address-cells = <1>;
+ #size-cells = <0>;

+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_i2c3mux>;
+ mux-gpios = <&gpio5 4 0>;
+ i2c-parent = <&i2c3>;
+ idle-state = <0>;
+
+ i2c@1 {

+ #address-cells = <1>;
+ #size-cells = <0>;

+ reg = <1>;
+

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:05 PM1/6/17

Add pinctrl groups for both GPT input capture channels.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
1 file changed, 12 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
index 967c3b8..495709f 100644
--- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi
@@ -457,6 +457,18 @@
>;
};

+ pinctrl_gpt_input_capture0: gptinputcapture0grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1 0x1b0b0
+ >;
+ };
+
+ pinctrl_gpt_input_capture1: gptinputcapture1grp {
+ fsl,pins = <
+ MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2 0x1b0b0
+ >;
+ };
+
pinctrl_spdif: spdifgrp {
fsl,pins = <
MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:06 PM1/6/17

Enable imx v4l2 staging drivers. For video capture on i.MX, the
video multiplexer subdev is required. On the SabreAuto, the ADV7180
video decoder is required along with i2c-mux-gpio. The Sabrelite
requires PWM clocks for the OV5640.

Increase max zoneorder to allow larger video buffer allocations.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/configs/imx_v6_v7_defconfig | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig
index cbe7faf..b67b99f 100644
--- a/arch/arm/configs/imx_v6_v7_defconfig
+++ b/arch/arm/configs/imx_v6_v7_defconfig
@@ -51,6 +51,7 @@ CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_AEABI=y
CONFIG_HIGHMEM=y
CONFIG_CMA=y
+CONFIG_FORCE_MAX_ZONEORDER=14
CONFIG_CMDLINE="noinitrd console=ttymxc0,115200"
CONFIG_KEXEC=y
CONFIG_CPU_FREQ=y
@@ -174,13 +175,13 @@ CONFIG_INPUT_MISC=y
CONFIG_INPUT_MMA8450=y
CONFIG_SERIO_SERPORT=m
# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
CONFIG_SERIAL_IMX=y
CONFIG_SERIAL_IMX_CONSOLE=y
CONFIG_SERIAL_FSL_LPUART=y
CONFIG_SERIAL_FSL_LPUART_CONSOLE=y
# CONFIG_I2C_COMPAT is not set
CONFIG_I2C_CHARDEV=y
+CONFIG_I2C_MUX=y
CONFIG_I2C_MUX_GPIO=y
# CONFIG_I2C_HELPER_AUTO is not set
CONFIG_I2C_ALGOPCF=m
@@ -194,11 +195,11 @@ CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_MC9S08DZ60=y
CONFIG_GPIO_PCA953X=y
CONFIG_GPIO_STMPE=y
-CONFIG_POWER_SUPPLY=y
CONFIG_POWER_RESET=y
CONFIG_POWER_RESET_IMX=y
CONFIG_POWER_RESET_SYSCON=y
CONFIG_POWER_RESET_SYSCON_POWEROFF=y
+CONFIG_POWER_SUPPLY=y
CONFIG_SENSORS_GPIO_FAN=y
CONFIG_SENSORS_IIO_HWMON=y
CONFIG_THERMAL=y
@@ -221,14 +222,19 @@ CONFIG_REGULATOR_PFUZE100=y
CONFIG_MEDIA_SUPPORT=y
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_RC_SUPPORT=y
+CONFIG_MEDIA_CONTROLLER=y
+CONFIG_VIDEO_V4L2_SUBDEV_API=y
CONFIG_RC_DEVICES=y
CONFIG_IR_GPIO_CIR=y
CONFIG_MEDIA_USB_SUPPORT=y
CONFIG_USB_VIDEO_CLASS=m
CONFIG_V4L_PLATFORM_DRIVERS=y
+CONFIG_VIDEO_MULTIPLEXER=y
CONFIG_SOC_CAMERA=y
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_CODA=y
+# CONFIG_MEDIA_SUBDRV_AUTOSELECT is not set
+CONFIG_VIDEO_ADV7180=m
CONFIG_SOC_CAMERA_OV2640=y
CONFIG_IMX_IPUV3_CORE=y
CONFIG_DRM=y
@@ -338,6 +344,8 @@ CONFIG_FSL_EDMA=y
CONFIG_IMX_SDMA=y
CONFIG_MXS_DMA=y
CONFIG_STAGING=y
+CONFIG_STAGING_MEDIA=y
+CONFIG_COMMON_CLK_PWM=y
CONFIG_IIO=y
CONFIG_VF610_ADC=y
CONFIG_MPL3115=y
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:06 PM1/6/17

In version 3:

Changes suggested by Rob Herring <ro...@kernel.org>:

- prepended FIM node properties with vendor prefix "fsl,".

- make mipi csi-2 receiver compatible string SoC specific:
"fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".

- redundant "_clk" removed from mipi csi-2 receiver clock-names property.

- removed board-specific info from the media driver binding doc. These
were all related to sensor bindings, which already are (adv7180)
or will be (ov564x) covered in separate binding docs. All reference
board info not related to DT bindings has been moved to
Documentation/media/v4l-drivers/imx.rst.

- removed "_mipi" from the OV5640 compatible string.

Changes suggested by Vladimir Zapolskiy <vladimir_...@mentor.com>:

Mostly cosmetic/non-functional changes which I won't list here, except
for the following:

- spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.

- fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.

- check/handle return code from required reg property of CSI port nodes.

- check/handle return code from clk_prepare_enable().

Changes suggested by Fabio Estevam <fest...@gmail.com>:

- switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.

- finally got around to passing valid IOMUX pin config values to the
pin groups.

Other changes:

- removed the FIM properties that overrided the v4l2 FIM control defaults
values. This was left-over from a requirement of a customer and is not
necessary here.

- The FIM must be explicitly enabled in the fim child node under the CSI
port nodes, using the status property. If not enabled, FIM v4l2 controls
will not appear in the video capture driver.

- brought in additional media types patch from Philipp Zabel. Use new
MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.

- brought in latest platform generic video multiplexer subdevice driver
from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).

- removed imx-media-of.h, moved those prototypes into imx-media.h.

Philipp Zabel (3):
ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
connections
add mux and video interface bridge entity functions
platform: add video-multiplexer subdevice driver

Steve Longerbeam (21):
[media] dt-bindings: Add bindings for i.MX media driver
ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
ARM: dts: imx6qdl: add media device

ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround

ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabreauto: create i2cmux for i2c3
ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
ARM: dts: imx6-sabreauto: add the ADV7180 video decoder

UAPI: Add media UAPI Kbuild file

media: Add userspace header file for i.MX

media: Add i.MX media core driver
media: imx: Add CSI subdev driver
media: imx: Add SMFC subdev driver
media: imx: Add IC subdev drivers
media: imx: Add Camera Interface subdev driver
media: imx: Add MIPI CSI-2 Receiver subdev driver
media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
media: imx: Add Parallel OV5642 sensor subdev driver
ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

Documentation/devicetree/bindings/media/imx.txt | 57 +
.../bindings/media/video-multiplexer.txt | 59 +
Documentation/media/uapi/mediactl/media-types.rst | 22 +
Documentation/media/v4l-drivers/imx.rst | 443 ++
arch/arm/boot/dts/imx6dl-sabrelite.dts | 5 +
arch/arm/boot/dts/imx6dl-sabresd.dts | 5 +
arch/arm/boot/dts/imx6dl.dtsi | 187 +
arch/arm/boot/dts/imx6q-sabrelite.dts | 6 +
arch/arm/boot/dts/imx6q-sabresd.dts | 5 +
arch/arm/boot/dts/imx6q.dtsi | 127 +
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 147 +-
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 122 +-
arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 +-
arch/arm/boot/dts/imx6qdl.dtsi | 25 +-
arch/arm/configs/imx_v6_v7_defconfig | 12 +-

drivers/media/platform/Kconfig | 8 +
drivers/media/platform/Makefile | 2 +
drivers/media/platform/video-multiplexer.c | 472 +++

drivers/staging/media/Kconfig | 2 +
drivers/staging/media/Makefile | 1 +

drivers/staging/media/imx/Kconfig | 36 +
drivers/staging/media/imx/Makefile | 15 +
drivers/staging/media/imx/TODO | 22 +
drivers/staging/media/imx/imx-camif.c | 1000 +++++
drivers/staging/media/imx/imx-csi.c | 644 +++
drivers/staging/media/imx/imx-ic-common.c | 109 +
drivers/staging/media/imx/imx-ic-pp.c | 636 +++
drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++
drivers/staging/media/imx/imx-ic-prpvf.c | 1179 ++++++
drivers/staging/media/imx/imx-ic.h | 38 +
drivers/staging/media/imx/imx-media-common.c | 981 +++++
drivers/staging/media/imx/imx-media-dev.c | 486 +++
drivers/staging/media/imx/imx-media-fim.c | 471 +++
drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++
drivers/staging/media/imx/imx-media-of.c | 289 ++
drivers/staging/media/imx/imx-media.h | 310 ++
drivers/staging/media/imx/imx-mipi-csi2.c | 501 +++
drivers/staging/media/imx/imx-smfc.c | 737 ++++
drivers/staging/media/imx/ov5640-mipi.c | 2348 +++++++++++
drivers/staging/media/imx/ov5642.c | 4363 ++++++++++++++++++++

include/media/imx.h | 15 +
include/uapi/Kbuild | 1 +

include/uapi/linux/media.h | 6 +

include/uapi/linux/v4l2-controls.h | 4 +
include/uapi/media/Kbuild | 2 +
include/uapi/media/imx.h | 30 +

46 files changed, 17505 insertions(+), 29 deletions(-)
create mode 100644 Documentation/devicetree/bindings/media/imx.txt
create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
create mode 100644 Documentation/media/v4l-drivers/imx.rst
create mode 100644 drivers/media/platform/video-multiplexer.c
create mode 100644 drivers/staging/media/imx/Kconfig
create mode 100644 drivers/staging/media/imx/Makefile
create mode 100644 drivers/staging/media/imx/TODO
create mode 100644 drivers/staging/media/imx/imx-camif.c
create mode 100644 drivers/staging/media/imx/imx-csi.c
create mode 100644 drivers/staging/media/imx/imx-ic-common.c
create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
create mode 100644 drivers/staging/media/imx/imx-ic.h
create mode 100644 drivers/staging/media/imx/imx-media-common.c
create mode 100644 drivers/staging/media/imx/imx-media-dev.c
create mode 100644 drivers/staging/media/imx/imx-media-fim.c
create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
create mode 100644 drivers/staging/media/imx/imx-media-of.c
create mode 100644 drivers/staging/media/imx/imx-media.h
create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
create mode 100644 drivers/staging/media/imx/imx-smfc.c
create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
create mode 100644 drivers/staging/media/imx/ov5642.c
create mode 100644 include/media/imx.h
create mode 100644 include/uapi/media/Kbuild
create mode 100644 include/uapi/media/imx.h

--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:06 PM1/6/17

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
clocks.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
index 53e6e63..42926e9 100644
--- a/arch/arm/boot/dts/imx6qdl.dtsi
+++ b/arch/arm/boot/dts/imx6qdl.dtsi
@@ -1125,7 +1125,14 @@
};

mipi_csi: mipi@021dc000 {
+ compatible = "fsl,imx6-mipi-csi2";
reg = <0x021dc000 0x4000>;
+ interrupts = <0 100 0x04>, <0 101 0x04>;
+ clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+ <&clks IMX6QDL_CLK_VIDEO_27M>,
+ <&clks IMX6QDL_CLK_EIM_SEL>;
+ clock-names = "dphy", "cfg", "pix";
+ status = "disabled";
};

mipi_dsi: mipi@021e0000 {
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:06 PM1/6/17

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 2 +-
drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
2 files changed, 1001 insertions(+), 1 deletion(-)
create mode 100644 drivers/staging/media/imx/imx-camif.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index d2a962c..fe9e992 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile

@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
diff --git a/drivers/staging/media/imx/imx-camif.c b/drivers/staging/media/imx/imx-camif.c
new file mode 100644
index 0000000..404f724
--- /dev/null
+++ b/drivers/staging/media/imx/imx-camif.c
@@ -0,0 +1,1000 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *

+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>

+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>

+#include <media/v4l2-event.h>

+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>

+#include <video/imx-ipu-v3.h>

+#include <media/imx.h>
+#include "imx-media.h"
+

+#define CAMIF_NUM_PADS 2
+
+struct camif_priv {
+ struct device *dev;
+ struct video_device vfd;
+ struct media_pipeline mp;
+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;
+ struct media_pad pad[CAMIF_NUM_PADS];
+ struct media_pad vd_pad;
+ int id;

+ int input_pad;
+ int output_pad;
+

+ struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+ const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+
+ /* dma buffer ring */
+ struct imx_media_dma_buf_ring *in_ring;

+ struct v4l2_subdev *src_sd;
+

+ struct v4l2_ctrl_handler ctrl_hdlr;
+

+ /* misc status */
+ int current_input; /* the current input */
+ v4l2_std_id current_std; /* current standard */
+ bool stop; /* streaming is stopping */
+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT SZ_64M
+
+static struct vb2_ops camif_qops;
+
+/*
+ * Video ioctls follow
+ */
+
+static int vidioc_querycap(struct file *file, void *fh,
+ struct v4l2_capability *cap)
+{
+ strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
+ strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
+ cap->bus_info[0] = 0;
+ cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+ cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

+
+ return 0;
+}
+

+ if (ret)
+ return ret;

+ cc = imx_media_find_format(0, code, true, true);
+ if (!cc)
+ return -EINVAL;
+
+ f->pixelformat = cc->fourcc;

+
+ return 0;
+}
+

+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+ struct v4l2_format *f)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct v4l2_mbus_framefmt *outfmt;
+

+ /* user format is the same as the format from output pad */

+ outfmt = &priv->format_mbus[priv->output_pad];

+ return -EBUSY;
+ }
+

+ return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return 0;
+}
+

+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct imx_media_subdev *sensor;
+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (vb2_is_busy(&priv->buffer_queue))
+ return -EBUSY;
+
+ ret = v4l2_subdev_call(sensor->sd, video, s_std, std);

+ if (ret < 0)
+ return ret;
+

+ priv->current_std = std;

+ return 0;
+}
+

+static int camif_enum_input(struct file *file, void *fh,
+ struct v4l2_input *input)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct imx_media_subdev *sensor;
+ int index = input->index;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (index >= sensor->input.num)
+ return -EINVAL;
+
+ input->type = V4L2_INPUT_TYPE_CAMERA;
+ strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+ if (index == priv->current_input) {
+ v4l2_subdev_call(sensor->sd, video, g_input_status,
+ &input->status);
+ v4l2_subdev_call(sensor->sd, video, querystd, &input->std);
+ } else {
+ input->status = V4L2_IN_ST_NO_SIGNAL;
+ input->std = V4L2_STD_UNKNOWN;

+ }
+
+ return 0;
+}
+

+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+ struct camif_priv *priv = video_drvdata(file);
+
+ *index = priv->current_input;

+ return 0;
+}
+

+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct imx_media_subdev *sensor;
+ int ret;
+
+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }
+
+ if (index >= sensor->input.num)
+ return -EINVAL;
+
+ if (index == priv->current_input)
+ return 0;
+
+ /* select the sensor's input */
+ ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+ sensor->input.value[index], 0, 0);
+ if (!ret)
+ priv->current_input = index;

+
+ return ret;
+}
+

+static int camif_g_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *a)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }

+
+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+

+ return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *a)
+{
+ struct camif_priv *priv = video_drvdata(file);

+ struct imx_media_subdev *sensor;
+

+ sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+ if (IS_ERR(sensor)) {

+ v4l2_err(&priv->sd, "no sensor attached\n");

+ return PTR_ERR(sensor);
+ }

+
+ if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+

+ return v4l2_subdev_call(sensor->sd, video, s_parm, a);
+}
+
+static const struct v4l2_ioctl_ops camif_ioctl_ops = {
+ .vidioc_querycap = vidioc_querycap,
+
+ .vidioc_enum_fmt_vid_cap = camif_enum_fmt_vid_cap,
+ .vidioc_g_fmt_vid_cap = camif_g_fmt_vid_cap,
+ .vidioc_try_fmt_vid_cap = camif_try_fmt_vid_cap,
+ .vidioc_s_fmt_vid_cap = camif_s_fmt_vid_cap,
+
+ .vidioc_querystd = camif_querystd,
+ .vidioc_g_std = camif_g_std,
+ .vidioc_s_std = camif_s_std,
+
+ .vidioc_enum_input = camif_enum_input,
+ .vidioc_g_input = camif_g_input,
+ .vidioc_s_input = camif_s_input,
+
+ .vidioc_g_parm = camif_g_parm,
+ .vidioc_s_parm = camif_s_parm,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+ struct v4l2_mbus_framefmt *outfmt;
+ const struct imx_media_pixfmt *outcc;
+

+ outfmt = &priv->format_mbus[priv->output_pad];

+
+ return 0;
+}
+

+ return -EINVAL;
+ }
+

+ return ret;
+ }
+ }
+

+ ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
+ if (ret)
+ goto free_ring;

+
+ return 0;
+

+free_ring:
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+ return ret;
+}
+

+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+ struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+ struct imx_media_buffer *buf = to_imx_media_vb(vb);

+ unsigned long flags;
+

+
+ return 0;
+

+ return ret;
+}
+

+static int camif_release(struct file *file)
+{
+ struct camif_priv *priv = video_drvdata(file);
+ struct vb2_queue *vq = &priv->buffer_queue;

+ int ret = 0;
+

+ mutex_lock(&priv->mutex);
+
+ if (file->private_data == vq->owner) {
+ vb2_queue_release(vq);
+ vq->owner = NULL;
+ }
+
+ v4l2_fh_release(file);
+ mutex_unlock(&priv->mutex);

+ return ret;
+}
+

+ unsigned long flags;
+

+ goto unlock;
+ }
+

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int camif_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)

+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct video_device *vfd = &priv->vfd;
+ struct v4l2_subdev *remote_sd;

+ int ret = 0;
+

+ dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+ local->entity->name);
+

+ if (is_media_entity_v4l2_video_device(remote->entity))
+ return 0;
+
+ WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);

+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ /* reset controls to refresh with inherited from subdevs */
+ v4l2_ctrl_handler_free(vfd->ctrl_handler);
+ v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);

+
+ if (flags & MEDIA_LNK_FL_ENABLED) {
+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;

+ ret = imx_media_inherit_controls(priv->md, vfd,
+ &priv->src_sd->entity);
+ } else {
+ priv->src_sd = NULL;
+ }
+

+ return ret;
+}
+

+static int camif_enum_mbus_code(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ if (code->pad >= CAMIF_NUM_PADS)
+ return -EINVAL;
+
+ return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+ if (sdformat->pad >= CAMIF_NUM_PADS)
+ return -EINVAL;
+

+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int camif_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ const struct imx_media_pixfmt *cc;
+ u32 code;
+
+ if (sdformat->pad >= CAMIF_NUM_PADS)
+ return -EINVAL;
+
+ if (vb2_is_busy(&priv->buffer_queue)) {
+ v4l2_err(&priv->sd, "%s queue busy\n", __func__);

+ return -EBUSY;
+ }
+

+ cc = imx_media_find_format(0, sdformat->format.code, true, true);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, true);
+ cc = imx_media_find_format(0, code, true, true);

+ sdformat->format.code = cc->codes[0];
+ }
+

+ /* Output pad mirrors input pad, no limitations on input pads */

+ if (sdformat->pad == priv->output_pad)

+ sdformat->format = priv->format_mbus[priv->input_pad];

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {
+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int camif_init_pads(struct camif_priv *priv)
+{
+ struct video_device *vfd = &priv->vfd;

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;

+ int i, ret;
+

+ imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+
+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {

+ v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+ imxsd->num_sink_pads, imxsd->num_src_pads);

+ return -EINVAL;
+ }
+

+ priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
+ if (ret) {
+ v4l2_err(&priv->sd, "failed to init device node pad\n");

+ return ret;
+ }
+

+ for (i = 0; i < CAMIF_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];
+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;
+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,
+ &priv->cc[i]);

+ if (ret)
+ return ret;
+ }
+

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ int ret;
+

+ /* get media device */
+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+

+ vfd->v4l2_dev = sd->v4l2_dev;
+
+ ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+ if (ret) {
+ v4l2_err(sd, "Failed to register video device\n");

+ return ret;
+ }
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ return ret;
+}
+

+static void camif_unregistered(struct v4l2_subdev *sd)
+{
+ struct camif_priv *priv = v4l2_get_subdevdata(sd);
+ struct video_device *vfd = &priv->vfd;
+
+ mutex_lock(&priv->mutex);
+
+ if (video_is_registered(vfd)) {
+ video_unregister_device(vfd);
+ media_entity_cleanup(&vfd->entity);
+ }
+
+ mutex_unlock(&priv->mutex);
+}
+
+static const struct v4l2_subdev_internal_ops camif_internal_ops = {
+ .registered = camif_registered,
+ .unregistered = camif_unregistered,
+};
+
+static struct v4l2_subdev_core_ops camif_core_ops = {
+ .ioctl = camif_ioctl,
+};
+
+static struct media_entity_operations camif_entity_ops = {
+ .link_setup = camif_link_setup,
+ .link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+ .enum_mbus_code = camif_enum_mbus_code,
+ .get_fmt = camif_get_fmt,
+ .set_fmt = camif_set_fmt,
+};
+
+static struct v4l2_subdev_ops camif_subdev_ops = {
+ .pad = &camif_pad_ops,
+ .core = &camif_core_ops,
+};
+
+static int camif_probe(struct platform_device *pdev)

+{
+ struct imx_media_internal_sd_platformdata *pdata;

+ struct camif_priv *priv;
+ struct video_device *vfd;
+ int ret;
+

+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)

+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, priv);

+ priv->dev = &pdev->dev;
+
+ pdata = priv->dev->platform_data;
+

+ priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ /* get our group id and camif id */

+ priv->sd.grp_id = pdata->grp_id;

+ priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));

+ snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+ "%s devnode", pdata->sd_name);
+

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+

+ vfd = &priv->vfd;
+ *vfd = camif_videodev;
+ vfd->lock = &priv->mutex;
+ vfd->queue = &priv->buffer_queue;
+
+ video_set_drvdata(vfd, priv);

+
+ v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+

+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+
+ return ret;
+}
+

+static int camif_remove(struct platform_device *pdev)
+{
+ struct camif_priv *priv =
+ (struct camif_priv *)platform_get_drvdata(pdev);
+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(&priv->sd);

+
+ return 0;
+}
+

+static const struct platform_device_id camif_ids[] = {
+ { .name = "imx-media-camif" },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, camif_ids);

+
+static struct platform_driver imx_camif_driver = {
+ .probe = camif_probe,
+ .remove = camif_remove,

+ .driver = {
+ .name = "imx-media-camif",
+ },
+};
+
+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:06 PM1/6/17

Adds MIPI CSI-2 Receiver subdev driver. This subdev is required
for sensors with a MIPI CSI2 interface.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 1 +
drivers/staging/media/imx/imx-mipi-csi2.c | 501 ++++++++++++++++++++++++++++++
2 files changed, 502 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c

diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
index fe9e992..0decef7 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-mipi-csi2.o
diff --git a/drivers/staging/media/imx/imx-mipi-csi2.c b/drivers/staging/media/imx/imx-mipi-csi2.c

new file mode 100644
index 0000000..daa6e1d
--- /dev/null
+++ b/drivers/staging/media/imx/imx-mipi-csi2.c
@@ -0,0 +1,501 @@

+/*
+ * MIPI CSI-2 Receiver Subdev for Freescale i.MX5/6 SOC.

+#include <linux/clk.h>

+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>

+#include <linux/irq.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+/*
+ * there must be 5 pads: 1 input pad from sensor, and
+ * the 4 virtual channel output pads
+ */
+#define CSI2_NUM_SINK_PADS 1
+#define CSI2_NUM_SRC_PADS 4
+#define CSI2_NUM_PADS 5
+
+struct imxcsi2_dev {
+ struct device *dev;

+ struct imx_media_dev *md;
+ struct v4l2_subdev sd;

+ struct media_pad pad[CSI2_NUM_PADS];
+ struct v4l2_mbus_framefmt format_mbus;
+ struct v4l2_subdev *src_sd;
+ struct v4l2_subdev *sink_sd[CSI2_NUM_SRC_PADS];
+ int input_pad;
+ struct clk *dphy_clk;
+ struct clk *cfg_clk;
+ struct clk *pix_clk; /* what is this? */
+ void __iomem *base;
+ int intr1;
+ int intr2;
+ struct v4l2_of_bus_mipi_csi2 bus;
+ bool on;
+ bool stream_on;
+};
+
+#define DEVICE_NAME "imx6-mipi-csi2"
+
+/* Register offsets */
+#define CSI2_VERSION 0x000
+#define CSI2_N_LANES 0x004
+#define CSI2_PHY_SHUTDOWNZ 0x008
+#define CSI2_DPHY_RSTZ 0x00c
+#define CSI2_RESETN 0x010
+#define CSI2_PHY_STATE 0x014
+#define CSI2_DATA_IDS_1 0x018
+#define CSI2_DATA_IDS_2 0x01c
+#define CSI2_ERR1 0x020
+#define CSI2_ERR2 0x024
+#define CSI2_MSK1 0x028
+#define CSI2_MSK2 0x02c
+#define CSI2_PHY_TST_CTRL0 0x030
+#define CSI2_PHY_TST_CTRL1 0x034
+#define CSI2_SFT_RESET 0xf00
+
+static inline struct imxcsi2_dev *sd_to_dev(struct v4l2_subdev *sdev)
+{
+ return container_of(sdev, struct imxcsi2_dev, sd);
+}
+
+static inline u32 imxcsi2_read(struct imxcsi2_dev *csi2, unsigned int regoff)
+{
+ return readl(csi2->base + regoff);
+}
+
+static inline void imxcsi2_write(struct imxcsi2_dev *csi2, u32 val,
+ unsigned int regoff)
+{
+ writel(val, csi2->base + regoff);
+}
+
+static void imxcsi2_set_lanes(struct imxcsi2_dev *csi2)
+{
+ int lanes = csi2->bus.num_data_lanes;
+
+ imxcsi2_write(csi2, lanes - 1, CSI2_N_LANES);
+}
+
+static void imxcsi2_enable(struct imxcsi2_dev *csi2, bool enable)
+{
+ if (enable) {
+ imxcsi2_write(csi2, 0xffffffff, CSI2_PHY_SHUTDOWNZ);
+ imxcsi2_write(csi2, 0xffffffff, CSI2_DPHY_RSTZ);
+ imxcsi2_write(csi2, 0xffffffff, CSI2_RESETN);
+ } else {
+ imxcsi2_write(csi2, 0x0, CSI2_PHY_SHUTDOWNZ);
+ imxcsi2_write(csi2, 0x0, CSI2_DPHY_RSTZ);
+ imxcsi2_write(csi2, 0x0, CSI2_RESETN);
+ }
+}
+
+static void imxcsi2_reset(struct imxcsi2_dev *csi2)
+{
+ imxcsi2_enable(csi2, false);
+
+ imxcsi2_write(csi2, 0x00000001, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL1);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00010044, CSI2_PHY_TST_CTRL1);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000014, CSI2_PHY_TST_CTRL1);
+ imxcsi2_write(csi2, 0x00000002, CSI2_PHY_TST_CTRL0);
+ imxcsi2_write(csi2, 0x00000000, CSI2_PHY_TST_CTRL0);
+
+ imxcsi2_enable(csi2, true);
+}
+
+static int imxcsi2_dphy_wait(struct imxcsi2_dev *csi2)
+{
+ u32 reg;
+ int i;
+
+ /* wait for mipi sensor ready */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+ if (reg != 0x200)
+ break;
+ usleep_range(10000, 20000);
+ }
+
+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for clock lane timeout, phy_state = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ /* wait for mipi stable */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_ERR1);
+ if (reg == 0x0)
+ break;
+ usleep_range(10000, 20000);
+ }
+
+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for controller timeout, err1 = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ /* finally let's wait for active clock on the clock lane */
+ for (i = 0; i < 50; i++) {
+ reg = imxcsi2_read(csi2, CSI2_PHY_STATE);
+ if (reg & (1 << 8))
+ break;
+ usleep_range(10000, 20000);
+ }
+
+ if (i >= 50) {
+ v4l2_err(&csi2->sd,
+ "wait for active clock timeout, phy_state = 0x%08x\n",
+ reg);
+ return -ETIME;
+ }
+
+ v4l2_info(&csi2->sd, "ready, dphy version 0x%x\n",
+ imxcsi2_read(csi2, CSI2_VERSION));

+
+ return 0;
+}
+

+/*
+ * V4L2 subdev operations
+ */
+
+static int imxcsi2_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(csi2->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+

+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+

+ if (local->flags & MEDIA_PAD_FL_SOURCE) {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (csi2->sink_sd[local->index])
+ return -EBUSY;
+ csi2->sink_sd[local->index] = remote_sd;
+ } else {
+ csi2->sink_sd[local->index] = NULL;
+ }
+ } else {

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (csi2->src_sd)
+ return -EBUSY;
+ csi2->src_sd = remote_sd;
+ } else {
+ csi2->src_sd = NULL;
+ }
+ }
+

+ return 0;
+}
+

+static int imxcsi2_s_power(struct v4l2_subdev *sd, int on)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ if (on && !csi2->on) {
+ v4l2_info(&csi2->sd, "power ON\n");
+ clk_prepare_enable(csi2->cfg_clk);
+ clk_prepare_enable(csi2->dphy_clk);
+ imxcsi2_set_lanes(csi2);
+ imxcsi2_reset(csi2);
+ } else if (!on && csi2->on) {
+ v4l2_info(&csi2->sd, "power OFF\n");
+ imxcsi2_enable(csi2, false);
+ clk_disable_unprepare(csi2->dphy_clk);
+ clk_disable_unprepare(csi2->cfg_clk);
+ }
+
+ csi2->on = on;

+ return 0;
+}
+

+static int imxcsi2_s_stream(struct v4l2_subdev *sd, int enable)
+{

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);

+ int i, ret = 0;
+
+ if (!csi2->src_sd)
+ return -EPIPE;

+ for (i = 0; i < CSI2_NUM_SRC_PADS; i++) {
+ if (csi2->sink_sd[i])
+ break;
+ }
+ if (i >= CSI2_NUM_SRC_PADS)

+ return -EPIPE;
+

+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !csi2->stream_on) {

+ ret = clk_prepare_enable(csi2->pix_clk);

+ if (ret)
+ return ret;
+

+ ret = imxcsi2_dphy_wait(csi2);
+ if (ret) {
+ clk_disable_unprepare(csi2->pix_clk);
+ return ret;
+ }
+ } else if (!enable && csi2->stream_on) {
+ clk_disable_unprepare(csi2->pix_clk);
+ }
+
+ csi2->stream_on = enable;

+ return 0;
+}
+

+static int imxcsi2_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ sdformat->format = csi2->format_mbus;

+
+ return 0;
+}
+

+static int imxcsi2_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+
+ if (sdformat->pad >= CSI2_NUM_PADS)
+ return -EINVAL;
+
+ if (csi2->stream_on)
+ return -EBUSY;
+
+ /* Output pads mirror active input pad, no limits on input pads */
+ if (sdformat->pad != csi2->input_pad)
+ sdformat->format = csi2->format_mbus;

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY)
+ cfg->try_fmt = sdformat->format;
+ else

+ csi2->format_mbus = sdformat->format;

+
+ return 0;
+}
+

+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int imxcsi2_registered(struct v4l2_subdev *sd)
+{
+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+

+ /* get media device */

+ csi2->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(csi2->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 4)
+ return -EINVAL;
+
+ for (i = 0; i < CSI2_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ csi2->pad[i] = pad->pad;
+ if (csi2->pad[i].flags & MEDIA_PAD_FL_SINK)
+ csi2->input_pad = i;
+ }

+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&csi2->format_mbus,
+ 640, 480, 0, V4L2_FIELD_NONE, NULL);

+ if (ret)
+ return ret;
+

+ return media_entity_pads_init(&sd->entity, CSI2_NUM_PADS, csi2->pad);
+}
+
+static struct media_entity_operations imxcsi2_entity_ops = {
+ .link_setup = imxcsi2_link_setup,

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_core_ops imxcsi2_core_ops = {
+ .s_power = imxcsi2_s_power,
+};
+
+static struct v4l2_subdev_video_ops imxcsi2_video_ops = {
+ .s_stream = imxcsi2_s_stream,
+};
+
+static struct v4l2_subdev_pad_ops imxcsi2_pad_ops = {
+ .get_fmt = imxcsi2_get_fmt,
+ .set_fmt = imxcsi2_set_fmt,
+};
+
+static struct v4l2_subdev_ops imxcsi2_subdev_ops = {
+ .core = &imxcsi2_core_ops,
+ .video = &imxcsi2_video_ops,
+ .pad = &imxcsi2_pad_ops,
+};
+
+static struct v4l2_subdev_internal_ops imxcsi2_internal_ops = {
+ .registered = imxcsi2_registered,
+};

+
+static int imxcsi2_parse_endpoints(struct imxcsi2_dev *csi2)
+{
+ struct device_node *node = csi2->dev->of_node;
+ struct device_node *epnode;
+ struct v4l2_of_endpoint ep;
+

+ epnode = of_graph_get_next_endpoint(node, NULL);
+ if (!epnode) {
+ v4l2_err(&csi2->sd, "failed to get endpoint node\n");

+ return -EINVAL;
+ }
+

+ v4l2_of_parse_endpoint(epnode, &ep);
+ of_node_put(epnode);
+
+ if (ep.bus_type != V4L2_MBUS_CSI2) {
+ v4l2_err(&csi2->sd, "invalid bus type, must be MIPI CSI2\n");

+ return -EINVAL;
+ }
+

+ csi2->bus = ep.bus.mipi_csi2;
+
+ v4l2_info(&csi2->sd, "data lanes: %d\n", csi2->bus.num_data_lanes);
+ v4l2_info(&csi2->sd, "flags: 0x%08x\n", csi2->bus.flags);

+ return 0;
+}
+

+static int imxcsi2_probe(struct platform_device *pdev)
+{
+ struct imxcsi2_dev *csi2;
+ struct resource *res;
+ int ret;
+
+ csi2 = devm_kzalloc(&pdev->dev, sizeof(*csi2), GFP_KERNEL);
+ if (!csi2)
+ return -ENOMEM;
+
+ csi2->dev = &pdev->dev;
+
+ v4l2_subdev_init(&csi2->sd, &imxcsi2_subdev_ops);
+ v4l2_set_subdevdata(&csi2->sd, &pdev->dev);
+ csi2->sd.internal_ops = &imxcsi2_internal_ops;
+ csi2->sd.entity.ops = &imxcsi2_entity_ops;
+ csi2->sd.dev = &pdev->dev;
+ csi2->sd.owner = THIS_MODULE;
+ csi2->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+ strcpy(csi2->sd.name, DEVICE_NAME);
+ csi2->sd.entity.function = MEDIA_ENT_F_VID_IF_BRIDGE;
+ csi2->sd.grp_id = IMX_MEDIA_GRP_ID_CSI2;
+
+ ret = imxcsi2_parse_endpoints(csi2);

+ if (ret)
+ return ret;
+

+ csi2->cfg_clk = devm_clk_get(&pdev->dev, "cfg");
+ if (IS_ERR(csi2->cfg_clk)) {
+ v4l2_err(&csi2->sd, "failed to get cfg clock\n");
+ ret = PTR_ERR(csi2->cfg_clk);

+ return ret;
+ }
+

+ csi2->dphy_clk = devm_clk_get(&pdev->dev, "dphy");
+ if (IS_ERR(csi2->dphy_clk)) {
+ v4l2_err(&csi2->sd, "failed to get dphy clock\n");
+ ret = PTR_ERR(csi2->dphy_clk);

+ return ret;
+ }
+

+ csi2->pix_clk = devm_clk_get(&pdev->dev, "pix");
+ if (IS_ERR(csi2->pix_clk)) {
+ v4l2_err(&csi2->sd, "failed to get pixel clock\n");
+ ret = PTR_ERR(csi2->pix_clk);

+ return ret;
+ }
+

+ return -ENODEV;
+ }
+

+ csi2->base = devm_ioremap(&pdev->dev, res->start, PAGE_SIZE);
+ if (!csi2->base) {
+ v4l2_err(&csi2->sd, "failed to map CSI-2 registers\n");

+ return -ENOMEM;
+ }
+

+ platform_set_drvdata(pdev, &csi2->sd);
+
+ return v4l2_async_register_subdev(&csi2->sd);
+}
+
+static int imxcsi2_remove(struct platform_device *pdev)
+{

+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imxcsi2_dev *csi2 = sd_to_dev(sd);
+

+ imxcsi2_s_power(sd, 0);
+
+ v4l2_async_unregister_subdev(&csi2->sd);
+ media_entity_cleanup(&csi2->sd.entity);
+ v4l2_device_unregister_subdev(sd);

+
+ return 0;
+}
+

+static const struct of_device_id imxcsi2_dt_ids[] = {

+ { .compatible = "fsl,imx6-mipi-csi2", },

+ { /* sentinel */ }
+};

+MODULE_DEVICE_TABLE(of, imxcsi2_dt_ids);
+
+static struct platform_driver imxcsi2_driver = {

+ .driver = {
+ .name = DEVICE_NAME,
+ .of_match_table = imxcsi2_dt_ids,
+ },
+ .probe = imxcsi2_probe,
+ .remove = imxcsi2_remove,
+};
+
+module_platform_driver(imxcsi2_driver);
+
+MODULE_DESCRIPTION("i.MX5/6 MIPI CSI-2 Receiver driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+
--
2.7.4

Steve Longerbeam

unread,

Jan 6, 2017, 9:20:07 PM1/6/17

This is a set of three media entity subdevice drivers for the i.MX
Image Converter. The i.MX IC module contains three independent
"tasks":

- Pre-processing Encode task: video frames are routed directly from
the CSI and can be scaled, color-space converted, and rotated.
Scaled output is limited to 1024x1024 resolution. Output frames
are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same
conversions as the pre-process encode task, but in addition can
be used for hardware motion compensated deinterlacing. Frames can
come either directly from the CSI or from the SMFC entities (memory
buffers via IDMAC channels). Scaled output is limited to 1024x1024
resolution. Output frames can be routed to various sinks including
the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However
this entity sends frames to the i.MX IPU image converter which supports
image tiling, which allows scaled output up to 4096x4096 resolution.
Output frames can be routed to the camera interfaces.

Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
---

drivers/staging/media/imx/Makefile | 2 +
drivers/staging/media/imx/imx-ic-common.c | 109 +++
drivers/staging/media/imx/imx-ic-pp.c | 636 ++++++++++++++++
drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
drivers/staging/media/imx/imx-ic-prpvf.c | 1179 +++++++++++++++++++++++++++++
drivers/staging/media/imx/imx-ic.h | 38 +
6 files changed, 2997 insertions(+)

imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \

imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o

obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o

+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o

obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o

diff --git a/drivers/staging/media/imx/imx-ic-common.c b/drivers/staging/media/imx/imx-ic-common.c
new file mode 100644
index 0000000..45706ca
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-common.c
@@ -0,0 +1,109 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>

+#include <media/v4l2-subdev.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+ [IC_TASK_ENCODER] = &imx_ic_prpenc_ops,
+ [IC_TASK_VIEWFINDER] = &imx_ic_prpvf_ops,
+ [IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+
+static int imx_ic_probe(struct platform_device *pdev)
+{
+ struct imx_media_internal_sd_platformdata *pdata;
+ struct imx_ic_priv *priv;
+ int ret;
+

+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)

+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, &priv->sd);
+ priv->dev = &pdev->dev;
+
+ /* get our ipu_id, grp_id and IC task id */

+ pdata = priv->dev->platform_data;

+ priv->ipu_id = pdata->ipu_id;

+ switch (pdata->grp_id) {
+ case IMX_MEDIA_GRP_ID_IC_PRPENC:
+ priv->task_id = IC_TASK_ENCODER;

+ break;
+ case IMX_MEDIA_GRP_ID_IC_PRPVF:

+ priv->task_id = IC_TASK_VIEWFINDER;
+ break;
+ case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+ priv->task_id = IC_TASK_POST_PROCESSOR;
+ break;
+ default:

+ return -EINVAL;
+ }
+

+ priv->sd.dev = &pdev->dev;
+ priv->sd.owner = THIS_MODULE;

+ priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;

+ priv->sd.grp_id = pdata->grp_id;

+ strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+

+ ret = ic_ops[priv->task_id]->init(priv);

+ if (ret)
+ return ret;
+

+ ret = v4l2_async_register_subdev(&priv->sd);
+ if (ret)

+ ic_ops[priv->task_id]->remove(priv);
+

+ return ret;
+}
+

+static int imx_ic_remove(struct platform_device *pdev)

+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);

+ struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+

+ ic_ops[priv->task_id]->remove(priv);
+

+ v4l2_async_unregister_subdev(&priv->sd);
+ media_entity_cleanup(&priv->sd.entity);

+ v4l2_device_unregister_subdev(sd);
+
+ return 0;
+}
+

+static const struct platform_device_id imx_ic_ids[] = {
+ { .name = "imx-ipuv3-ic" },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, imx_ic_ids);

+
+static struct platform_driver imx_ic_driver = {
+ .probe = imx_ic_probe,
+ .remove = imx_ic_remove,
+ .id_table = imx_ic_ids,

+ .driver = {
+ .name = "imx-ipuv3-ic",
+ },
+};
+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");

+MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
+MODULE_LICENSE("GPL");

+MODULE_ALIAS("platform:imx-ipuv3-ic");
diff --git a/drivers/staging/media/imx/imx-ic-pp.c b/drivers/staging/media/imx/imx-ic-pp.c
new file mode 100644
index 0000000..1f75616
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-pp.c
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>

+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>

+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>

+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-image-convert.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {
+ struct imx_media_dev *md;
+ struct imx_ic_priv *ic_priv;
+ int pp_id;
+
+ struct ipu_soc *ipu;
+ struct ipu_image_convert_ctx *ic_ctx;
+
+ struct media_pad pad[PP_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ bool stop; /* streaming is stopping */

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);

+
+ done = imx_media_dma_buf_get_active(priv->out_ring);
+ /* give the completed buffer to the sink */
+ if (!WARN_ON(!done))

+ spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+

+ unsigned long flags;
+

+ switch (cmd) {
+ case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:

+ /* src asks for a buffer ring */
+ if (!priv->in_ring)

+ return -EINVAL;
+ ring = (struct imx_media_dma_buf_ring **)arg;
+ *ring = priv->in_ring;
+ break;
+ case IMX_MEDIA_NEW_DMA_BUF:

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ return ret;
+

+ imx_media_mbus_fmt_to_ipu_image(&image_in,

+ &priv->format_mbus[priv->input_pad]);

+ infmt = &priv->format_mbus[priv->input_pad];

+ incc = priv->cc[priv->input_pad];

+ in_size = (infmt->width * incc->bpp * infmt->height) >> 3;

+
+ if (priv->in_ring) {

+ v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ }

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ goto out_unprep;
+ }
+
+ for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+ imx_media_dma_buf_queue(priv->in_ring, i);

+
+ priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+ sizeof(*priv->out_run), GFP_KERNEL);
+ if (!priv->out_run) {

+ v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+ ret = -ENOMEM;
+ goto out_free_ring;

+ }
+
+ priv->stop = false;

+
+ return 0;
+

+out_free_ring:

+ imx_media_free_dma_buf_ring(priv->in_ring);
+ priv->in_ring = NULL;

+out_unprep:
+ ipu_image_convert_unprepare(priv->ic_ctx);

+ return ret;
+}
+

+static void pp_stop(struct pp_priv *priv)
+{

+ unsigned long flags;
+

+ spin_lock_irqsave(&priv->irqlock, flags);

+ priv->stop = true;

+ spin_unlock_irqrestore(&priv->irqlock, flags);
+
+ ipu_image_convert_unprepare(priv->ic_ctx);
+ kfree(priv->out_run);

+
+ priv->out_ring = NULL;
+

+ /* inform sink that its sink buffer ring can now be freed */

+ v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+

+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct pp_priv *priv = sd_to_priv(sd);
+ int ret = 0;
+
+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !priv->stream_on)

+ ret = pp_start(priv);
+ else if (!enable && priv->stream_on)
+ pp_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int pp_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_mbus_code_enum *code)

+{
+ const struct imx_media_pixfmt *cc;

+ u32 fourcc;
+ int ret;
+
+ if (code->pad >= PP_NUM_PADS)
+ return -EINVAL;
+
+ ret = ipu_image_convert_enum_format(code->index, &fourcc);

+ if (ret)
+ return ret;
+

+ /* convert returned fourcc to mbus code */
+ cc = imx_media_find_format(fourcc, 0, true, true);
+ if (WARN_ON(!cc))
+ return -EINVAL;
+
+ code->code = cc->codes[0];

+ return 0;
+}
+

+static int pp_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct pp_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PP_NUM_PADS)
+ return -EINVAL;
+

+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int pp_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct pp_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ struct ipu_image test_in, test_out;
+ u32 code;
+
+ if (sdformat->pad >= PP_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+
+ if (sdformat->pad == priv->output_pad) {
+ imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
+ imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+ ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+ imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
+ } else {
+ imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
+ imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+ ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+ imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
+ }

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ if (sdformat->pad == priv->output_pad) {

+ *outfmt = sdformat->format;
+ imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+ } else {
+ *infmt = sdformat->format;
+ imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+ }

+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int pp_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct pp_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;

+ }
+ } else {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;

+ }
+ }
+
+ return 0;
+}
+

+ return -EINVAL;
+ }
+

+ ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);

+ if (ret)
+ return ret;
+

+ if (rot_mode != priv->rot_mode) {

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ struct ipu_image test_in, test_out;
+
+ /* can't change rotation mid-streaming */
+ if (priv->stream_on)

+ return -EBUSY;
+
+ /*

+ * make sure this rotation will work with current input/output
+ * formats before setting
+ */

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+ imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+ ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);

+ if (ret)
+ return ret;
+

+ priv->rot_mode = rot_mode;
+ priv->rotation = rotation;
+ priv->hflip = hflip;
+ priv->vflip = vflip;

+ }
+
+ return 0;
+}
+

+ int i, ret;
+

+ return ret;
+ }
+

+ v4l2_ctrl_handler_setup(hdlr);

+
+ return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int pp_registered(struct v4l2_subdev *sd)
+{
+ struct pp_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+
+ /* get media device */

+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+ return -EINVAL;
+
+ for (i = 0; i < PP_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;

+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);

+ if (ret)
+ return ret;
+ }
+

+ ret = pp_init_controls(priv);

+ if (ret)
+ return ret;
+

+ ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
+ if (ret)

+ goto free_ctrls;
+
+ return 0;

+free_ctrls:
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);

+ return ret;
+}
+

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+static struct v4l2_subdev_ops pp_subdev_ops = {
+ .video = &pp_video_ops,
+ .pad = &pp_pad_ops,
+ .core = &pp_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops pp_internal_ops = {
+ .registered = pp_registered,
+};
+
+static int pp_init(struct imx_ic_priv *ic_priv)
+{
+ struct pp_priv *priv;
+
+ priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ ic_priv->task_priv = priv;
+ priv->ic_priv = ic_priv;
+ spin_lock_init(&priv->irqlock);
+
+ /* get our PP id */
+ priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;

+
+ return 0;
+}
+

+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+ struct pp_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

+
+struct imx_ic_ops imx_ic_pp_ops = {
+ .subdev_ops = &pp_subdev_ops,
+ .internal_ops = &pp_internal_ops,
+ .entity_ops = &pp_entity_ops,
+ .init = pp_init,
+ .remove = pp_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpenc.c b/drivers/staging/media/imx/imx-ic-prpenc.c
new file mode 100644
index 0000000..3d85a82
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpenc.c
@@ -0,0 +1,1033 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *

+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.

+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>

+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>

+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"

+#include "imx-ic.h"
+
+#define PRPENC_NUM_PADS 2
+
+#define MAX_W_IC 1024
+#define MAX_H_IC 1024
+#define MAX_W_SINK 4096
+#define MAX_H_SINK 4096
+
+struct prpenc_priv {
+ struct imx_media_dev *md;
+ struct imx_ic_priv *ic_priv;
+
+ /* IPU units we require */
+ struct ipu_soc *ipu;
+ struct ipu_ic *ic_enc;
+
+ struct media_pad pad[PRPENC_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ struct ipuv3_channel *enc_ch;
+ struct ipuv3_channel *enc_rot_in_ch;
+ struct ipuv3_channel *enc_rot_out_ch;

+
+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+ struct imx_media_dma_buf *next;
+
+ int ipu_buf_num; /* ipu double buffer index: 0-1 */
+

+ struct v4l2_subdev *src_sd;
+ struct v4l2_subdev *sink_sd;
+
+ /* the CSI id at link validate */
+ int csi_id;
+

+ /* the attached sensor at stream on */

+ struct imx_media_subdev *sensor;
+

+ goto out;
+ }
+

+ }
+
+ return 0;

+out:
+ prpenc_put_ipu_resources(priv);

+ return ret;
+}
+

+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+ struct prpenc_priv *priv = dev_id;

+ struct imx_media_dma_buf *done, *next;

+ struct ipuv3_channel *channel;

+
+ spin_lock(&priv->irqlock);
+
+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;

+ goto unlock;
+ }
+

+ /* inform CSI of this EOF so it can monitor frame intervals */
+ v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+ 0, NULL);
+

+ channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+ priv->enc_rot_out_ch : priv->enc_ch;

+ if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
+ ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);

+
+ /* get next queued buffer */
+ next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+

+ ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+ ipu_idmac_select_buffer(channel, priv->ipu_buf_num);

+
+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ priv->next = next;
+
+unlock:
+ spin_unlock(&priv->irqlock);
+ return IRQ_HANDLED;
+}
+

+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct prpenc_priv *priv = dev_id;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void prpenc_eof_timeout(unsigned long data)
+{
+ struct prpenc_priv *priv = (struct prpenc_priv *)data;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ unsigned int burst_size;
+ struct ipu_image image;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ if (rot_swap_width_height)
+ swap(outfmt->width, outfmt->height);
+
+ ipu_cpmem_zero(channel);

+
+ imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;
+ }
+

+ ipu_cpmem_set_burstsize(channel, burst_size);
+
+ if (rot_mode)
+ ipu_cpmem_set_rotation(channel, rot_mode);

+
+ if (outfmt->field == V4L2_FIELD_NONE &&
+ (V4L2_FIELD_HAS_BOTH(infmt->field) ||

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ struct imx_media_dma_buf *buf0, *buf1;

+ int out_size, ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+
+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->enc_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_ch, 1);
+ ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);

+
+ /* enable the channels */

+
+ return 0;
+

+free_rot1:
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:
+ imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);

+ return ret;
+}
+

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;

+ struct imx_media_dma_buf *buf0, *buf1;

+ int ret;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ return ret;
+ }
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ priv->next = buf1;
+

+
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->enc_ch, 0);
+ ipu_idmac_select_buffer(priv->enc_ch, 1);

+
+ /* enable the channels */

+ ipu_idmac_enable_channel(priv->enc_ch);
+
+ /* enable the IC ENCODE task */
+ ipu_ic_task_enable(priv->ic_enc);

+
+ return 0;
+}
+

+ return -EINVAL;
+ }
+

+ ret = prpenc_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ /* set IC to receive from CSI */
+ ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ goto out_put_ipu;
+
+ priv->ipu_buf_num = 0;
+

+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ if (ipu_rot_mode_is_irt(priv->rot_mode))
+ ret = prpenc_setup_rotation(priv);
+ else
+ ret = prpenc_setup_norotation(priv);
+ if (ret)
+ goto out_put_ipu;

+
+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->enc_ch,
+ IPU_IRQ_NFB4EOF);
+ ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+ prpenc_nfb4eof_interrupt, 0,
+ "imx-ic-prpenc-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);
+ goto out_unsetup;
+ }
+

+
+ return 0;
+

+ return ret;
+}
+

+static void prpenc_stop(struct prpenc_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ unsigned long flags;
+ int ret;
+

+ /* mark next EOF interrupt as the last before stream off */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ priv->last_eof = true;
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+
+ /*

+ * and then wait for interrupt handler to mark completion.
+ */
+ ret = wait_for_completion_timeout(
+ &priv->last_eof_comp,

+ msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+ if (ret == 0)

+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,

+ struct v4l2_subdev_mbus_code_enum *code)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);
+ bool allow_planar;
+
+ if (code->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+
+ allow_planar = (code->pad == priv->output_pad);
+

+ return imx_media_enum_format(&code->code, code->index,

+ true, allow_planar);
+}
+
+static int prpenc_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+

+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int prpenc_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpenc_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ bool allow_planar;
+ u32 code;

+
+ if (sdformat->pad >= PRPENC_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+

+ cc = imx_media_find_format(0, sdformat->format.code,

+ true, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, true, false);
+ cc = imx_media_find_format(0, code, true, false);

+ sdformat->format.code = cc->codes[0];
+ }

+ if (sdformat->pad == priv->output_pad) {

+ sdformat->format.width = min_t(__u32,
+ sdformat->format.width,
+ MAX_W_IC);
+ sdformat->format.height = min_t(__u32,
+ sdformat->format.height,
+ MAX_H_IC);
+

+ if (sdformat->format.field != V4L2_FIELD_NONE)
+ sdformat->format.field = infmt->field;
+

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int prpenc_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+ struct prpenc_priv *priv = ic_priv->task_priv;

+ struct v4l2_subdev *remote_sd;
+

+ dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+
+ /* this is sink pad */

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->src_sd)
+ return -EBUSY;
+ priv->src_sd = remote_sd;
+ } else {
+ priv->src_sd = NULL;

+ return 0;
+ }
+

+ switch (remote_sd->grp_id) {

+ case IMX_MEDIA_GRP_ID_CSI0:
+ priv->csi_id = 0;
+ break;
+ case IMX_MEDIA_GRP_ID_CSI1:

+ priv->csi_id = 1;
+ break;
+ default:

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static int prpenc_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);
+ priv->sensor = NULL;

+ return ret;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {

+ int vc_num = 0;

+ /* see NOTE in imx-csi.c */
+#if 0

+ vc_num = imx_media_find_mipi_csi2_channel(
+ priv->md, &ic_priv->sd.entity);
+ if (vc_num < 0)
+ return vc_num;
+#endif
+ /* only virtual channel 0 can be sent to IC */
+ if (vc_num != 0)
+ return -EINVAL;
+ } else {
+ /*
+ * only 8-bit pixels can be sent to IC for parallel
+ * busses
+ */
+ if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+ return -EINVAL;
+ }
+

+ ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);

+ if (ret)
+ return ret;
+

+ }
+
+ return 0;
+}
+

+ int i, ret;
+

+ return ret;
+}
+

+static int prpenc_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct prpenc_priv *priv = sd_to_priv(sd);
+ int ret = 0;
+
+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !priv->stream_on)

+ ret = prpenc_start(priv);
+ else if (!enable && priv->stream_on)
+ prpenc_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;
+ return ret;

+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int prpenc_registered(struct v4l2_subdev *sd)
+{
+ struct prpenc_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+
+ /* get media device */

+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+ return -EINVAL;
+
+ for (i = 0; i < PRPENC_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;

+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);

+ if (ret)
+ return ret;
+ }
+

+ ret = prpenc_init_controls(priv);

+ if (ret)
+ return ret;
+

+ ret = media_entity_pads_init(&sd->entity, PRPENC_NUM_PADS, priv->pad);

+ if (ret)
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+

+ return ret;
+}
+

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = prpenc_eof_timeout;

+
+ return 0;
+}
+

+static void prpenc_remove(struct imx_ic_priv *ic_priv)
+{
+ struct prpenc_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

+
+struct imx_ic_ops imx_ic_prpenc_ops = {
+ .subdev_ops = &prpenc_subdev_ops,
+ .internal_ops = &prpenc_internal_ops,
+ .entity_ops = &prpenc_entity_ops,
+ .init = prpenc_init,
+ .remove = prpenc_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic-prpvf.c b/drivers/staging/media/imx/imx-ic-prpvf.c
new file mode 100644
index 0000000..dc2d402
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic-prpvf.c
@@ -0,0 +1,1179 @@
+/*
+ * V4L2 IC Deinterlacer Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.

+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>

+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>

+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>

+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>

+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"

+#include "imx-ic.h"
+
+/*
+ * This subdev implements two different video pipelines:
+ *
+ * CSI -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends a single interlaced field F(n-1)
+ * directly to the VDIC (and optionally the following field F(n)
+ * can be sent to memory via IDMAC channel 13). So only two fields
+ * can be processed by the VDIC. This pipeline only works in VDIC's
+ * high motion mode, which only requires a single field for processing.
+ * The other motion modes (low and medium) require three fields, so this
+ * pipeline does not work in those modes. Also, it is not clear how this
+ * pipeline can deal with the various field orders (sequential BT/TB,
+ * interlaced BT/TB).
+ *
+ * CSI -> CH[0-3] -> MEM -> CH8,9,10 -> VDIC -> IC -> CH21 -> MEM
+ *
+ * In this pipeline, the CSI sends raw and full frames to memory buffers
+ * via the SMFC channels 0-3. Fields from these frames are then
+ * transferred to the VDIC via IDMAC channels 8,9,10. The VDIC requires
+ * three fields: previous field F(n-1), current field F(n), and next
+ * field F(n+1), so we need three raw frames in memory: two completed frames
+ * to send F(n-1), F(n), F(n+1) to the VDIC, and a third frame for active
+ * CSI capture while the completed fields are sent through the VDIC->IC for
+ * processing.
+ *
+ * The "direct" CSI->VDIC pipeline requires less memory bus bandwidth
+ * (just 1 channel vs. 5 channels for indirect pipeline), but it can
+ * only be used in high motion mode, and it only processes a single
+ * field (so half the original image resolution is lost).
+ */
+
+struct prpvf_priv;
+
+struct prpvf_pipeline_ops {
+ int (*setup)(struct prpvf_priv *priv);
+ void (*start)(struct prpvf_priv *priv);
+ void (*stop)(struct prpvf_priv *priv);
+ void (*disable)(struct prpvf_priv *priv);
+};
+
+#define PRPVF_NUM_PADS 2
+
+#define MAX_W_IC 1024
+#define MAX_H_IC 1024
+#define MAX_W_VDIC 968
+#define MAX_H_VDIC 2048
+
+struct prpvf_priv {
+ struct imx_media_dev *md;
+ struct imx_ic_priv *ic_priv;
+
+ /* IPU units we require */
+ struct ipu_soc *ipu;
+ struct ipu_ic *ic_vf;
+ struct ipu_vdi *vdi;
+
+ struct media_pad pad[PRPVF_NUM_PADS];

+ int input_pad;
+ int output_pad;
+

+ /* the dma buffer ring to send to sink */
+ struct imx_media_dma_buf_ring *out_ring;
+

+ struct v4l2_subdev *sink_sd;
+

+ /* the attached CSI at stream on */
+ struct v4l2_subdev *csi_sd;

+
+ /* the attached sensor at stream on */

+ struct imx_media_subdev *sensor;
+

+ struct timer_list eof_timeout_timer;
+

+ int nfb4eof_irq; /* CSI or PRPVF channel NFB4EOF IRQ */
+ int out_eof_irq; /* PRPVF channel EOF IRQ */
+ spinlock_t irqlock;

+
+ bool stream_on; /* streaming is on */
+ bool last_eof; /* waiting for last EOF at stream off */
+ struct completion last_eof_comp;

+ goto out;
+ }
+

+ priv->vdi = ipu_vdi_get(priv->ipu);
+ if (IS_ERR(priv->vdi)) {
+ v4l2_err(&ic_priv->sd, "failed to get VDIC\n");
+ ret = PTR_ERR(priv->vdi);

+ goto out;
+ }
+

+ }
+ }
+
+ return 0;
+

+out_err_chan:
+ v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n", err_chan);
+out:
+ prpvf_put_ipu_resources(priv);

+ return ret;
+}
+

+/* prpvf_out_ch EOF interrupt (progressive frame ready) */

+static irqreturn_t prpvf_out_eof_interrupt(int irq, void *dev_id)
+{
+ struct prpvf_priv *priv = dev_id;
+ struct imx_media_dma_buf *done;

+
+ spin_lock(&priv->irqlock);
+
+ if (priv->last_eof) {
+ complete(&priv->last_eof_comp);
+ priv->last_eof = false;

+ goto unlock;
+ }
+

+ if (priv->csi_direct) {

+ /* inform CSI of this EOF so it can monitor frame intervals */

+ /* FIXME: frames are coming in twice as fast in direct path! */

+
+ /* bump the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+

+ if (priv->csi_direct) {
+ prepare_prpvf_out_buffer(priv);

+ /* toggle IPU double-buffer index */
+ priv->ipu_buf_num ^= 1;
+ }

+
+unlock:
+ spin_unlock(&priv->irqlock);
+ return IRQ_HANDLED;
+}
+

+static long prpvf_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);
+ struct imx_media_dma_buf_ring **ring;
+ struct imx_media_dma_buf *buf;

+ unsigned long flags;
+

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+static irqreturn_t nfb4eof_interrupt(int irq, void *dev_id)
+{
+ struct prpvf_priv *priv = dev_id;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_NFB4EOF,
+ };
+

+ v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+ v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+ return IRQ_HANDLED;
+}
+
+/*

+ * EOF timeout timer function.
+ */

+static void prpvf_eof_timeout(unsigned long data)
+{
+ struct prpvf_priv *priv = (struct prpvf_priv *)data;
+ struct imx_ic_priv *ic_priv = priv->ic_priv;

+ static const struct v4l2_event ev = {
+ .type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+ };
+

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ unsigned int burst_size;
+ struct ipu_image image;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];
+

+ burst_size = (outfmt->width & 0xf) ? 8 : 16;

+
+ priv->ipu_buf_num = 0;
+

+ buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+ imx_media_dma_buf_set_active(buf0);
+ buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);

+ priv->next_out_buf = buf1;
+
+ /* init the prpvf out channel */
+ setup_vdi_channel(priv, priv->prpvf_out_ch,
+ buf0->phys, buf1->phys, true);

+
+ return 0;
+}
+

+static void prpvf_start_direct(struct prpvf_priv *priv)

+{
+ /* set buffers ready */

+ ipu_idmac_select_buffer(priv->prpvf_out_ch, 0);
+ ipu_idmac_select_buffer(priv->prpvf_out_ch, 1);

+
+ /* enable the channels */

+ infmt = &priv->format_mbus[priv->input_pad];

+ incc = priv->cc[priv->input_pad];
+

+ in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+ /* 1/2 full image size */
+ priv->field_size = in_size / 2;
+ priv->in_stride = incc->planar ?
+ infmt->width : (infmt->width * incc->bpp) >> 3;
+
+ priv->ipu_buf_num = 0;
+
+ if (priv->in_ring) {
+ v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",

+ __func__);
+ imx_media_free_dma_buf_ring(priv->in_ring);
+ }

+ priv->in_ring = imx_media_alloc_dma_buf_ring(
+ priv->md, &priv->src_sd->entity,

+ &ic_priv->sd.entity,
+ in_size, IMX_MEDIA_MIN_RING_BUFS_PRPVF, true);
+ if (IS_ERR(priv->in_ring)) {
+ v4l2_err(&ic_priv->sd, "failed to alloc dma-buf ring\n");

+ ret = PTR_ERR(priv->in_ring);
+ priv->in_ring = NULL;

+ return ret;
+ }
+

+
+ return 0;
+}
+

+static void prpvf_start_indirect(struct prpvf_priv *priv)

+{
+ /* enable the channels */

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *outcc, *incc;
+ int ret;
+
+ if (!priv->sensor) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ return -EINVAL;
+ }
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ incc = priv->cc[priv->input_pad];

+ outcc = priv->cc[priv->output_pad];
+

+ priv->ops = priv->csi_direct ? &direct_ops : &indirect_ops;
+
+ ret = prpvf_get_ipu_resources(priv);

+ if (ret)
+ return ret;
+

+ /* set IC to receive from VDIC */
+ ipu_set_ic_src_mux(priv->ipu, 0, true);

+
+ /* ask the sink for the buffer ring */
+ ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+ IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+ &priv->out_ring);

+ if (ret)
+ goto out_put_ipu;

+
+ /* init EOF completion waitq */
+ init_completion(&priv->last_eof_comp);
+ priv->last_eof = false;
+

+ priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,

+ priv->prpvf_out_ch,
+ IPU_IRQ_NFB4EOF);
+ ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+ nfb4eof_interrupt, 0,
+ "imx-ic-prpvf-nfb4eof", priv);
+ if (ret) {
+ v4l2_err(&ic_priv->sd,

+ "Error registering NFB4EOF irq: %d\n", ret);

+ goto out_free_eof_irq;
+ }
+
+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_std, &priv->std);
+ if (ret)
+ goto out_free_nfb4eof_irq;
+
+ /* init the VDIC */
+ ipu_vdi_setup(priv->vdi, infmt->code,
+ infmt->width, infmt->height);
+ ipu_vdi_set_field_order(priv->vdi, priv->std, infmt->field);
+ ipu_vdi_set_motion(priv->vdi, priv->motion);
+
+ ret = ipu_ic_task_init(priv->ic_vf,
+ infmt->width, infmt->height,
+ outfmt->width, outfmt->height,
+ incc->cs, outcc->cs);
+ if (ret) {
+ v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+ goto out_free_nfb4eof_irq;
+ }
+
+ ret = priv->ops->setup(priv);
+ if (ret)
+ goto out_free_nfb4eof_irq;
+
+ ipu_vdi_enable(priv->vdi);
+ ipu_ic_enable(priv->ic_vf);
+
+ priv->ops->start(priv);
+
+ /* enable the IC VF task */
+ ipu_ic_task_enable(priv->ic_vf);

+
+ /* start the EOF timeout timer */
+ mod_timer(&priv->eof_timeout_timer,
+ jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));

+
+ return 0;
+

+ return ret;
+}
+

+static void prpvf_stop(struct prpvf_priv *priv)
+{
+ struct imx_ic_priv *ic_priv = priv->ic_priv;
+ unsigned long flags;
+ int ret;
+

+ /* mark next EOF interrupt as the last before stream off */
+ spin_lock_irqsave(&priv->irqlock, flags);
+ priv->last_eof = true;
+ spin_unlock_irqrestore(&priv->irqlock, flags);

+
+ /*

+ * and then wait for interrupt handler to mark completion.
+ */
+ ret = wait_for_completion_timeout(
+ &priv->last_eof_comp, msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+ if (ret == 0)

+ return -EINVAL;
+ }
+

+ return 0;
+}
+

+ int i, ret;
+

+ return ret;
+}
+

+static int prpvf_s_stream(struct v4l2_subdev *sd, int enable)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);
+ int ret = 0;
+
+ if (!priv->src_sd || !priv->sink_sd)

+ return -EPIPE;
+
+ v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+

+ if (enable && !priv->stream_on)

+ ret = prpvf_start(priv);
+ else if (!enable && priv->stream_on)
+ prpvf_stop(priv);
+
+ if (!ret)
+ priv->stream_on = enable;

+ return ret;
+}
+

+static int prpvf_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_pad_config *cfg,

+ struct v4l2_subdev_mbus_code_enum *code)
+{

+
+ return imx_media_enum_format(&code->code, code->index,

+ allow_rgb, allow_planar);
+}
+
+static int prpvf_get_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);
+
+ if (sdformat->pad >= PRPVF_NUM_PADS)
+ return -EINVAL;
+

+ sdformat->format = priv->format_mbus[sdformat->pad];

+
+ return 0;
+}
+

+static int prpvf_set_fmt(struct v4l2_subdev *sd,

+ struct v4l2_subdev_pad_config *cfg,
+ struct v4l2_subdev_format *sdformat)
+{

+ struct prpvf_priv *priv = sd_to_priv(sd);

+ struct v4l2_mbus_framefmt *infmt, *outfmt;

+ const struct imx_media_pixfmt *cc;

+ bool allow_planar, allow_rgb;
+ u32 code;
+
+ if (sdformat->pad >= PRPVF_NUM_PADS)
+ return -EINVAL;
+
+ if (priv->stream_on)
+ return -EBUSY;
+

+ infmt = &priv->format_mbus[priv->input_pad];
+ outfmt = &priv->format_mbus[priv->output_pad];

+ allow_planar = (sdformat->pad == priv->output_pad);
+ allow_rgb = allow_planar;

+
+ cc = imx_media_find_format(0, sdformat->format.code,

+ allow_rgb, allow_planar);
+ if (!cc) {
+ imx_media_enum_format(&code, 0, false, false);
+ cc = imx_media_find_format(0, code, false, false);

+ sdformat->format.code = cc->codes[0];
+ }

+ if (sdformat->pad == priv->output_pad) {

+
+ if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+ cfg->try_fmt = sdformat->format;
+ } else {

+ priv->format_mbus[sdformat->pad] = sdformat->format;
+ priv->cc[sdformat->pad] = cc;

+ }
+
+ return 0;
+}
+

+static int prpvf_link_setup(struct media_entity *entity,

+ const struct media_pad *local,
+ const struct media_pad *remote, u32 flags)
+{
+ struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);

+ local->entity->name);
+
+ remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+ if (local->flags & MEDIA_PAD_FL_SOURCE) {
+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->sink_sd)
+ return -EBUSY;
+ priv->sink_sd = remote_sd;
+ } else {
+ priv->sink_sd = NULL;
+ }
+
+ return 0;
+ }
+
+ /* this is sink pad */

+ if (flags & MEDIA_LNK_FL_ENABLED) {

+ if (priv->src_sd)
+ return -EBUSY;

+ priv->src_sd = remote_sd;
+

+ priv->csi_direct = ((priv->src_sd->grp_id &
+ IMX_MEDIA_GRP_ID_CSI) != 0);
+
+ ret = prpvf_init_controls(priv);

+ if (ret)
+ return ret;

+ } else {
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+ priv->src_sd = NULL;

+ }
+
+ return 0;
+}
+

+static int prpvf_link_validate(struct v4l2_subdev *sd,

+ struct media_link *link,
+ struct v4l2_subdev_format *source_fmt,
+ struct v4l2_subdev_format *sink_fmt)
+{

+ if (ret)
+ return ret;
+

+ priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+ if (IS_ERR(priv->sensor)) {
+ v4l2_err(&ic_priv->sd, "no sensor attached\n");

+ ret = PTR_ERR(priv->sensor);
+ priv->sensor = NULL;

+ return ret;
+ }
+

+ priv->csi_sd = csi->sd;

+ return 0;
+ }
+

+ priv->csi_sd = priv->src_sd;
+
+ if (priv->motion != HIGH_MOTION) {
+ v4l2_err(&ic_priv->sd,
+ "direct CSI pipeline requires HIGH_MOTION\n");

+ return -EINVAL;
+ }
+

+ ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,

+ &sensor_mbus_cfg);

+ if (ret)
+ return ret;
+

+ if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {

+ int vc_num = 0;

+ /* see NOTE in imx-csi.c */
+#if 0

+ vc_num = imx_media_find_mipi_csi2_channel(
+ priv->md, &ic_priv->sd.entity);
+ if (vc_num < 0)
+ return vc_num;
+#endif
+ /* only virtual channel 0 can be sent to IC */
+ if (vc_num != 0)
+ return -EINVAL;
+ } else {
+ /* only 8-bit pixels can be sent to IC for parallel busses */
+ if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)

+ return -EINVAL;
+ }
+

+ return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */

+static int prpvf_registered(struct v4l2_subdev *sd)
+{
+ struct prpvf_priv *priv = sd_to_priv(sd);

+ struct imx_media_subdev *imxsd;
+ struct imx_media_pad *pad;
+ int i, ret;
+
+ /* get media device */

+ priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+ imxsd = imx_media_find_subdev_by_sd(priv->md, sd);

+ if (IS_ERR(imxsd))
+ return PTR_ERR(imxsd);
+

+ if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+ return -EINVAL;
+
+ for (i = 0; i < PRPVF_NUM_PADS; i++) {

+ pad = &imxsd->pad[i];

+ priv->pad[i] = pad->pad;
+ if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+ priv->input_pad = i;
+ else
+ priv->output_pad = i;

+
+ /* set a default mbus format */

+ ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],

+ 640, 480, 0, V4L2_FIELD_NONE,

+ &priv->cc[i]);

+ if (ret)
+ return ret;
+ }
+

+ .link_validate = v4l2_subdev_link_validate,
+};
+

+ init_timer(&priv->eof_timeout_timer);
+ priv->eof_timeout_timer.data = (unsigned long)priv;

+ priv->eof_timeout_timer.function = prpvf_eof_timeout;

+
+ return 0;
+}
+

+static void prpvf_remove(struct imx_ic_priv *ic_priv)
+{
+ struct prpvf_priv *priv = ic_priv->task_priv;

+
+ v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}

+
+struct imx_ic_ops imx_ic_prpvf_ops = {
+ .subdev_ops = &prpvf_subdev_ops,
+ .internal_ops = &prpvf_internal_ops,
+ .entity_ops = &prpvf_entity_ops,
+ .init = prpvf_init,
+ .remove = prpvf_remove,
+};
diff --git a/drivers/staging/media/imx/imx-ic.h b/drivers/staging/media/imx/imx-ic.h
new file mode 100644
index 0000000..1ab222b
--- /dev/null
+++ b/drivers/staging/media/imx/imx-ic.h
@@ -0,0 +1,38 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.

+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+

+#include <media/v4l2-subdev.h>
+
+struct imx_ic_priv {
+ struct device *dev;
+ struct v4l2_subdev sd;
+ int ipu_id;
+ int task_id;
+ void *task_priv;
+};
+
+struct imx_ic_ops {
+ struct v4l2_subdev_ops *subdev_ops;
+ struct v4l2_subdev_internal_ops *internal_ops;
+ struct media_entity_operations *entity_ops;
+
+ int (*init)(struct imx_ic_priv *ic_priv);
+ void (*remove)(struct imx_ic_priv *ic_priv);
+};
+
+extern struct imx_ic_ops imx_ic_prpenc_ops;
+extern struct imx_ic_ops imx_ic_prpvf_ops;
+extern struct imx_ic_ops imx_ic_pp_ops;
+
+#endif
+
--
2.7.4

Rob Herring

unread,

Jan 10, 2017, 12:40:04 AM1/10/17

This should have an SoC/chip specific compatible string additionally.
Just need a note to that effect here, but i.MX will need a compat
string.

Tim Harvey

unread,

Jan 11, 2017, 6:20:04 PM1/11/17

<snip>

Hi Steve,

I took a stab at testing this today on a gw51xx which has an adv7180
hooked up as follows:
- i2c3@0x20
- 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
(CSI0_IPU1)
- PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
- IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
- all three analog inputs available to off-board connector

My patch to the imx6qdl-gw51xx dtsi is:
diff --git a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
index afec2c7..2583d72 100644
--- a/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-gw51xx.dtsi
@@ -165,6 +174,52 @@

pinctrl-names = "default";
pinctrl-0 = <&pinctrl_i2c3>;
status = "okay";

+
+ camera: adv7180@20 {
+ compatible = "adi,adv7180";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_adv7180>;
+ reg = <0x20>;
+ powerdown-gpio = <&gpio5 20 GPIO_ACTIVE_LOW>;
+ interrupt-parent = <&gpio5>;
+ interrupts = <23 GPIO_ACTIVE_LOW>;
+ inputs = <0x00 0x01 0x02>;

+ input-names = "ADV7180 Composite on Ain1",

+ "ADV7180 Composite on Ain2",

+ "ADV7180 Composite on Ain3";
+
+ port {

+ adv7180_to_ipu1_csi0_mux: endpoint {

+ remote-endpoint =
<&ipu1_csi0_mux_from_parallel_sensor>;
+ bus-width = <8>;

+ };
+ };
+ };
+};
+

+&ipu1_csi0_from_ipu1_csi0_mux {
+ bus-width = <8>;

+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+ remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+ bus-width = <8>;
+};
+
+&ipu1_csi0 {
+ pinctrl-names = "default";

+ pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+ /* enable frame interval monitor on this port */
+ fim {

+ status = "okay";
+ };

};

&pcie {
@@ -236,6 +291,13 @@

&iomuxc {
imx6qdl-gw51xx {
+ pinctrl_adv7180: adv7180grp {
+ fsl,pins = <
+ MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
0x0001b0b0 /* VIDDEC_IRQ# */
+ MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
0x4001b0b0 /* VIDDEC_EN */
+ >;
+ };
+
pinctrl_enet: enetgrp {
fsl,pins = <
MX6QDL_PAD_RGMII_RXC__RGMII_RXC 0x1b030
@@ -306,6 +368,22 @@
>;
};

+ pinctrl_ipu1_csi0: ipu1csi0grp { /* IPU1_CSI0: 8-bit input */

+ fsl,pins = <
+
MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18 0x1b0b0
+
MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19 0x1b0b0

+ MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC
0x1b0b0
+ MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC
0x1b0b0
+

MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0

+ >;
+ };
+
pinctrl_pcie: pciegrp {
fsl,pins = <

MX6QDL_PAD_GPIO_0__GPIO1_IO00 0x1b0b0

On an IMX6Q I'm getting the following when the adv7180 module loads:
[ 12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
[ 12.907767] imx-media: Registered subdev adv7180 2-0020
[ 12.907793] imx-media soc:media@0: Entity type for entity adv7180
2-0020 was not initialized!
[ 12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
ipu1_csi0_mux:1

Is the warning that adv7180 was not initialized expected and or an issue?

Now that your driver is hooking into the current media framework, I'm
not at all clear on how to link and configure the media entities.
Using 'media-ctl -p' to list the entities I see:
Media controller API version 0.1.0

Media device information
------------------------
driver imx-media
model imx-media
serial
bus info
hw revision 0x0
driver version 0.0.0

Device topology
- entity 1: ipu2_csi1_mux (3 pads, 1 link)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:unknown/0x0]
pad1: Sink
[fmt:unknown/0x0]
pad2: Source
[fmt:unknown/0x0]
-> "ipu2_csi1":0 []

- entity 5: ipu1_csi0_mux (3 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:unknown/0x0]
pad1: Sink
[fmt:unknown/0x0]
<- "adv7180 2-0020":0 []
pad2: Source
[fmt:unknown/0x0]
-> "ipu1_csi0":0 []

- entity 9: ipu1_ic_prpenc (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_csi0":1 []
<- "ipu1_csi1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif0":0 []
-> "camif1":0 []

- entity 12: ipu1_ic_prpvf (2 pads, 8 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev3
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_csi0":1 []
<- "ipu1_csi1":1 []
<- "ipu1_smfc0":1 []
<- "ipu1_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif0":0 []
-> "camif1":0 []
-> "ipu1_ic_pp0":0 []
-> "ipu1_ic_pp1":0 []

- entity 15: ipu1_ic_pp0 (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev4
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_ic_prpvf":1 []
<- "ipu1_smfc0":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif0":0 []
-> "camif1":0 []

- entity 18: ipu1_ic_pp1 (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev5
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_ic_prpvf":1 []
<- "ipu1_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif0":0 []
-> "camif1":0 []

- entity 21: ipu2_ic_prpenc (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev6
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_csi0":1 []
<- "ipu2_csi1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif2":0 []
-> "camif3":0 []

- entity 24: ipu2_ic_prpvf (2 pads, 8 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev7
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_csi0":1 []
<- "ipu2_csi1":1 []
<- "ipu2_smfc0":1 []
<- "ipu2_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif2":0 []
-> "camif3":0 []
-> "ipu2_ic_pp0":0 []
-> "ipu2_ic_pp1":0 []

- entity 27: ipu2_ic_pp0 (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev8
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_ic_prpvf":1 []
<- "ipu2_smfc0":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif2":0 []
-> "camif3":0 []

- entity 30: ipu2_ic_pp1 (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev9
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_ic_prpvf":1 []
<- "ipu2_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif2":0 []
-> "camif3":0 []

- entity 33: ipu1_csi0 (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev10
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_csi0_mux":2 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none
crop.bounds:(0,0)/640x480
crop:(0,0)/0x0]
-> "ipu1_ic_prpenc":0 []
-> "ipu1_ic_prpvf":0 []
-> "ipu1_smfc0":0 []

- entity 36: ipu1_csi1 (2 pads, 3 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev11
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
pad1: Source
[fmt:UYVY2X8/640x480 field:none
crop.bounds:(0,0)/640x480
crop:(0,0)/0x0]
-> "ipu1_ic_prpenc":0 []
-> "ipu1_ic_prpvf":0 []
-> "ipu1_smfc1":0 []

- entity 39: ipu2_csi0 (2 pads, 3 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev12
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
pad1: Source
[fmt:UYVY2X8/640x480 field:none
crop.bounds:(0,0)/640x480
crop:(0,0)/0x0]
-> "ipu2_ic_prpenc":0 []
-> "ipu2_ic_prpvf":0 []
-> "ipu2_smfc0":0 []

- entity 42: ipu2_csi1 (2 pads, 4 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev13
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_csi1_mux":2 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none
crop.bounds:(0,0)/640x480
crop:(0,0)/0x0]
-> "ipu2_ic_prpenc":0 []
-> "ipu2_ic_prpvf":0 []
-> "ipu2_smfc1":0 []

- entity 45: ipu1_smfc0 (2 pads, 5 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev14
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_csi0":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "ipu1_ic_prpvf":0 []
-> "ipu1_ic_pp0":0 []
-> "camif0":0 []
-> "camif1":0 []

- entity 48: ipu1_smfc1 (2 pads, 5 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev15
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_csi1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "ipu1_ic_prpvf":0 []
-> "ipu1_ic_pp1":0 []
-> "camif0":0 []
-> "camif1":0 []

- entity 51: ipu2_smfc0 (2 pads, 5 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev16
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_csi0":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "ipu2_ic_prpvf":0 []
-> "ipu2_ic_pp0":0 []
-> "camif2":0 []
-> "camif3":0 []

- entity 54: ipu2_smfc1 (2 pads, 5 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev17
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_csi1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "ipu2_ic_prpvf":0 []
-> "ipu2_ic_pp1":0 []
-> "camif2":0 []
-> "camif3":0 []

- entity 57: camif0 (2 pads, 7 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev18
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_ic_prpenc":1 []
<- "ipu1_ic_prpvf":1 []
<- "ipu1_ic_pp0":1 []
<- "ipu1_ic_pp1":1 []
<- "ipu1_smfc0":1 []
<- "ipu1_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif0 devnode":0 [ENABLED,IMMUTABLE]

- entity 58: camif0 devnode (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Sink
<- "camif0":1 [ENABLED,IMMUTABLE]

- entity 66: camif1 (2 pads, 7 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev19
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu1_ic_prpenc":1 []
<- "ipu1_ic_prpvf":1 []
<- "ipu1_ic_pp0":1 []
<- "ipu1_ic_pp1":1 []
<- "ipu1_smfc0":1 []
<- "ipu1_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif1 devnode":0 [ENABLED,IMMUTABLE]

- entity 67: camif1 devnode (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video1
pad0: Sink
<- "camif1":1 [ENABLED,IMMUTABLE]

- entity 75: camif2 (2 pads, 7 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev20
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_ic_prpenc":1 []
<- "ipu2_ic_prpvf":1 []
<- "ipu2_ic_pp0":1 []
<- "ipu2_ic_pp1":1 []
<- "ipu2_smfc0":1 []
<- "ipu2_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif2 devnode":0 [ENABLED,IMMUTABLE]

- entity 76: camif2 devnode (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video2
pad0: Sink
<- "camif2":1 [ENABLED,IMMUTABLE]

- entity 84: camif3 (2 pads, 7 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev21
pad0: Sink
[fmt:UYVY2X8/640x480 field:none]
<- "ipu2_ic_prpenc":1 []
<- "ipu2_ic_prpvf":1 []
<- "ipu2_ic_pp0":1 []
<- "ipu2_ic_pp1":1 []
<- "ipu2_smfc0":1 []
<- "ipu2_smfc1":1 []
pad1: Source
[fmt:UYVY2X8/640x480 field:none]
-> "camif3 devnode":0 [ENABLED,IMMUTABLE]

- entity 85: camif3 devnode (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video3
pad0: Sink
<- "camif3":1 [ENABLED,IMMUTABLE]

- entity 93: adv7180 2-0020 (1 pad, 1 link)
type V4L2 subdev subtype Unknown flags 20004
device node name /dev/v4l-subdev22
pad0: Source
[fmt:UYVY2X8/720x480 field:interlaced]
-> "ipu1_csi0_mux":1 []

How do I link the entities here to be able to capture frames with
v4l2-ctl or gstreamer?

Additionally I've found that on an IMX6S/IMX6DL we crash while
registering the media-ic subdev's:
[ 3.975473] imx-media: Registered subdev ipu1_csi1_mux
[ 3.980921] imx-media: Registered subdev ipu1_csi0_mux
[ 4.003205] imx-media: Registered subdev ipu1_ic_prpenc
[ 4.025373] imx-media: Registered subdev ipu1_ic_prpvf
[ 4.037944] ------------[ cut here ]------------
[ 4.042571] Kernel BUG at c06717dc [verbose debug info unavailable]
[ 4.048845] Internal error: Oops - BUG: 0 [#1] SMP ARM
[ 4.053990] Modules linked in:
[ 4.057076] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
4.9.0-rc6-00524-g84dad6e-dirty #446
[ 4.065260] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
...
[ 4.296250] [<c0671780>] (v4l2_subdev_init) from [<c06fb02c>]
(imx_ic_probe+0x94/0x1ac)
[ 4.304271] [<c06faf98>] (imx_ic_probe) from [<c05173d8>]
(platform_drv_probe+0x54/0xb8)
[ 4.312373] r9:c0d5e858 r8:00000000 r7:fffffdfb r6:c0e5dbf8
r5:da603810 r4:c16738d8
[ 4.320129] [<c0517384>] (platform_drv_probe) from [<c0515978>]
(driver_probe_device+0x20c/0x2c0)
[ 4.329010] r7:c0e5dbf8 r6:00000000 r5:da603810 r4:c16738d8
[ 4.334681] [<c051576c>] (driver_probe_device) from [<c0515af4>]
(__driver_attach+0xc8/0xcc)
[ 4.343129] r9:c0d5e858 r8:00000000 r7:00000000 r6:da603844
r5:c0e5dbf8 r4:da603810
[ 4.350889] [<c0515a2c>] (__driver_attach) from [<c0513adc>]
(bus_for_each_dev+0x74/0xa8)
[ 4.359078] r7:00000000 r6:c0515a2c r5:c0e5dbf8 r4:00000000
[ 4.364753] [<c0513a68>] (bus_for_each_dev) from [<c05151d4>]
(driver_attach+0x20/0x28)

I assume there is an iteration that needs a test on a missing pointer
only available on chips with both IPU's or PRP

Regards,

Tim

Steve Longerbeam

unread,

Jan 11, 2017, 10:30:05 PM1/11/17

Hi Tim,

On 01/11/2017 03:14 PM, Tim Harvey wrote:
>
> <snip>
>
> Hi Steve,
>
> I took a stab at testing this today on a gw51xx which has an adv7180
> hooked up as follows:
> - i2c3@0x20
> - 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads
> (CSI0_IPU1)
> - PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20
> - IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23
> - all three analog inputs available to off-board connector
>
> My patch to the imx6qdl-gw51xx dtsi is:

As long as you used the patch to imx6qdl-sabreauto.dtsti that adds
the adv7180 support as a guide, you should be ok here.

> <snip>

>
>
>
> On an IMX6Q I'm getting the following when the adv7180 module loads:
> [ 12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c)
> [ 12.907767] imx-media: Registered subdev adv7180 2-0020
> [ 12.907793] imx-media soc:media@0: Entity type for entity adv7180
> 2-0020 was not initialized!
> [ 12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 ->
> ipu1_csi0_mux:1
>
> Is the warning that adv7180 was not initialized expected and or an issue?

Yeah it's still a bug in the adv7180 driver, needs fixing.

>
> Now that your driver is hooking into the current media framework, I'm
> not at all clear on how to link and configure the media entities.

It's all documented at Documentation/media/v4l-drivers/imx.rst.
Follow the SabreAuto pipeline setup example.

> <snip>

Yep, I only have quad boards here so I haven't gotten around to
testing on S/DL.

But it looks like I forgot to clear out the csi subdev pointer array before
passing it to imx_media_of_parse(). I think that might explain the OOPS
above. Try this patch:

diff --git a/drivers/staging/media/imx/imx-media-dev.c
b/drivers/staging/media/imx/imx-media-dev.c
index 357654d..0cf2d61 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct device_node *node = dev->of_node;
- struct imx_media_subdev *csi[4];
+ struct imx_media_subdev *csi[4] = {0};
struct imx_media_dev *imxmd;
int ret;

Steve

Tim Harvey

unread,

Jan 12, 2017, 3:40:06 PM1/12/17

On Fri, Jan 6, 2017 at 6:11 PM, Steve Longerbeam <slong...@gmail.com> wrote:

Steve,

These are not used anywhere.

Tim

Steve Longerbeam

unread,

Jan 12, 2017, 6:50:05 PM1/12/17

Yes, maybe I should just remove this patch for now. I'm only keeping it
because eventually it will be needed to support i.MX6 input capture.

Steve

Philipp Zabel

unread,

Jan 13, 2017, 7:00:05 AM1/13/17

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
> clocks.
>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---
> arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
> index 53e6e63..42926e9 100644
> --- a/arch/arm/boot/dts/imx6qdl.dtsi
> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
> @@ -1125,7 +1125,14 @@
> };
>
> mipi_csi: mipi@021dc000 {
> + compatible = "fsl,imx6-mipi-csi2";
> reg = <0x021dc000 0x4000>;
> + interrupts = <0 100 0x04>, <0 101 0x04>;
> + clocks = <&clks IMX6QDL_CLK_HSI_TX>,
> + <&clks IMX6QDL_CLK_VIDEO_27M>,
> + <&clks IMX6QDL_CLK_EIM_SEL>;

I think the latter should be EIM_PODF

> + clock-names = "dphy", "cfg", "pix";

and I'm not sure dphy is the right name for this one. Is that the pll
ref input?

> + status = "disabled";
> };
>
> mipi_dsi: mipi@021e0000 {

regards
Philipp

Philipp Zabel

unread,

Jan 13, 2017, 7:10:07 AM1/13/17

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

This is superfluous, you can use clk_get_rate on mipi_xclk.

> + reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
> + pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
> +
> + port {
> + #address-cells = <1>;
> + #size-cells = <0>;
> +
> + ov5640_to_mipi_csi: endpoint@1 {
> + reg = <1>;
> + remote-endpoint = <&mipi_csi_from_mipi_sensor>;
> + data-lanes = <0 1>;
> + clock-lanes = <2>;
> + };
> + };
> + };
> +
> + ov5642: camera@42 {
> + compatible = "ovti,ov5642";
> + pinctrl-names = "default";
> + pinctrl-0 = <&pinctrl_ov5642>;
> + clocks = <&clks IMX6QDL_CLK_CKO2>;
> + clock-names = "xclk";
> + reg = <0x42>;
> + xclk = <24000000>;

Same here, use assigned-clock-rates on IMX6QDL_CLK_CKO2 if necessary.

regards
Philipp

Philipp Zabel

unread,

Jan 13, 2017, 10:30:08 AM1/13/17

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

> Add the core media driver for i.MX SOC.

>
> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

> Documentation/media/v4l-drivers/imx.rst | 443 ++++++++++

> drivers/staging/media/Kconfig | 2 +
> drivers/staging/media/Makefile | 1 +

> drivers/staging/media/imx/Kconfig | 8 +
> drivers/staging/media/imx/Makefile | 6 +
> drivers/staging/media/imx/TODO | 22 +
> drivers/staging/media/imx/imx-media-common.c | 981 ++++++++++++++++++++++
> drivers/staging/media/imx/imx-media-dev.c | 486 +++++++++++
> drivers/staging/media/imx/imx-media-fim.c | 471 +++++++++++
> drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
> drivers/staging/media/imx/imx-media-of.c | 289 +++++++
> drivers/staging/media/imx/imx-media.h | 310 +++++++
> include/media/imx.h | 15 +
> include/uapi/linux/v4l2-controls.h | 4 +
> 14 files changed, 3495 insertions(+)
> create mode 100644 Documentation/media/v4l-drivers/imx.rst

> create mode 100644 drivers/staging/media/imx/Kconfig
> create mode 100644 drivers/staging/media/imx/Makefile
> create mode 100644 drivers/staging/media/imx/TODO

> create mode 100644 include/media/imx.h
>
> diff --git a/Documentation/media/v4l-drivers/imx.rst b/Documentation/media/v4l-drivers/imx.rst
> new file mode 100644
> index 0000000..87b37b5
> --- /dev/null
> +++ b/Documentation/media/v4l-drivers/imx.rst
> @@ -0,0 +1,443 @@
> +i.MX Video Capture Driver
> +=========================
> +
> +Introduction
> +------------
> +
> +The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
> +handles the flow of image frames to and from capture devices and
> +display devices.
> +
> +For image capture, the IPU contains the following internal subunits:
> +
> +- Image DMA Controller (IDMAC)
> +- Camera Serial Interface (CSI)
> +- Image Converter (IC)
> +- Sensor Multi-FIFO Controller (SMFC)
> +- Image Rotator (IRT)
> +- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

> +
> +The IDMAC is the DMA controller for transfer of image frames to and from
> +memory. Various dedicated DMA channels exist for both video capture and
> +display paths.
> +
> +The CSI is the frontend capture unit that interfaces directly with
> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
> +
> +The IC handles color-space conversion, resizing, and rotation
> +operations.

And horizontal flipping.

> There are three independent "tasks" within the IC that can
> +carry out conversions concurrently: pre-processing encoding,
> +pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

This paragraph could mention that a single hardware unit is used
transparently time multiplexed by the three tasks at different
granularity for the downsizing, main processing, and rotation sections.
The downscale unit switches between tasks at 8-pixel burst granularity,
the main processing unit at line granularity. The rotation units switch
only at frame granularity.

> +The SMFC is composed of four independent channels that each can transfer
> +captured frames from sensors directly to memory concurrently.
> +
> +The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block
transfers, block reordering, and vertical flipping.

> +The VDIC handles the conversion of interlaced video to progressive, with
> +support for different motion compensation modes (low, medium, and high
> +motion). The deinterlaced output frames from the VDIC can be sent to the
> +IC pre-process preview task for further conversions.
> +
> +In addition to the IPU internal subunits, there are also two units
> +outside the IPU that are also involved in video capture on i.MX:
> +
> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
> + interface. This is a Synopsys DesignWare core.
> +- A video multiplexer for selecting among multiple sensor inputs to
> + send to a CSI.

Two of them, actually.

> +For more info, refer to the latest versions of the i.MX5/6 reference
> +manuals listed under References.
> +
> +
> +Features
> +--------
> +
> +Some of the features of this driver include:
> +
> +- Many different pipelines can be configured via media controller API,
> + that correspond to the hardware video capture pipelines supported in
> + the i.MX.
> +
> +- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
> +
> +- Up to four concurrent sensor acquisitions, by configuring each
> + sensor's pipeline using independent entities. This is currently
> + demonstrated with the SabreSD and SabreLite reference boards with
> + independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
> +
> +- Scaling, color-space conversion, and image rotation via IC task
> + subdevs.
> +
> +- Many pixel formats supported (RGB, packed and planar YUV, partial
> + planar YUV).
> +
> +- The IC pre-process preview subdev supports motion compensated
> + de-interlacing using the VDIC, with three motion compensation modes:
> + low, medium, and high motion. The mode is specified with a custom
> + control. Pipelines are defined that allow sending frames to the
> + preview subdev directly from the CSI or from the SMFC.
> +
> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
> + problems with the ADV718x video decoders. See below for a description
> + of the FIM.

Could this also be used to calculate more precise capture timestamps?

> +Capture Pipelines
> +-----------------
> +
> +The following describe the various use-cases supported by the pipelines.
> +
> +The links shown do not include the frontend sensor, video mux, or mipi
> +csi-2 receiver links. This depends on the type of sensor interface
> +(parallel or mipi csi-2). So in all cases, these pipelines begin with:
> +
> +sensor -> ipu_csi_mux -> ipu_csi -> ...
> +
> +for parallel sensors, or:
> +
> +sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
> +
> +for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
> +to the video mux (ipu_csi_mux) before sending to the CSI, depending
> +on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
> +parenthesis.
> +
> +Unprocessed Video Capture:
> +--------------------------
> +
> +Send frames directly from sensor to camera interface, with no
> +conversions:
> +
> +-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe
idmac if you want to mirror hardware names?

> +Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

> +For example, its sink pad can take UYVY2X8, but its source pad can
> +output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
in the CSI chapter, for 8-bit per component data, the internal format
between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
(or "bayer/generic data"). In either case, the internal format does not
change along the way.

> +IC Direct Conversions:
> +----------------------
> +
> +This pipeline uses the preprocess encode entity to route frames directly
> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
> +1024x1024 resolution, CSC, and image rotation:
> +
> +-> ipu_ic_prpenc -> camif
> +
> +This can be a useful capture pipeline for heavily loaded memory bus
> +traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and
rotation still have to go through memory once.

> +Post-Processing Conversions:
> +----------------------------
> +
> +This pipeline routes frames from the SMFC to the post-processing
> +entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into
the post-processing entity.

> In addition to CSC and rotation, this entity supports tiling
> +which allows scaled output beyond the 1024x1024 limitation of the IC
> +(up to 4096x4096 scaling output is supported):
> +
> +-> ipu_smfc -> ipu_ic_pp -> camif
> +
> +Motion Compensated De-interlace:
> +--------------------------------
> +
> +This pipeline routes frames from the SMFC to the preprocess preview
> +entity to support motion-compensated de-interlacing using the VDIC,
> +scaling up to 1024x1024, and CSC:
> +
> +-> ipu_smfc -> ipu_ic_prpvf -> camif

Same as above.

> +This pipeline also carries out the same conversions as above, but routes
> +frames directly from the CSI to the IC preprocess preview entity for
> +minimal memory bandwidth usage (note: this pipeline only works in
> +"high motion" mode):
> +
> +-> ipu_ic_prpvf -> camif
> +
> +This pipeline takes the motion-compensated de-interlaced frames and
> +sends them to the post-processor, to support motion-compensated
> +de-interlacing, scaling up to 4096x4096, CSC, and rotation:
> +
> +-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
> +
> +
> +Usage Notes
> +-----------
[...]
> +SabreLite with OV5642 and OV5640
> +--------------------------------
> +
> +This platform requires the OmniVision OV5642 module with a parallel
> +camera interface, and the OV5640 module with a MIPI CSI-2
> +interface. Both modules are available from Boundary Devices:
> +
> +https://boundarydevices.com/products/nit6x_5mp
> +https://boundarydevices.com/product/nit6x_5mp_mipi
> +
> +Note that if only one camera module is available, the other sensor
> +node can be disabled in the device tree.
> +
> +The OV5642 module is connected to the parallel bus input on the i.MX
> +internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
> +
> +The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
> +receiver, and the four virtual channel outputs from the receiver are
> +routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
> +vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
> +also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
> +OV5640 must not share the same i2c slave address.
> +
> +The following basic example configures unprocessed video capture
> +pipelines for both sensors. The OV5642 is routed to camif0
> +(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
> +virtual channel 1) is routed to camif1 (usually /dev/video1). Both
> +sensors are configured to output 640x480, UYVY (not shown: all pad
> +field types should be set to "NONE"):
> +
> +.. code-block:: none
> +
> + # Setup links for OV5642
> + media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
> + media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> + media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> + media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
> + media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> + # Setup links for OV5640
> + media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
> + media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
> + media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
> + media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
> + media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
> + # Configure pads for OV5642 pipeline
> + media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
> + media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
> + media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
> + media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
> + media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
> + media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
> + media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

> + media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> + media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> + # Configure pads for OV5640 pipeline
> + media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> + media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> + media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> + media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> + media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
[...]
> + media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the
csi1:1 and camif1:0 pads would be a 32-bit YUV format.
(MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format as that will allow
to chose the regular vs. companded packings in the future for formats
with more than 8 bits per component.

> + media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
>+
> +Streaming can then begin independently on device nodes /dev/video0
> +and /dev/video1.
> +
> +SabreAuto with ADV7180 decoder
> +------------------------------
> +
> +On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
> +parallel bus input on the internal video mux to IPU1 CSI0.
> +
> +The following example configures a pipeline to capture from the ADV7180
> +video decoder, assuming NTSC 720x480 input signals, with Motion
> +Compensated de-interlacing (not shown: all pad field types should be set
> +as indicated). $outputfmt can be any format supported by the
> +ipu1_ic_prpvf entity at its output pad:
> +
> +.. code-block:: none
> +
> + # Setup links
> + media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
> + media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
> + media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
> + media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
> + media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
> + media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
> + # Configure pads
> + # pad field types for below pads must be an interlaced type
> + # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI
can only capture NTSC/PAL fields in a fixed order.

> + media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
> + media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

From here the interlaced field type should be sequential in the correct
order depending on NTSC/PAL.

> + media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
> + media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
> + media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
> + media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
> + media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
> + media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
> + # pad field types for below pads must be "NONE"
> + media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
> + media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
> + media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
> +
> +Streaming can then begin on /dev/video0.
> +
> +This platform accepts Composite Video analog inputs to the ADV7180 on
> +Ain1 (connector J42) and Ain3 (connector J43).
> +
> +To switch to Ain1:
> +
> +.. code-block:: none
> +
> + # v4l2-ctl -i0
> +
> +To switch to Ain3:
> +
> +.. code-block:: none
> +
> + # v4l2-ctl -i1
> +
> +
> +Frame Interval Monitor
> +----------------------
> +
> +The adv718x decoders can occasionally send corrupt fields during
> +NTSC/PAL signal re-sync (too little or too many video lines). When
> +this happens, the IPU triggers a mechanism to re-establish vertical
> +sync by adding 1 dummy line every frame, which causes a rolling effect
> +from image to image, and can last a long time before a stable image is
> +recovered. Or sometimes the mechanism doesn't work at all, causing a
> +permanent split image (one frame contains lines from two consecutive
> +captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the
pad routing?

[...]
> +/*
> + * DMA buffer ring handling
> + */
> +struct imx_media_dma_buf_ring {
> + struct imx_media_dev *imxmd;
> +
> + /* the ring */
> + struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
> + /* the scratch buffer for underruns */
> + struct imx_media_dma_buf scratch;
> +
> + /* buffer generator */
> + struct media_entity *src;
> + /* buffer receiver */
> + struct media_entity *sink;
> +
> + spinlock_t lock;
> +
> + int num_bufs;
> + unsigned long last_seq;
> +};

I don't think this belongs in the capture driver at all.
Memory-to-memory transfers should be handled at the videobuf2 level.

[...]
> +static struct imx_media_dma_buf *
> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)

> +{
> + struct imx_media_dma_buf *buf;
> +

> + if (index >= ring->num_bufs)
> + return ERR_PTR(-EINVAL);
> +
> + buf = &ring->buf[index];
> + if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
> + return ERR_PTR(-EINVAL);
> +
> + buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
> + buf->seq = ring->last_seq++;
> +
> + return buf;
> +}

Is this a whole software buffer queue implementation? I thought the
whole point of putting the custom mem2mem framework into the capture
driver was to use the hardware FSU channel linking?

> +int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)

> +{
> + struct imx_media_dma_buf *buf;

> + unsigned long flags;
> +

> + spin_lock_irqsave(&ring->lock, flags);
> + buf = __dma_buf_queue(ring, index);
> + spin_unlock_irqrestore(&ring->lock, flags);
> +
> + if (IS_ERR(buf))
> + return PTR_ERR(buf);
> +
> + dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued\n",
> + index, ring->src->name, ring->sink->name);

> +
> + return 0;
> +}

> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue);
> +
> +int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
> + struct vb2_buffer *vb)

> +{
> + struct imx_media_dma_buf *buf;

> + unsigned long flags;
> + dma_addr_t phys;
> + void *virt;
> +
> + if (vb->index >= ring->num_bufs)
> + return -EINVAL;
> +
> + virt = vb2_plane_vaddr(vb, 0);
> + phys = vb2_dma_contig_plane_dma_addr(vb, 0);
> +
> + spin_lock_irqsave(&ring->lock, flags);
> + buf = __dma_buf_queue(ring, vb->index);
> + if (IS_ERR(buf))
> + goto err_unlock;
> +
> + buf->virt = virt;
> + buf->phys = phys;
> + buf->vb = vb;
> + spin_unlock_irqrestore(&ring->lock, flags);
> +
> + dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued from vb\n",
> + buf->index, ring->src->name, ring->sink->name);
> +
> + return 0;
> +err_unlock:
> + spin_unlock_irqrestore(&ring->lock, flags);
> + return PTR_ERR(buf);
> +}
> +EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue_from_vb);
> +
> +void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
> + enum imx_media_dma_buf_status status)
> +{
> + struct imx_media_dma_buf_ring *ring = buf->ring;

> + unsigned long flags;
> +

> + spin_lock_irqsave(&ring->lock, flags);
> + WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_ACTIVE);
> + buf->state = buf->status = status;
> + spin_unlock_irqrestore(&ring->lock, flags);
> +
> + if (buf == &ring->scratch)
> + dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] done\n",
> + ring->src->name, ring->sink->name);
> + else
> + dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] done\n",
> + buf->index, ring->src->name, ring->sink->name);
> +
> + /* if the sink is a subdev, inform it that new buffers are available */
> + if (is_media_entity_v4l2_subdev(ring->sink)) {

> + struct v4l2_subdev *sd =

> + media_entity_to_v4l2_subdev(ring->sink);
> + v4l2_subdev_call(sd, core, ioctl, IMX_MEDIA_NEW_DMA_BUF, NULL);

What is the purpose of this if the sink should be triggered by the FSU?

[...]
> +/*
> + * The subdevs have to be powered on/off, and streaming
> + * enabled/disabled, in a specific sequence.
> + */
> +static const u32 stream_on_seq[] = {
> + IMX_MEDIA_GRP_ID_IC_PP,
> + IMX_MEDIA_GRP_ID_IC_PRPVF,
> + IMX_MEDIA_GRP_ID_IC_PRPENC,
> + IMX_MEDIA_GRP_ID_SMFC,
> + IMX_MEDIA_GRP_ID_SENSOR,
> + IMX_MEDIA_GRP_ID_CSI2,
> + IMX_MEDIA_GRP_ID_VIDMUX,
> + IMX_MEDIA_GRP_ID_CSI,
> +};
> +
> +static const u32 stream_off_seq[] = {
> + IMX_MEDIA_GRP_ID_IC_PP,
> + IMX_MEDIA_GRP_ID_IC_PRPVF,
> + IMX_MEDIA_GRP_ID_IC_PRPENC,
> + IMX_MEDIA_GRP_ID_SMFC,
> + IMX_MEDIA_GRP_ID_CSI,
> + IMX_MEDIA_GRP_ID_VIDMUX,
> + IMX_MEDIA_GRP_ID_CSI2,
> + IMX_MEDIA_GRP_ID_SENSOR,
> +};
> +
> +#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
> +
> +static const u32 power_on_seq[] = {
> + IMX_MEDIA_GRP_ID_CSI2,
> + IMX_MEDIA_GRP_ID_SENSOR,
> + IMX_MEDIA_GRP_ID_VIDMUX,
> + IMX_MEDIA_GRP_ID_CSI,
> + IMX_MEDIA_GRP_ID_SMFC,
> + IMX_MEDIA_GRP_ID_IC_PRPENC,
> + IMX_MEDIA_GRP_ID_IC_PRPVF,
> + IMX_MEDIA_GRP_ID_IC_PP,
> +};
> +
> +static const u32 power_off_seq[] = {
> + IMX_MEDIA_GRP_ID_IC_PP,
> + IMX_MEDIA_GRP_ID_IC_PRPVF,
> + IMX_MEDIA_GRP_ID_IC_PRPENC,
> + IMX_MEDIA_GRP_ID_SMFC,
> + IMX_MEDIA_GRP_ID_CSI,
> + IMX_MEDIA_GRP_ID_VIDMUX,
> + IMX_MEDIA_GRP_ID_SENSOR,
> + IMX_MEDIA_GRP_ID_CSI2,
> +};

This seems somewhat arbitrary. Why is a power sequence needed?

[...]
> +/*
> + * Turn current pipeline power on/off starting from start_entity.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
> + struct media_entity_graph *graph,
> + struct media_entity *start_entity, bool on)
> +{
> + struct media_entity *entity;
> + struct v4l2_subdev *sd;

> + int i, ret = 0;

> + u32 id;
> +
> + for (i = 0; i < NUM_POWER_ENTITIES; i++) {
> + id = on ? power_on_seq[i] : power_off_seq[i];
> + entity = find_pipeline_entity(imxmd, graph, start_entity, id);
> + if (!entity)
> + continue;
> +
> + sd = media_entity_to_v4l2_subdev(entity);
> +
> + ret = v4l2_subdev_call(sd, core, s_power, on);
> + if (ret && ret != -ENOIOCTLCMD)
> + break;
> + }
> +
> + return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

> +/*
> + * Inherit the v4l2 controls from all entities in a pipeline
> + * to the given video device.
> + * Must be called with mdev->graph_mutex held.
> + */
> +int imx_media_inherit_controls(struct imx_media_dev *imxmd,
> + struct video_device *vfd,
> + struct media_entity *start_entity)
> +{
> + struct media_entity_graph graph;
> + struct media_entity *entity;
> + struct v4l2_subdev *sd;
> + int ret;
> +
> + ret = media_entity_graph_walk_init(&graph, &imxmd->md);

> + if (ret)
> + return ret;
> +

> + media_entity_graph_walk_start(&graph, start_entity);
> +
> + while ((entity = media_entity_graph_walk_next(&graph))) {
> + if (is_media_entity_v4l2_video_device(entity))
> + continue;
> +
> + sd = media_entity_to_v4l2_subdev(entity);
> +
> + dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
> + __func__, sd->name);
> +
> + ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
> + sd->ctrl_handler,

> + NULL);
> + if (ret)

> + break;
> + }
> +
> + media_entity_graph_walk_cleanup(&graph);
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
> +
> +MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");

> +MODULE_AUTHOR("Steve Longerbeam <steve_lo...@mentor.com>");
> +MODULE_LICENSE("GPL");

> diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
> new file mode 100644
> index 0000000..357654d
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-dev.c

This file is full of code that should live in the v4l2 core.

[...]
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-internal-sd.c
[...]
> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> + struct imx_media_subdev *csi[4])
> +{
> + int ret;
> +
> + /* there must be at least one CSI in first IPU */

Why?

> + if (!(csi[0] || csi[1]))
> + return -EINVAL;
> +
> + ret = add_ipu_internal_subdevs(imxmd, csi[0], csi[1], 0);

> + if (ret)
> + return ret;
> +

> + if (csi[2] || csi[3])
> + ret = add_ipu_internal_subdevs(imxmd, csi[2], csi[3], 1);

> +
> + return ret;
> +}

> diff --git a/drivers/staging/media/imx/imx-media-of.c b/drivers/staging/media/imx/imx-media-of.c
> new file mode 100644
> index 0000000..a939c34
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-media-of.c
> @@ -0,0 +1,289 @@
> +/*
> + * Media driver for Freescale i.MX5/6 SOC
> + *
> + * Open Firmware parsing.

> + *
> + * Copyright (c) 2016 Mentor Graphics Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */

> +#include <linux/of_platform.h>
> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>

> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>

> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +
> +static int of_add_pad_link(struct imx_media_dev *imxmd,
> + struct imx_media_pad *pad,
> + struct device_node *local_sd_node,
> + struct device_node *remote_sd_node,
> + int local_pad, int remote_pad)
> +{
> + dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
> + local_sd_node->name, local_pad,
> + remote_sd_node->name, remote_pad);
> +
> + return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
> + local_pad, remote_pad);
> +}
> +
> +/* parse inputs property from a sensor node */
> +static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
> + struct imx_media_subdev *sensor,
> + struct device_node *sensor_np)
> +{
> + struct imx_media_sensor_input *sinput = &sensor->input;
> + int ret, i;
> +
> + for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
> + const char *input_name;
> + u32 val;
> +
> + ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
> + if (ret)
> + break;
> +
> + sinput->value[i] = val;
> +
> + ret = of_property_read_string_index(sensor_np, "input-names",
> + i, &input_name);
> + /*
> + * if input-names not provided, they will be set using
> + * the subdev name once the sensor is known during
> + * async bind
> + */
> + if (!ret)
> + strncpy(sinput->name[i], input_name,
> + sizeof(sinput->name[i]));
> + }
> +
> + sinput->num = i;
> +
> + /* if no inputs provided just assume a single input */
> + if (sinput->num == 0)
> + sinput->num = 1;
> +}

This should be parsed by the sensor driver, not imx-media.

> +static void of_parse_sensor(struct imx_media_dev *imxmd,
> + struct imx_media_subdev *sensor,
> + struct device_node *sensor_np)
> +{

> + struct device_node *endpoint;
> +

Steve Longerbeam

unread,

Jan 13, 2017, 5:50:05 PM1/13/17

On 01/13/2017 03:57 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> Add to the MIPI CSI2 receiver node: compatible string, interrupt sources,
>> clocks.
>>
>> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
>> ---
>> arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
>> 1 file changed, 7 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi
>> index 53e6e63..42926e9 100644
>> --- a/arch/arm/boot/dts/imx6qdl.dtsi
>> +++ b/arch/arm/boot/dts/imx6qdl.dtsi
>> @@ -1125,7 +1125,14 @@
>> };
>>
>> mipi_csi: mipi@021dc000 {
>> + compatible = "fsl,imx6-mipi-csi2";
>> reg = <0x021dc000 0x4000>;
>> + interrupts = <0 100 0x04>, <0 101 0x04>;
>> + clocks = <&clks IMX6QDL_CLK_HSI_TX>,
>> + <&clks IMX6QDL_CLK_VIDEO_27M>,
>> + <&clks IMX6QDL_CLK_EIM_SEL>;
> I think the latter should be EIM_PODF

done.

>
>> + clock-names = "dphy", "cfg", "pix";
> and I'm not sure dphy is the right name for this one. Is that the pll
> ref input?

I believe this naming came from FSL's mipi csi-2 driver. It is the "hsi_tx"
clock (presumably for the MIPI HSI controller) whose parents are selected
by the CDCDR register as PLL3_120M or PLL2_PFD2. I have no clue whether
this is indeed also used as the clock for the MIPI CSI-2 D-PHY, but
according
to FSL naming convention it might be.

Steve

Steve Longerbeam

unread,

Jan 13, 2017, 6:20:04 PM1/13/17

This property is actually there to tell the driver what to set the
rate to, with clk_set_rate(). So you are saying it would be better
to set the rate in the device tree and the driver should only
retrieve the rate?

Steve

Steve Longerbeam

unread,

Jan 14, 2017, 5:50:05 PM1/14/17

(sorry sending again w/o html)

On 01/13/2017 07:20 AM, Philipp Zabel wrote:
> Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
>> +For image capture, the IPU contains the following internal subunits:
>> +
>> +- Image DMA Controller (IDMAC)
>> +- Camera Serial Interface (CSI)
>> +- Image Converter (IC)
>> +- Sensor Multi-FIFO Controller (SMFC)
>> +- Image Rotator (IRT)
>> +- Video De-Interlace Controller (VDIC)
> Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

>> +
>> +The IDMAC is the DMA controller for transfer of image frames to and from
>> +memory. Various dedicated DMA channels exist for both video capture and
>> +display paths.
>> +
>> +The CSI is the frontend capture unit that interfaces directly with
>> +capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
>> +
>> +The IC handles color-space conversion, resizing, and rotation
>> +operations.
> And horizontal flipping.

done.

>> There are three independent "tasks" within the IC that can
>> +carry out conversions concurrently: pre-processing encoding,
>> +pre-processing preview, and post-processing.
> s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

> This paragraph could mention that a single hardware unit is used
> transparently time multiplexed by the three tasks at different
> granularity for the downsizing, main processing, and rotation sections.
> The downscale unit switches between tasks at 8-pixel burst granularity,
> the main processing unit at line granularity. The rotation units switch
> only at frame granularity.

I've added that info.

>> +The SMFC is composed of four independent channels that each can transfer
>> +captured frames from sensors directly to memory concurrently.
>> +
>> +The IRT carries out 90 and 270 degree image rotation operations.
> ... on 8x8 pixel blocks, supported by the IDMAC which handles block
> transfers, block reordering, and vertical flipping.

done.

>> +The VDIC handles the conversion of interlaced video to progressive, with
>> +support for different motion compensation modes (low, medium, and high
>> +motion). The deinterlaced output frames from the VDIC can be sent to the
>> +IC pre-process preview task for further conversions.
>> +
>> +In addition to the IPU internal subunits, there are also two units
>> +outside the IPU that are also involved in video capture on i.MX:
>> +
>> +- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
>> + interface. This is a Synopsys DesignWare core.
>> +- A video multiplexer for selecting among multiple sensor inputs to
>> + send to a CSI.
> Two of them, actually.

done.

>> +
>> +- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
>> + problems with the ADV718x video decoders. See below for a description
>> + of the FIM.
> Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD
signal such as on the ADV718x. The FIM is only used to calculate
frame intervals at this point, but its input capture method could be
used to also record more accurate timestamps.

Camif is so named because it is the V4L2 user interface for video
capture. I suppose it could be named "capif", but that doesn't role
off the tongue quite as well.

>> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

>> +For example, its sink pad can take UYVY2X8, but its source pad can
>> +output YUYV2X8.
> I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> in the CSI chapter, for 8-bit per component data, the internal format
> between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> (or "bayer/generic data"). In either case, the internal format does not
> change along the way.

these are pixels in memory buffers, not the IPU internal formats.

>> +IC Direct Conversions:
>> +----------------------
>> +
>> +This pipeline uses the preprocess encode entity to route frames directly
>> +from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
>> +1024x1024 resolution, CSC, and image rotation:
>> +
>> +-> ipu_ic_prpenc -> camif
>> +
>> +This can be a useful capture pipeline for heavily loaded memory bus
>> +traffic environments, since it has minimal IDMAC channel usage.
> Note that if rotation is enabled, transfers between IC processing and
> rotation still have to go through memory once.

yep.

>> +Post-Processing Conversions:
>> +----------------------------
>> +
>> +This pipeline routes frames from the SMFC to the post-processing
>> +entity.
> No, frames written by the CSI -> SMFC -> IDMAC path are read back into
> the post-processing entity.

that's true. The post-processing entity kicks off its read channels
to transfer those frames into the post-processor. Anyway this wording
will change after doing away with the SMFC entity.

>> + media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
>> + media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
>> + media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
> I think the smfc entities should be dropped.

yes working on that.

>> + media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
>> + media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
>> + # Configure pads for OV5640 pipeline
>> + media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
>> + media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
>> + media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
>> + media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
>> + media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> [...]
>> + media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> I agree this looks very intuitive, but technically correct for the
> csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
>
> I think it would be better to use the correct format

I'm not sure I follow you here.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending
on sensor std in imx-vdic.c, but it should move upstream. Will fix.

>> + media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
>> + media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
> From here the interlaced field type should be sequential in the correct
> order depending on NTSC/PAL.

right.

>> +
>> +Frame Interval Monitor
>> +----------------------
>> +
>> +The adv718x decoders can occasionally send corrupt fields during
>> +NTSC/PAL signal re-sync (too little or too many video lines). When
>> +this happens, the IPU triggers a mechanism to re-establish vertical
>> +sync by adding 1 dummy line every frame, which causes a rolling effect
>> +from image to image, and can last a long time before a stable image is
>> +recovered. Or sometimes the mechanism doesn't work at all, causing a
>> +permanent split image (one frame contains lines from two consecutive
>> +captured images).
> Is it only SabreAuto on which the FIM mechanism can be used due to the
> pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just
needs to be enabled via status property.

> [...]
>> +/*
>> + * DMA buffer ring handling
>> + */
>> +struct imx_media_dma_buf_ring {
>> + struct imx_media_dev *imxmd;
>> +
>> + /* the ring */
>> + struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
>> + /* the scratch buffer for underruns */
>> + struct imx_media_dma_buf scratch;
>> +
>> + /* buffer generator */
>> + struct media_entity *src;
>> + /* buffer receiver */
>> + struct media_entity *sink;
>> +
>> + spinlock_t lock;
>> +
>> + int num_bufs;
>> + unsigned long last_seq;
>> +};
> I don't think this belongs in the capture driver at all.
> Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

> [...]
>> +static struct imx_media_dma_buf *
>> +__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
>> +{
>> + struct imx_media_dma_buf *buf;
>> +
>> + if (index >= ring->num_bufs)
>> + return ERR_PTR(-EINVAL);
>> +
>> + buf = &ring->buf[index];
>> + if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
>> + return ERR_PTR(-EINVAL);
>> +
>> + buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
>> + buf->seq = ring->last_seq++;
>> +
>> + return buf;
>> +}
> Is this a whole software buffer queue implementation? I thought the
> whole point of putting the custom mem2mem framework into the capture
> driver was to use the hardware FSU channel linking?

see below.

> What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries
in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel
to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
is support for the direct link from CSI (which I am using), but that's
not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
0..2 (and 3? it's not clear, and not clear whether this includes channel 1).
But I think this links to channel 12, and not to channels 8,9,10 to the
VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed
which
IDMAC channels these FSU links correspond to, instead of making us guess
at it.

In any event, the docs are not clear enough to implement a real FSU
link to the VDIC read channels, if it's even possible. And trying to get
programming help from FSL can be difficult, and no coding examples
for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software
approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC
entity via a v4l2_subdev_ioctl() call that a buffer is available. The
VDIC then manually kicks off its read channels to bring that buffer
(and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared
to a FSU hardware link: there is the EOF irq latency (a few usec), and
the CPU overhead for the VDIC to manually start the read channels,
which is also a few usec at most (see prepare_vdi_in_buffers() in
imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement
a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
at stream on, and the source requests a pointer to this ring in its own
stream on. Passing buffers from source to sink then follows a
straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
grabs queued buffers and makes them active, src signals completed
buffers to sink, sink dequeues buffers in response, and sink queues
buffers back when it is finished with them.

The CSI-2 receiver must be powered up before the sensor, that's the
only requirement IIRC. The others have no s_power requirement. So I
can probably change this to power up in the frontend -> backend order
(IC_PP to sensor). And vice-versa for power off.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be
doing some other stuff that bothered me, at least that's what I remember.
I will revisit this.

Agreed. Stuff like imx_media_inherit_controls() really should be widely
available.

>> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
>> + struct imx_media_subdev *csi[4])
>> +{
>> + int ret;
>> +
>> + /* there must be at least one CSI in first IPU */
> Why?

Well yeah, imx-media doesn't necessarily need a CSI if things
like the VDIC or post-processor are being used by an output
overlay pipeline, for example. I'll fix this.

you're probably right. I'll submit a patch for adv7180.c.

More that eventually needs exporting, agreed.

Steve

Philipp Zabel

unread,

Jan 16, 2017, 8:00:04 AM1/16/17

On Fri, 2017-01-13 at 15:04 -0800, Steve Longerbeam wrote:
>
> On 01/13/2017 04:03 AM, Philipp Zabel wrote:
> > Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:
> >> Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
> >> Both hang off the same i2c2 bus, so they require different (and non-
> >> default) i2c slave addresses.
> >>
> >> The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.
> >>
> >> The OV5640 connects to the input port on the MIPI CSI-2 receiver on
> >> mipi_csi. It is set to transmit over MIPI virtual channel 1.
> >>
> >> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> >> ---
> >> arch/arm/boot/dts/imx6dl-sabrelite.dts | 5 ++
> >> arch/arm/boot/dts/imx6q-sabrelite.dts | 6 ++
> >> arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
> >> 3 files changed, 129 insertions(+)
> >>
> >> diff --git a/arch/arm/boot/dts/imx6dl-sabrelite.dts b/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> index 0f06ca5..fec2524 100644
> >> --- a/arch/arm/boot/dts/imx6dl-sabrelite.dts
> >> +++ b/arch/arm/boot/dts/imx6dl-sabrelite.dts

[...]

> >> @@ -299,6 +326,52 @@
> >> pinctrl-names = "default";
> >> pinctrl-0 = <&pinctrl_i2c2>;
> >> status = "okay";
> >> +
> >> + ov5640: camera@40 {
> >> + compatible = "ovti,ov5640";
> >> + pinctrl-names = "default";
> >> + pinctrl-0 = <&pinctrl_ov5640>;
> >> + clocks = <&mipi_xclk>;
> >> + clock-names = "xclk";
> >> + reg = <0x40>;
> >> + xclk = <22000000>;
> > This is superfluous, you can use clk_get_rate on mipi_xclk.
>
> This property is actually there to tell the driver what to set the
> rate to, with clk_set_rate(). So you are saying it would be better
> to set the rate in the device tree and the driver should only
> retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a
given board and never changes during runtime, I think this is the
correct way. The clock will be fixed rate on most boards, I assume.

regards
Philipp

Philipp Zabel

unread,

Jan 16, 2017, 8:50:05 AM1/16/17

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]

> >> +Unprocessed Video Capture:
> >> +--------------------------
> >> +
> >> +Send frames directly from sensor to camera interface, with no
> >> +conversions:
> >> +
> >> +-> ipu_smfc -> camif
> > I'd call this capture interface, this is not just for cameras. Or maybe
> > idmac if you want to mirror hardware names?
>
> Camif is so named because it is the V4L2 user interface for video
> capture. I suppose it could be named "capif", but that doesn't role
> off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because
Samsung S3C has a camera interface that is actually called "CAMIF".

> >> +Note the ipu_smfc can do pixel reordering within the same colorspace.
> > That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).
>
> yes, the doc is re-worded to make that more clear.
>
> >> +For example, its sink pad can take UYVY2X8, but its source pad can
> >> +output YUYV2X8.
> > I don't think this is correct. Re-reading "37.4.3.7 Packing to memory"
> > in the CSI chapter, for 8-bit per component data, the internal format
> > between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant
> > (or "bayer/generic data"). In either case, the internal format does not
> > change along the way.
>
> these are pixels in memory buffers, not the IPU internal formats.

As long as we are talking about the CSI -> SMFC -> IDMAC path, these
should be IPU internal formats. How else would one choose between 8-bit
companded RGB, and 16-bit expanded RGB for a 10-bit per component input
signal? This is the same issue as in the next comment.

> >> + media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
> >> + media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
> >> + # Configure pads for OV5640 pipeline
> >> + media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
> >> + media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
> >> + media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
> >> + media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
> >> + media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
> > [...]
> >> + media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
> > I agree this looks very intuitive, but technically correct for the
> > csi1:1 and camif1:0 pads would be a 32-bit YUV format.
> > (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).
> >
> > I think it would be better to use the correct format
>
> I'm not sure I follow you here.

The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0
up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values
into the SMFC, so the CSI output pad and the IDMAC input pad should have
a YUV8_1X32 format.

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the
IDMAC chapter states that all internal submodules only work on 8-bit per
component formats with four components: YUVA or RGBA.

> > [...]

> > Is this a whole software buffer queue implementation? I thought the
> > whole point of putting the custom mem2mem framework into the capture
> > driver was to use the hardware FSU channel linking?
>
> see below.
>
> > What is the purpose of this if the sink should be triggered by the FSU?
>
> Ok, here is where I need to make an admission.
>
> The only FSU links I have attempted (and which currently have entries
> in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

Which are not described as media entity links because the rotation units
do not have separate media entities. So me arguing against handling
mem2mem chaining via media entity links doesn't concern these implicit
links.

> There does not appear to be support in the FSU for linking a write channel
> to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There
> is support for the direct link from CSI (which I am using), but that's
> not an
> IDMAC channel link.
>
> There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels
> 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are
only 2 CSIs, that shouldn't be a problem.

> But I think this links to channel 12, and not to channels 8,9,10 to the
> VDIC.
> Or will it? It's worth experimenting. It would have helped if FSL listed
> which
> IDMAC channels these FSU links correspond to, instead of making us guess
> at it.

I would have assumed that the FSU triggering only works on 1:1 channels
and the VDIC with its three input channels is different. But then
there's the alleged VDOA link to ch8/9/10 ro ch9, depending on
VDI_MOT_SEL.

This makes me more convinced that the CSI -> VDIC link should only
describe the direct path (real-time mode, single field).

> In any event, the docs are not clear enough to implement a real FSU
> link to the VDIC read channels, if it's even possible. And trying to get
> programming help from FSL can be difficult, and no coding examples
> for this link AFAIK.
>
> So I ended resorted to linking to VDIC channels 8,9,10 with a software
> approach, instead of attempting a hardware FSU link.
>
> The EOF interrupt handler for the SMFC channels informs the VDIC
> entity via a v4l2_subdev_ioctl() call that a buffer is available. The
> VDIC then manually kicks off its read channels to bring that buffer
> (and a previous buffer for F(n-1) field) into the VDIC.
>
> There is a small amount of extra overhead going this route compared
> to a FSU hardware link: there is the EOF irq latency (a few usec), and
> the CPU overhead for the VDIC to manually start the read channels,
> which is also a few usec at most (see prepare_vdi_in_buffers() in
> imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU
> use plus irq latency) under normal system load.

That the same low overhead could be reached by linking videobuf2 queues
of different video devices, that would be a lot more flexible.

> Of course, in order to implement this software link, I had to implement
> a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring
> at stream on, and the source requests a pointer to this ring in its own
> stream on. Passing buffers from source to sink then follows a
> straightforward
> FIFO queue/done/dequeue/queue model: sink queues buffers to src, src
> grabs queued buffers and makes them active, src signals completed
> buffers to sink, sink dequeues buffers in response, and sink queues
> buffers back when it is finished with them.

Thank you for the explanation.

[...]

> >> +static const u32 power_off_seq[] = {
> >> + IMX_MEDIA_GRP_ID_IC_PP,
> >> + IMX_MEDIA_GRP_ID_IC_PRPVF,
> >> + IMX_MEDIA_GRP_ID_IC_PRPENC,
> >> + IMX_MEDIA_GRP_ID_SMFC,
> >> + IMX_MEDIA_GRP_ID_CSI,
> >> + IMX_MEDIA_GRP_ID_VIDMUX,
> >> + IMX_MEDIA_GRP_ID_SENSOR,
> >> + IMX_MEDIA_GRP_ID_CSI2,
> >> +};
> > This seems somewhat arbitrary. Why is a power sequence needed?
>
> The CSI-2 receiver must be powered up before the sensor, that's the
> only requirement IIRC. The others have no s_power requirement. So I
> can probably change this to power up in the frontend -> backend order
> (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause
any problems. It would be better to reuse and, if necessary, fix the
existing infrastructure where available.

> >> +int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
> >> + struct imx_media_subdev *csi[4])
> >> +{
> >> + int ret;
> >> +
> >> + /* there must be at least one CSI in first IPU */
> > Why?
>
> Well yeah, imx-media doesn't necessarily need a CSI if things
> like the VDIC or post-processor are being used by an output
> overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a
parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power
saving reasons.

regards
Philipp

Philipp Zabel

unread,

Jan 16, 2017, 10:10:05 AM1/16/17

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
> This is a media entity subdevice for the i.MX Camera
> Serial Interface module.

s/Serial/Sensor/

> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
> ---

> drivers/staging/media/imx/Kconfig | 13 +
> drivers/staging/media/imx/Makefile | 2 +
> drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
> 3 files changed, 659 insertions(+)
> create mode 100644 drivers/staging/media/imx/imx-csi.c
>
> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
> index bfde58d..ce2d2c8 100644
> --- a/drivers/staging/media/imx/Kconfig
> +++ b/drivers/staging/media/imx/Kconfig
> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
> Say yes here to enable support for video4linux media controller
> driver for the i.MX5/6 SOC.
>
> +if VIDEO_IMX_MEDIA
> +menu "i.MX5/6 Media Sub devices"
> +
> +config VIDEO_IMX_CAMERA

s/CAMERA/CSI/ ?

> + tristate "i.MX5/6 Camera driver"

i.MX5/6 Camera Sensor Interface driver

> + depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
> + select VIDEOBUF2_DMA_CONTIG

> + default y
> + ---help---

> + A video4linux camera capture driver for i.MX5/6.
> +
> +endmenu
> +endif
> diff --git a/drivers/staging/media/imx/Makefile b/drivers/staging/media/imx/Makefile
> index ef9f11b..133672a 100644
> --- a/drivers/staging/media/imx/Makefile
> +++ b/drivers/staging/media/imx/Makefile
> @@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \

> obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
> obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
>

> +obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
> +
> diff --git a/drivers/staging/media/imx/imx-csi.c b/drivers/staging/media/imx/imx-csi.c
> new file mode 100644
> index 0000000..64ef862
> --- /dev/null
> +++ b/drivers/staging/media/imx/imx-csi.c
> @@ -0,0 +1,644 @@
> +/*
> + * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
> + *
> + * Copyright (c) 2014-2016 Mentor Graphics Inc.

> +#include <linux/module.h>
> +#include <linux/platform_device.h>

> +#include <media/v4l2-ctrls.h>
> +#include <media/v4l2-device.h>
> +#include <media/v4l2-of.h>
> +#include <media/v4l2-subdev.h>
> +#include <media/videobuf2-dma-contig.h>
> +#include <video/imx-ipu-v3.h>
> +#include "imx-media.h"
> +

> +#define CSI_NUM_PADS 2
> +
> +struct csi_priv {
> + struct device *dev;
> + struct ipu_soc *ipu;
> + struct imx_media_dev *md;
> + struct v4l2_subdev sd;
> + struct media_pad pad[CSI_NUM_PADS];
> + struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
> + struct v4l2_mbus_config sensor_mbus_cfg;
> + struct v4l2_rect crop;
> + struct ipu_csi *csi;
> + int csi_id;

> + int input_pad;
> + int output_pad;

> + bool power_on; /* power is on */

> + bool stream_on; /* streaming is on */
> +

> + /* the sink for the captured frames */
> + struct v4l2_subdev *sink_sd;
> + enum ipu_csi_dest dest;
> + struct v4l2_subdev *src_sd;

src_sd is not used except that its presence marks an enabled input link.
-> could be changed to bool.

> + struct v4l2_ctrl_handler ctrl_hdlr;
> + struct imx_media_fim *fim;

> +
> + /* the attached sensor at stream on */
> + struct imx_media_subdev *sensor;
> +};
> +

> +static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
> +{
> + return container_of(sdev, struct csi_priv, sd);
> +}
> +
> +/* Update the CSI whole sensor and active windows */
> +static int csi_setup(struct csi_priv *priv)
> +{
> + struct v4l2_mbus_framefmt infmt;
> +
> + ipu_csi_set_window(priv->csi, &priv->crop);
> +
> + /*
> + * the ipu-csi doesn't understand ALTERNATE, but it only
> + * needs to know whether the stream is interlaced, so set
> + * to INTERLACED if infmt field is ALTERNATE.
> + */
> + infmt = priv->format_mbus[priv->input_pad];
> + if (infmt.field == V4L2_FIELD_ALTERNATE)
> + infmt.field = V4L2_FIELD_INTERLACED;

That should be SEQ_TB/BT depending on video standard.

> + ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
> +
> + ipu_csi_set_dest(priv->csi, priv->dest);
> +
> + ipu_csi_dump(priv->csi);

> +
> + return 0;
> +}
> +

> +static int csi_start(struct csi_priv *priv)

> +{
> + int ret;
> +

> + if (!priv->sensor) {
> + v4l2_err(&priv->sd, "no sensor attached\n");

> + return -EINVAL;
> + }
> +

> + ret = csi_setup(priv);

> + if (ret)
> + return ret;
> +

> + /* start the frame interval monitor */
> + if (priv->fim) {
> + ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);

> + if (ret)
> + return ret;
> + }
> +

> + ret = ipu_csi_enable(priv->csi);
> + if (ret) {
> + v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
> + goto fim_off;
> + }

> +
> + return 0;
> +

> +fim_off:
> + if (priv->fim)
> + imx_media_fim_set_stream(priv->fim, priv->sensor, false);

> + return ret;
> +}
> +

> +static void csi_stop(struct csi_priv *priv)
> +{
> + /* stop the frame interval monitor */
> + if (priv->fim)
> + imx_media_fim_set_stream(priv->fim, priv->sensor, false);
> +
> + ipu_csi_disable(priv->csi);
> +}
> +
> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
> +{
> + struct csi_priv *priv = v4l2_get_subdevdata(sd);

> + int ret = 0;
> +
> + if (!priv->src_sd || !priv->sink_sd)
> + return -EPIPE;
> +
> + v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

These could be silenced a bit.

[...]
> +static int csi_s_power(struct v4l2_subdev *sd, int on)
> +{
> + struct csi_priv *priv = v4l2_get_subdevdata(sd);

> + int ret = 0;
> +

> + v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
> +
> + if (priv->fim && on != priv->power_on)
> + ret = imx_media_fim_set_power(priv->fim, on);
> +
> + if (!ret)
> + priv->power_on = on;
> + return ret;
> +}

Is this called multiple times? I'd expect a poweron during open and a
poweroff during close, so no need for priv->power_on.

> +static int csi_link_setup(struct media_entity *entity,

> + const struct media_pad *local,
> + const struct media_pad *remote, u32 flags)
> +{

> + struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> + struct csi_priv *priv = v4l2_get_subdevdata(sd);

> + struct v4l2_subdev *remote_sd;
> +

> + dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,

> + local->entity->name);
> +
> + remote_sd = media_entity_to_v4l2_subdev(remote->entity);
> +

> + if (local->flags & MEDIA_PAD_FL_SINK) {

> + if (flags & MEDIA_LNK_FL_ENABLED) {
> + if (priv->src_sd)
> + return -EBUSY;
> + priv->src_sd = remote_sd;
> + } else {
> + priv->src_sd = NULL;
> + }

> +
> + return 0;
> + }
> +

> + if (flags & MEDIA_LNK_FL_ENABLED) {
> + if (priv->sink_sd)
> + return -EBUSY;
> + priv->sink_sd = remote_sd;
> + } else {
> + priv->sink_sd = NULL;

> + return 0;
> + }
> +

> + /* set CSI destination */
> + switch (remote_sd->grp_id) {
> + case IMX_MEDIA_GRP_ID_SMFC0:
> + case IMX_MEDIA_GRP_ID_SMFC1:
> + case IMX_MEDIA_GRP_ID_SMFC2:
> + case IMX_MEDIA_GRP_ID_SMFC3:

With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1
to the SMFC2 channel.

[...]
> +static int csi_set_fmt(struct v4l2_subdev *sd,

> + struct v4l2_subdev_pad_config *cfg,
> + struct v4l2_subdev_format *sdformat)
> +{

> + struct csi_priv *priv = v4l2_get_subdevdata(sd);

> + struct v4l2_mbus_framefmt *infmt, *outfmt;

> + struct v4l2_rect crop;
> + int ret;
> +
> + if (sdformat->pad >= CSI_NUM_PADS)
> + return -EINVAL;
> +

> + if (priv->stream_on)
> + return -EBUSY;
> +
> + infmt = &priv->format_mbus[priv->input_pad];
> + outfmt = &priv->format_mbus[priv->output_pad];
> +

> + if (sdformat->pad == priv->output_pad) {

> + sdformat->format.code = infmt->code;

> + sdformat->format.field = infmt->field;

> + crop.left = priv->crop.left;
> + crop.top = priv->crop.top;
> + crop.width = sdformat->format.width;
> + crop.height = sdformat->format.height;
> + ret = csi_try_crop(priv, &crop);

> + if (ret)
> + return ret;

> + sdformat->format.width = crop.width;
> + sdformat->format.height = crop.height;

> + }
> +
> + if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {

Should there be some limitations on the format here?

regards
Philipp

Steve Longerbeam

unread,

Jan 16, 2017, 12:20:06 PM1/16/17

Ok, that makes sense, I'll make that change.

Steve

Steve Longerbeam

unread,

Jan 16, 2017, 4:20:05 PM1/16/17

On 01/16/2017 07:03 AM, Philipp Zabel wrote:
> On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
>> This is a media entity subdevice for the i.MX Camera
>> Serial Interface module.
> s/Serial/Sensor/

done.

>> Signed-off-by: Steve Longerbeam <steve_lo...@mentor.com>
>> ---
>> drivers/staging/media/imx/Kconfig | 13 +
>> drivers/staging/media/imx/Makefile | 2 +
>> drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
>> 3 files changed, 659 insertions(+)
>> create mode 100644 drivers/staging/media/imx/imx-csi.c
>>
>> diff --git a/drivers/staging/media/imx/Kconfig b/drivers/staging/media/imx/Kconfig
>> index bfde58d..ce2d2c8 100644
>> --- a/drivers/staging/media/imx/Kconfig
>> +++ b/drivers/staging/media/imx/Kconfig
>> @@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
>> Say yes here to enable support for video4linux media controller
>> driver for the i.MX5/6 SOC.
>>
>> +if VIDEO_IMX_MEDIA
>> +menu "i.MX5/6 Media Sub devices"
>> +
>> +config VIDEO_IMX_CAMERA
> s/CAMERA/CSI/ ?

done.

>> + tristate "i.MX5/6 Camera driver"
> i.MX5/6 Camera Sensor Interface driver

done.

>
>> +
>> +struct csi_priv {
>> + struct device *dev;
>> + struct ipu_soc *ipu;
>> + struct imx_media_dev *md;
>> + struct v4l2_subdev sd;
>> + struct media_pad pad[CSI_NUM_PADS];
>> + struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
>> + struct v4l2_mbus_config sensor_mbus_cfg;
>> + struct v4l2_rect crop;
>> + struct ipu_csi *csi;
>> + int csi_id;
>> + int input_pad;
>> + int output_pad;
>> + bool power_on; /* power is on */
>> + bool stream_on; /* streaming is on */
>> +
>> + /* the sink for the captured frames */
>> + struct v4l2_subdev *sink_sd;
>> + enum ipu_csi_dest dest;
>> + struct v4l2_subdev *src_sd;
> src_sd is not used except that its presence marks an enabled input link.
> -> could be changed to bool.

For now I prefer to keep it a pointer to the src/sink subdevs.
At some point the CSI may have some reason to know the
identity of the source.

fixed.

>> +
>> +static int csi_s_stream(struct v4l2_subdev *sd, int enable)
>> +{
>> + struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> + int ret = 0;
>> +
>> + if (!priv->src_sd || !priv->sink_sd)
>> + return -EPIPE;
>> +
>> + v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
> These could be silenced a bit.

yeah, I think it is time for that. I've silenced all the
v4l2_info()'s for stream on/off, power on/off, as well
as some others.

>
>> +static int csi_s_power(struct v4l2_subdev *sd, int on)
>> +{
>> + struct csi_priv *priv = v4l2_get_subdevdata(sd);
>> + int ret = 0;
>> +
>> + v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
>> +
>> + if (priv->fim && on != priv->power_on)
>> + ret = imx_media_fim_set_power(priv->fim, on);
>> +
>> + if (!ret)
>> + priv->power_on = on;
>> + return ret;
>> +}
> Is this called multiple times? I'd expect a poweron during open and a
> poweroff during close, so no need for priv->power_on.

It is actually called multiple times. The s_power subdev callbacks are
made every time there is a new link established, in the imx-media core's
link_notify(), in order to re-establish power to the active subdevs in the
new pipeline.

This might change after I look into using v4l2_pipeline_pm_use().

right, I'll do that.

done, I've added call to v4l_bound_align_image(), passing
it the min/max frame sizes. CSI's sensor/active frame size
register fields are 12 bits, so max is 4096 for both width and
height.

Steve

Steve Longerbeam

unread,

Jan 18, 2017, 8:50:05 PM1/18/17

> you're probably right. I'll submit a patch for adv7180.c.

Actually, the problem here is that this parses an input routing value to
pass to s_routing, and an input name string. There would need to be
another subdev callback, maybe enum_imput, that would return this
information for the bridge driver, if this info were to be parsed and
maintained by the sensor.

But this info should really be known and parsed by the bridge anyway,
because as the header for s_routing states,

"An i2c device shouldn't know about whether an input pin is connected
to a Composite connector, because on another board or platform it
might be connected to something else entirely. The calling driver is
responsible for mapping a user-level input to the right pins on the i2c
device."

Steve

Hans Verkuil

unread,

Jan 20, 2017, 9:00:07 AM1/20/17

Hi Steve, Philipp,

Based on the discussion on the mailinglist it seems everyone agrees that this
is the preferred driver, correct?

There are a bunch of review comments, so I will wait for a v4. I plan to merge
that staging driver unless there are major issues with it.

I am not sure if I would merge those sensor drivers in staging, I'll have to
take a closer look at those once v4 is posted.

Regards,

Hans

Hans Verkuil

unread,

Jan 20, 2017, 9:00:08 AM1/20/17

On 01/07/2017 03:11 AM, Steve Longerbeam wrote:

> From: Philipp Zabel <p.z...@pengutronix.de>
>
> Signed-off-by: Philipp Zabel <p.z...@pengutronix.de>
> ---
> Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
> include/uapi/linux/media.h | 6 ++++++
> 2 files changed, 28 insertions(+)
>
> diff --git a/Documentation/media/uapi/mediactl/media-types.rst b/Documentation/media/uapi/mediactl/media-types.rst
> index 3e03dc2..023be29 100644
> --- a/Documentation/media/uapi/mediactl/media-types.rst
> +++ b/Documentation/media/uapi/mediactl/media-types.rst
> @@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
> received on its sink pad and outputs the statistics data on
> its source pad.
>
> + - .. row 29
> +
> + .. _MEDIA-ENT-F-MUX:
> +
> + - ``MEDIA_ENT_F_MUX``

I would prefer MEDIA_ENT_F_VID_MUX since this is video specific.

Regards,

Hans

> +
> + - Video multiplexer. An entity capable of multiplexing must have at
> + least two sink pads and one source pad, and must pass the video
> + frame(s) received from the active sink pad to the source pad. Video
> + frame(s) from the inactive sink pads are discarded.
> +
> + - .. row 30
> +
> + .. _MEDIA-ENT-F-VID-IF-BRIDGE:
> +
> + - ``MEDIA_ENT_F_VID_IF_BRIDGE``
> +
> + - Video interface bridge. A video interface bridge entity must have at
> + least one sink pad and one source pad. It receives video frame(s) on
> + its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
> + converts them and outputs them on its source pad in another bus format
> + (eDP, MIPI CSI-2, parallel, ...).
>
> .. tabularcolumns:: |p{5.5cm}|p{12.0cm}|
>
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index 4890787..08a8bfa 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -105,6 +105,12 @@ struct media_device_info {
> #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS (MEDIA_ENT_F_BASE + 0x4006)
>
> /*
> + * Switch and bridge entitites
> + */
> +#define MEDIA_ENT_F_MUX (MEDIA_ENT_F_BASE + 0x5001)
> +#define MEDIA_ENT_F_VID_IF_BRIDGE (MEDIA_ENT_F_BASE + 0x5002)
> +
> +/*
> * Connectors
> */
> /* It is a responsibility of the entity drivers to add connectors and links */
>

It is loading more messages.

0 new messages