Help me choose an FPGA to design network protocols

[PM X]

Hi all,
I have over a decade of experience in hardware design, but almost all of it is in ASIC. I had done some FPGA projects in school, but nothing after that. So after all these years, I want to work on some personal FPGA projects (mainly to prepare myself for future job interviews). I have the following two questions.

1. What is an FPGA board that I can buy for this purpose? I am probably looking to do something not too basic (since I already have a lot of experience in design), at the same time I do not want to make it a super complicated full time project either. I prefer Xilinx (but I am open) and something less than $250 will be good. Note that question #2 might also affect the choice of board.

2. Once I have the FPGA board, I would like to implement some design involving network protocol (like TCP/IP, UDP, etc.). However, I have not worked on these network layers before and don't have an extensive knowledge on them either (other than what I had read in school long ago), so I do not have a very clear picture of what to do. Is there any open source design available on this? Or any projects with specific definitions that I can understand and then start implementing?

Thanks!

[rickman]

On 8/26/2016 6:47 PM, PM X wrote:
> Hi all, I have over a decade of experience in hardware design, but
> almost all of it is in ASIC. I had done some FPGA projects in school,
> but nothing after that. So after all these years, I want to work on
> some personal FPGA projects (mainly to prepare myself for future job
> interviews). I have the following two questions.
>
> 1. What is an FPGA board that I can buy for this purpose? I am
> probably looking to do something not too basic (since I already have
> a lot of experience in design), at the same time I do not want to
> make it a super complicated full time project either. I prefer Xilinx
> (but I am open) and something less than $250 will be good. Note that
> question #2 might also affect the choice of board.

There are lots of boards available under $100 but likely won't have
networking support. Here is one with Ethernet for $200.

https://www.arrow.com/en/products/sf2plus-dev-kit/arrow-development-tools#page-8

It uses a Microsemi Smartfusion2 part which contains an ARM CM4 I believe.

> 2. Once I have the FPGA board, I would like to implement some design
> involving network protocol (like TCP/IP, UDP, etc.). However, I have
> not worked on these network layers before and don't have an extensive
> knowledge on them either (other than what I had read in school long
> ago), so I do not have a very clear picture of what to do. Is there
> any open source design available on this? Or any projects with
> specific definitions that I can understand and then start
> implementing?

This is often done with a CPU rather than hardware. TCP/IP protocols
are very complicated.

As for Open Source, what's wrong with Linux? Or do I misunderstand?

--

Rick C

[PM X]

On Friday, August 26, 2016 at 10:28:45 PM UTC-7, rickman wrote:
> On 8/26/2016 6:47 PM, PM X wrote:
> > Hi all, I have over a decade of experience in hardware design, but
> > almost all of it is in ASIC. I had done some FPGA projects in school,
> > but nothing after that. So after all these years, I want to work on
> > some personal FPGA projects (mainly to prepare myself for future job
> > interviews). I have the following two questions.
> >
> > 1. What is an FPGA board that I can buy for this purpose? I am
> > probably looking to do something not too basic (since I already have
> > a lot of experience in design), at the same time I do not want to
> > make it a super complicated full time project either. I prefer Xilinx
> > (but I am open) and something less than $250 will be good. Note that
> > question #2 might also affect the choice of board.
>
> There are lots of boards available under $100 but likely won't have
> networking support. Here is one with Ethernet for $200.
>
> https://www.arrow.com/en/products/sf2plus-dev-kit/arrow-development-tools#page-8
>
> It uses a Microsemi Smartfusion2 part which contains an ARM CM4 I believe.
>

Sorry, I have some more newbie type questions. How does a board like the above supposed to work considering the ARM core in there? My understanding is that whatever design I implement in HDL runs in the actual FPGA, and any additional software program that I also want to run goes to the ARM processor? So we have two separate runs going on - one in the FPGA and the other in the ARM - and these two can communicate with each other?

One other thing I am confused about is that, are the development boards always made by third parties using the FPGA from companies like Xilinx? Or does Xilinx ever make their own development board?

I am thinking about getting a board with Xilinx FPGA, probably one of the older Virtex ones. Found this on ebay: http://www.ebay.com/itm/XILINX-VIRTEX-4-XC4VFX100-FPGA-kit-Development-board-XKF4/181791876772 , any comments?

>
> > 2. Once I have the FPGA board, I would like to implement some design
> > involving network protocol (like TCP/IP, UDP, etc.). However, I have
> > not worked on these network layers before and don't have an extensive
> > knowledge on them either (other than what I had read in school long
> > ago), so I do not have a very clear picture of what to do. Is there
> > any open source design available on this? Or any projects with
> > specific definitions that I can understand and then start
> > implementing?
>
> This is often done with a CPU rather than hardware. TCP/IP protocols
> are very complicated.
>
> As for Open Source, what's wrong with Linux? Or do I misunderstand?

I meant some part of the TCP/IP stack implemented in actual hardware and running in the FPGA. For example, a TCP Offload Engine, which implements the whole TCP/IP stack in hardware. Now, I understand that is something enormous and do not want to work on something that big for just a side/hobby project. But something in that area with less complexity is what I am looking for. I just don't have a good idea of what that could be, so having some clear definition will help.

[rickman]

Hmmm.... Yes, the ARM will run software written in your favorite
sequential language (C, Python, etc) while the FPGA doesn't exactly
"run" anything. HDL stands for Hardware Description Language. The HDL
doesn't so much get compiled to something that "runs" like a computer
program, but rather it is compiled to a configuration file that controls
how the various elements of the FPGA hardware are interconnected. The
hardware needed in the FPGA is "described".

Exactly how the two communicate depends on the vendor. Xilinx and
Altera have similar offerings which their own interfaces between the CPU
and the FPGA fabric. But basically the CPU program will perform port
I/Os or perhaps DMA is used to move data between the FPGA and memory. I
haven't done a project with this chip as yet.

> One other thing I am confused about is that, are the development
> boards always made by third parties using the FPGA from companies
> like Xilinx? Or does Xilinx ever make their own development board?

The only boards I've seen made by the FPGA vendors tend to be a bit
pricier than the ones made by third parties. That's not to say third
parties don't make pricey boards, they span a wider range I guess. FPGA
vendors make boards so they can sell their chips. Board vendors are
selling boards and so make them as fully functional as they can for the
price.

> I am thinking about getting a board with Xilinx FPGA, probably one of
> the older Virtex ones. Found this on ebay:
> http://www.ebay.com/itm/XILINX-VIRTEX-4-XC4VFX100-FPGA-kit-Development-board-XKF4/181791876772
> , any comments?

Why an older FPGA? Is it price?

As you may have figured out, I'm not a big fan of Xilinx. I keep
hearing about all manner of issues with their in house tool, Vivaldo.
It used to be XST, but the scrapped that. In fact, my very first FPGA
design was a Xilinx and during a four month project (or maybe six, I
don't recall exactly) I had to ditch tools twice. The first time was to
drop the Orcad VHDL tool as it was largely non-functional, switching to
the Xilinx tool. Then a second time when Xilinx dropped their earlier
tool and only supported a new one. Later they came out with XST and now
Vivaldo. I don't get that they have to keep tossing tools. It makes it
hard to maintain lifetime support for your product.

On the other hand Lattice ditched the chip I have designed on a board I
am still making good money from, *very* good money. Fortunately there
is sufficient inventory that I won't have to redesign the board for a
number of years. At least they use third party synthesis so it isn't a
problem to keep supporting products over their lifetime.

>>> 2. Once I have the FPGA board, I would like to implement some
>>> design involving network protocol (like TCP/IP, UDP, etc.).
>>> However, I have not worked on these network layers before and
>>> don't have an extensive knowledge on them either (other than what
>>> I had read in school long ago), so I do not have a very clear
>>> picture of what to do. Is there any open source design available
>>> on this? Or any projects with specific definitions that I can
>>> understand and then start implementing?
>>
>> This is often done with a CPU rather than hardware. TCP/IP
>> protocols are very complicated.
>>
>> As for Open Source, what's wrong with Linux? Or do I
>> misunderstand?
>
> I meant some part of the TCP/IP stack implemented in actual hardware
> and running in the FPGA. For example, a TCP Offload Engine, which
> implements the whole TCP/IP stack in hardware. Now, I understand that
> is something enormous and do not want to work on something that big
> for just a side/hobby project. But something in that area with less
> complexity is what I am looking for. I just don't have a good idea of
> what that could be, so having some clear definition will help.

I have *no* idea what that could be. As I mentioned, my understanding
is the TCP/IP protocol in particular is *very* complex and people don't
want to implement their own software on a CPU, much less in hardware.
I'm not sure what an "offload engine" would be other than a dedicated
CPU, optimized for TCP/IP. But if you have a CM4 in the device, why not
use that?

To make a hybrid implementation with part of the TCP/IP stack in the CPU
and part in the FPGA fabric would require you to intimately understand
this algorithm. I believe I've read that a lot of the details are not
in any specification, rather are coded in the applications. So you
might need to reverse engineer one or more implementations.

--

Rick C

[Tom Gardner]

On 27/08/16 18:20, PM X wrote:
> I meant some part of the TCP/IP stack implemented in actual hardware and running in the FPGA. For example, a TCP Offload Engine, which implements the whole TCP/IP stack in hardware. Now, I understand that is something enormous and do not want to work on something that big for just a side/hobby project. But something in that area with less complexity is what I am looking for. I just don't have a good idea of what that could be, so having some clear definition will help.

Protocol stack offload engines have two fundamental "issues".

Firstly the time and latency required to get the packets
to/from the main CPU. That significantly affects performance.

Secondly the point that TCP is and end-to-end protocol
and will correct protocol errors between or in the
endpoints. An offload engine becomes the endpoint, so
errors between the offload engine and the main CPU will
not be corrected.

The processing required to reliably transfer packets
to/from the main CPU bears a lot of similarity to TCP!

TCP in FPGA makes sense where
- the lowest latency is required, and
- where simplifying assumptions can be made, and
- where the end terminal logic is also done in the FPGA
e.g. high frequency trading, where they put the business
trading rules in hardware to minimise latency

[PM X]

On Saturday, August 27, 2016 at 11:09:26 PM UTC-7, rickman wrote:

> > I am thinking about getting a board with Xilinx FPGA, probably one of
> > the older Virtex ones. Found this on ebay:
> > http://www.ebay.com/itm/XILINX-VIRTEX-4-XC4VFX100-FPGA-kit-Development-board-XKF4/181791876772
> > , any comments?
>
> Why an older FPGA? Is it price?
>

Yes. The newer Virtex boards cost a lot more (the ones using Virtex 6 or 7 run into thousands of dollars).
The only reason I am thinking of Xilinx is because it is probably the most used FPGA (along with maybe Altera), so I thought having it on my resume might provide some advantage.

> I have *no* idea what that could be. As I mentioned, my understanding
> is the TCP/IP protocol in particular is *very* complex and people don't
> want to implement their own software on a CPU, much less in hardware.
> I'm not sure what an "offload engine" would be other than a dedicated
> CPU, optimized for TCP/IP. But if you have a CM4 in the device, why not
> use that?
>

Because the main purpose of this exercise is to design some network protocol in real hardware (FPGA) so I can make myself marketable for jobs in that area. If the network protocol part is too complicated as a side project, then I might just have to write some other (simpler) design and at least become familiar with the FPGA aspects of design.

[PM X]

As a matter of fact, I am doing this to prepare myself for interviews in FPGA design role in high frequency trading (in maybe 3-6 months time). Is there any simpler project that you could suggest that would be in the general area of network protocols (maybe like a stripped down version) that could be implemented in FPGA? Thanks!

[Tom Gardner]

Based on precisely *zero* evidence and only 30s thought,
I would guess the networking stack is known technology,
whereas business/trading rules are newer.

ISTR seeing horrendously expensive boards (worth about
1ms of HFT!) with large numbers of big FPGAs and
networking. If that's correct then partitioning the
trading rules across FPGAs might be interesting. Or not.

[rickman]

Ok, that explains a lot. I would suggest that you start with learning
IP stack software. Before you can implement it in the FPGA you have to
understand it. So first learn IP stack software. Once you understand
how that works you can decide how best to implement it in logic.

--

Rick C

[Rob Doyle]

UDP/IP is much simpler the TCP/IP. It is commonly done in FPGAs.

For example:

http://www.fpga4fun.com/10BASE-T.html

OK. It is only 10Base-T. But it's not that different than the 10GbE that
we do.

You can get a crappy NIC and Basys 3 Artix 7 board for less than $200.00

http://store.digilentinc.com/pmodnic100-network-interface-controller/

It won't be low latency (the NIC has an SPI serial interface) but it
will teach concepts.

Rob.

[PM X]

>
> UDP/IP is much simpler the TCP/IP. It is commonly done in FPGAs.
>
> For example:
>
> http://www.fpga4fun.com/10BASE-T.html
>
> OK. It is only 10Base-T. But it's not that different than the 10GbE that
> we do.
>
> You can get a crappy NIC and Basys 3 Artix 7 board for less than $200.00
>
> http://store.digilentinc.com/pmodnic100-network-interface-controller/
>
> It won't be low latency (the NIC has an SPI serial interface) but it
> will teach concepts.
>
> Rob.

Thanks. Is this the board you are referring to?
http://store.digilentinc.com/basys-3-artix-7-fpga-trainer-board-recommended-for-introductory-users/

If so, this board doesn't seem to have SPI (at least no listed in the description). Also, do you think this board has enough capacity (in terms of logic elements, etc.) to support a fairly complicated design like UDP/IP?

[Theo Markettos]

PM X <pin...@gmail.com> wrote:
> As a matter of fact, I am doing this to prepare myself for interviews in
> FPGA design role in high frequency trading (in maybe 3-6 months time). Is
> there any simpler project that you could suggest that would be in the
> general area of network protocols (maybe like a stripped down version)
> that could be implemented in FPGA? Thanks!

I'd start bottom-up.

First, get the PHY working. You probably want a single speed without
any rate switching (eg 1G, 10G), because anything else gets messy with
clock reconfiguration. The choice will likely depend on your board.
(You can't really test it at this point). Hopefully your board vendor has
example code for this.

Then drop in a vendor MAC component. I've used the Altera ones and they're
not too bad: there are pipes on the side for streams in and out which are
easy to deal with. There is a memory mapped interface for configuration -
you may need to implement that to configure it, or maybe the defaults are
OK. (To configure, use a soft core - NIOS or Microblaze or whatever).

On the subject of board choice, I'd suggest avoiding anything with an
external MAC chip (external PHY is OK) because they expect to be driven from
a processor, while on FPGA it's all about packet pipes. Likewise a MAC as
part of a CPU subsystem (eg a Zynq PS, Altera SoC-FPGA HPS) should be avoided
because they usually have the same problem, as well as being awkward
poorly-documented third-party IP where you essentially have to use the Linux
driver.

Once you have the MAC working, you have layer 2 packets in and out (you can
see MAC addresses etc). Then you can start building up the layers (eg do IP
and ARP). The MAC will probably give you some help for layer 2 (eg compute
checksums for you) but you're on your own after that.

Building up the layers is something you can simulate, while doing the
plumbing of the MAC you can only test in hardware. I'd suggest writing a
testbench that simulates pushing layer 2 packets between two simulated
endpoints, and you can replace that by MAC+PHY on hardware when necessary.

If you want experience of networking on FPGA, you don't need to do a full
TCP stack to do that: much of the principles of pushing packets about and
dealing with vendor IP cores is the same irrespective of the packet format.
If it makes you think about latency and meeting timing, that's a useful
thing.

If you want to learn about the vagaries of TCP, IPv6, etc I'd suggest
starting from software first. Maybe find some NIC with a non-scary
interface and program it directly (with no OS support). I'm thinking
something old like a PCI NE2000 (eg RTL8139) that isn't as complex as a
modern card - the Intel gigabit E1000(e) series (8254x, 8257x, I21x) are
well documented but a bit more complex. While you can do this from FPGA,
it's much harder to debug.

Theo

[Cecil Bayona]

Right off the bat I'm not much of an expert so maybe some that are more
experienced can comment. Have you looked at the Digilent Arty Artix 7
FPGA board? It's inexpensive at $99, and has a built in Ethernet
Physical, and it mentions that it comes with A MAC IP so you are off to
a start. It has 35K cells so I'm not familiar if that would be enough
to implement the rest of the needed logic. It has both Arduino, and Pmod
connectors so if desired you can add an external Ethernet module that
are available in several chip versions.

<
http://store.digilentinc.com/arty-board-artix-7-fpga-development-board-for-makers-and-hobbyists/
>

Here is one external Ethernet adapter, there are many other available
from other places.

< http://store.digilentinc.com/pmodnic100-network-interface-controller/ >


--
Cecil - k5nwa

[rickman]

What do you mean it doesn't have SPI? SPI is a simple shift register
interface which can *easily* be implemented in an FPGA (or MCU) using
the GPIOs.

Do you mean ISP, in system programming? If it doesn't have ISP how do
you load your design?

--

Rick C

[Cecil Bayona]

It's a FPGA, you can add SPI easily, there are IPs for free to allow
that to happen.

In my post I was also going to mention the BASYS-3 board, I left it out
because the Arty Board has a ton of memory available that this one
doesn't but this on has a lot of switches and LED which can be handy.
--
Cecil - k5nwa

[PM X]

I meant I didn't see a (dedicated) SPI interface. But you're right. SPI protocol just needs 4 general purpose pins, so using GPIOs should do it.

[PM X]

OK, thanks. I will check out both of them. What is the largest design you (or someone you know) have implemented on these boards? The Artix line seems to be lower end than Virtex line, so trying to get an idea if they can support somewhat complicated designs.

[Cecil Bayona]

Nothing Fancy, that is why in my earlier post I mentioned that I don't
have a lot of experience. I been working a 32 bit stack based CPU, but
it's a work in progress, I'm still sorting it out, it taken less than
20% of the chip, but a stack CPU are rather simple compared to other
CPU's, when finished it should be pretty nice, most instructions take
one clock to execute, and it used packed instructions, 5 instructions to
a word fetch. Originally it was on a Lattice Brevia2, I am now
converting it to a Artix-7 board, but there is software involved too so
it's going slow and I'm learning as I go.
--
Cecil - k5nwa

[rickman]

No sure what "lower end" means in technical terms. I expect the Artix
line of FPGAs will easily support somewhat complicated designs for
reasonable values of "somewhat complicated".

I suggest you not give much credence to marketing information and
consider the chip specifications. For the most part the important issue
is the LUT count. Otherwise the extra features are only useful if you
need them.

--

Rick C

[rickman]

Just a comment on your stack processor. I've done some design work with
stack processors and read about a lot of designs. In my humble opinion,
if you have multiple cycle instructions, you are doing it wrong. I
don't want to steal the thread. If you care to discuss this we can
start another thread.

--

Rick C

[Michael Kellett]

PM X:

I haven't used the Arty except on a 1 day Xilinx "get to know Vivado"
jolly. It's quite anice board. We are looking at alternatives to Lattice
(no big reason, just due diligence) with a design currently running on a
Lattice ECP3-35. The FPGA on the Arty could easily do Ethernet trispeed
MAC, IP, ARP, UDP and some more. (We do this on the Lattice (about 20%
of it) so I know of what I speak (at least that far)) TCP on the FPGA
would be bigger (could be much bigger).

As an aside - why does HF trading use TCP - (think of this as an
interview question :-)

Our UDP support is good at transmitting, poor at receiving ('coz that's
what we need) - it does support an in-house protocol for re-transmission
and message integrity - much simpler than TCP and faster.

By the time you've outgrown the Arty you've either done the career shift
or it isn't going happen.

Michael Kellett

[PM X]

Since many people have suggested Arty, I was giving it a serious thought. But although it does have ethernet (which I need), it does not have any kind of video output like VGA (I would strongly prefer to get a board with some kind of video output as well besides ethernet). I do not want to try and add VGA to it as I have no experience with that stuff. Another Digilent board that I really liked is the Nexys 4 DDR, which has both ethernet and VGA. But the price is a little over my budget at $320. I am trying to do this within $200, although I it can go up a bit (to maybe $300) if I don't find any good option in the "< $200" range.

To re-iterate, I am looking for a board with a Xilinx FPGA (preferably Artix-7) that uses the Vivado software. And as stated above, I want ethernet and video output connectivity support. So now that I have a clearer idea than before, if somebody can suggest a board meeting the above requirements (in the $200 range), that will be great.

So far I had only been looking at boards from Digilent, but I just came across another company called Avnet which also sells Xilinx boards. They seem to have a lot more choices when I search for a board with Artix-7 (while Digilent has only 3 or 4). I didn't understand very clearly what type of boards they sell. Do they make them or do they just distribute the boards made by Xilinx? Which one would be a better option - Digilent or Avnet? Or something else?

Thanks.

[lasselangwad...@gmail.com]

Den søndag den 28. august 2016 kl. 08.09.26 UTC+2 skrev rickman:
>
> As you may have figured out, I'm not a big fan of Xilinx. I keep
> hearing about all manner of issues with their in house tool, Vivaldo.
> It used to be XST, but the scrapped that. In fact, my very first FPGA
> design was a Xilinx and during a four month project (or maybe six, I
> don't recall exactly) I had to ditch tools twice. The first time was to
> drop the Orcad VHDL tool as it was largely non-functional, switching to
> the Xilinx tool. Then a second time when Xilinx dropped their earlier
> tool and only supported a new one. Later they came out with XST and now
> Vivaldo. I don't get that they have to keep tossing tools. It makes it
> hard to maintain lifetime support for your product.
>
> On the other hand Lattice ditched the chip I have designed on a board I
> am still making good money from, *very* good money. Fortunately there
> is sufficient inventory that I won't have to redesign the board for a
> number of years. At least they use third party synthesis so it isn't a
> problem to keep supporting products over their lifetime.
>

if you had any current experience you would know that the name is Vivado
and that Xilinx have all versions of design tools going back something like
15 years still available


-Lasse