Is it possible to write new Historial Data Source and RealTime data feed

[Chaiyakorn Yingsaeree]

Hi,

   I would like to use tbg-Quant to develope and test a trading strategy in a foreign market which yahoo doesnot have a historical data and I need a way to load this historical into the system. My question is what do you think is the best way for me to do this, and is it possible to manually writing a new historical data source to accomplish this.

Regards,

Chai

[Sfl]

Hello,

Yes it is possible.

The simpler way is to use the CsvMarketDataFeed and load the historical data from an external csv file.

This class will be available soon, if it is what you are looking for, please let me know, I will push this feature for you.

If you need a real-time feeder you need to find a data service with APIs.

In any case you should be able to implement a customMarketDataFeed to push the data into tbg-quant event system.

I will provide you more details.

thank you

alberto

[Sfl]

Hello,

Here is a raw version of CsvCustomMarketDataFeed.

This works with a downloaded yahoo csv format, check the code, you can change it and build on it.

The goal is to change this class to build a fully configurable CsvCustom loader ( defining fields, dataformat, etc... )

/*

 * TBG-QUANT

 * The Bonnot Gang Quantitative Trading Framework 

 *  

 * This file is subject to the terms and conditions defined in

 * file 'LICENSE.txt', which is part of this source code package.

 * 

 */

package com.test;

import java.io.BufferedReader;

import java.io.FileReader;

import java.io.IOException;

import java.sql.Timestamp;

import java.text.ParseException;

import java.text.SimpleDateFormat;

import java.util.ArrayList;

import java.util.Collections;

import java.util.Date;

import java.util.List;

import java.util.StringTokenizer;

import org.apache.log4j.Logger;

import com.tbg.core.model.Security;

import com.tbg.core.model.marketDataFeed.BacktestMarketDataFeedBase;

import com.tbg.core.model.marketDataFeed.IMarketDataFeed;

import com.tbg.core.model.types.MarketDataEventType;

import com.tbg.core.model.types.TimeStamp;

import com.tbg.core.model.types.event.CandleEvent;

/**

 * Csv Custom MarketDataFeed <br>

 * Supports Daily, Weekly and Monthly.

 * <br>

 * <b>History:</b><br>

 *  - [15/lug/2012] Created. (sfl)<br>

 *

 *  @author Alberto Sfolcini <a.sfolcini(at)gmail.com>

 */

public class CsvCustomMarketDataFeed extends BacktestMarketDataFeedBase implements IMarketDataFeed{

protected final static Logger log = Logger.getLogger(CsvCustomMarketDataFeed.class);

private BufferedReader input;

private boolean skipFirstLine = true;

public CsvCustomMarketDataFeed(){

this.setMarketDataEvent(MarketDataEventType.CANDLE_EVENT);

}

/*

* (non-Javadoc)

* @see com.tbg.core.model.marketDataFeed.BacktestMarketDataFeedBase#onSubscribeMarketData(com.tbg.core.model.Security)

*/

@Override

public void onSubscribeMarketData(Security security) {

downloadQuotes(security);

}

/**

* Downloads Quotes

* @param security

*/

private void downloadQuotes(Security security){

List<String> content = new ArrayList<String>();

String csvFile = security.getSymbol().toString();

log.info("Fetching "+csvFile+" content...");

try {

input = new BufferedReader(new FileReader(csvFile));

} catch (IOException e1) {

log.error(e1);

}

if (skipFirstLine){ 

try {input.readLine();}catch(Exception e){}

}

// caricare qui i dati in arrayList

String str;

try{

while ((str = input.readLine()) != null) {

content.add(str);

}

}catch(Exception e){}

// reverse the order

Collections.reverse(content);

loadData(content,security);

}

/**

* loads data, one candle at time and generate candleEvents...

*/

private void loadData(List<String> content,Security security) {

double open = 0.0; 

double high = 0.0;

double low = 0.0;

double close = 0.0;

double adjClose = 0.0;

int volume = 0;

TimeStamp timestamp = null;

ArrayList<CandleEvent> candles = new ArrayList<CandleEvent>();

for(int i=0;i<content.size();i++){

StringTokenizer st = new StringTokenizer(content.get(i).toString(), ","); 

while(st.hasMoreTokens()){

String data_tmp = st.nextToken();

Date dt = null;

try {

dt = new SimpleDateFormat("yyyy-MM-dd").parse(data_tmp);

} catch (ParseException e) {

log.error(e);

}

timestamp = new TimeStamp(new Timestamp(dt.getTime()));

open = Double.parseDouble(st.nextToken());

high = Double.parseDouble(st.nextToken());

low = Double.parseDouble(st.nextToken());

close = Double.parseDouble(st.nextToken());

volume = Integer.parseInt(st.nextToken());

adjClose = Double.parseDouble(st.nextToken());

CandleEvent candleEvent = new CandleEvent();

candleEvent.setSymbol(security.getSymbol().toString());

candleEvent.setOpenPrice(open);

candleEvent.setHighPrice(high);

candleEvent.setLowPrice(low);

//candleEvent.setClosePrice(adjClose);

candleEvent.setClosePrice(close);

candleEvent.setTimeStamp(timestamp);

candleEvent.setVolume(volume);

if (debug) log.info(" ====> Csv Candle: "+candleEvent.toString());

candles.add(candleEvent);

candleEvents.put(security, candles);

}

} // end for

}

@Override

public void connectToMarketFeed() {

log.info("Connected to YahooMarketDataFeed.");

}

@Override

public void disconnectFromMarketDataFeed() {

log.info("Connection to YahooMarketDataFeed closed.");

}

} // end class

Then, take a Skel strategy and use the CsvCustomMarketDataFeed :

private final IMarketDataFeed marketDataFeed = new CsvCustomMarketDataFeed();

I did not release this code under my git repository nor on tbg-quant official repository because it is just a raw implementation I did now.

If you are going to implement it please send it to us, we will be happy to include it in the next release if you want.

hope this helps, let me know if you need more details.

Alberto