Question about the send of file versus the django server

[simone monteleone]

Hi all,

I have to send a file (ex. csv file) to the Django server.

I think to use the requests python module.

Is this the best way?

Second question; Django can receive this file?

And also in this case, what it is the best way?

My goal is to load the csv file information in the dabatase handles by django,

My best regards.

Thanks in advice for the helpuful.

Simone Monteleone

[Russell Keith-Magee]

On Thu, Feb 13, 2014 at 4:11 PM, simone monteleone <ares...@gmail.com> wrote:

Hi all,

I have to send a file (ex. csv file) to the Django server.

I think to use the requests python module.

Is this the best way?

The "best" way will depend on your exact requirements - I'll come back to this in a moment. However, Requests will certainly make it easy to write a Python script that will send a file to a Django server.

 

Second question; Django can receive this file?

If it can be sent over HTTP, Django can receive it.

 

And also in this case, what it is the best way?

Django's file handling documentation contains details on how to set up this sort of thing:

https://docs.djangoproject.com/en/1.6/topics/http/file-uploads/

My goal is to load the csv file information in the dabatase handles by django,

 

We need to be clear about your needs here.

Do you actually need to do this over HTTP? I.e., you may want the data to be put into your database, but is there actually a requirement to upload that file? Could you just load the file onto the database directly? 

Django contains a bunch of tools for uploading files directly, without going through a web interface. Look into loaddata management command for details.

https://docs.djangoproject.com/en/1.6/ref/django-admin/

There isn't a native CSV importer, but there are a couple floating around the web that you might be able to use.

The other approach - just write the data directly to the database. Write some code to read the CSV file, and use Django's model objects to write the data directly; e.g., if you've got a model called MyModel, with two fields field1 and field2:

  MyModel.objects.create(field1=value1, field2=value2) 

will create a record in the database with those values.

Yours,

Russ Magee %-)

[simone monteleone]

Hi Russ,

My goal is to load the csv file directly in the database.

So my idea is define a script containing the follow pseudo-code:

 while(1) and receive_csv():

    value1, value2 = read_csv()

  
    MyModel.objects.create(field1=value1, field2=value2)

The script run on the server, ex http://127.0.0.1:8000/admin

At this point it's possible using the script reported?

import requests

url = 'http://127.0.0.1:8000/admin'

files = {'file': open('log', 'rb')}

r = requests.post(url, files=files)

I don't understand where send the file respect the Django server.

Thanks very much.

SM 

[François Schiettecatte]

Hi

See this:

https://docs.djangoproject.com/en/1.6/topics/http/file-uploads/

Cheers

François

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a6d9b203-8f37-4e5b-b49e-d12c96981c0a%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Russell Keith-Magee]

Hi Simone,

On Sun, Feb 16, 2014 at 2:53 AM, simone monteleone <ares...@gmail.com> wrote:

Hi Russ,

My goal is to load the csv file directly in the database.

So my idea is define a script containing the follow pseudo-code:

 while(1) and receive_csv():

    value1, value2 = read_csv()

  
    MyModel.objects.create(field1=value1, field2=value2)

The script run on the server, ex http://127.0.0.1:8000/admin

Do you want the *file*, or the *data contained in the file*?

Based on the script you've provided, you want the *data*. If so, you don't run a script like this at a URL. You run it at a Python prompt.

When you went through the Django tutorial, the first steps were all done at a prompt. Those commands let you inspect the data that was in the database. You can also use the prompt to *create* data in the database - that's what your script needs to do.

You don't need to upload the script to the server, or write any view code - you're writing a set of commands that you will run at a prompt to open a file, read some data, and write to the database. Then you can start your Django server, and see that data *on* your server.

 

Is that more clear?

Yours

Russ Magee %-)

[simone monteleone]

Hi Russ,

The script is now running in a normal bash shell, where I import the DJANGO_SETTINGS_MODULE.

So it's correct to run the script on the machine where the server goes?

Or it's possible (and better) runs the commands inside Django?

Thanks

SM

[simone monteleone]

In other hand, it's possible to use the Django server to intercept
a requests?

SM

[Russell Keith-Magee]

On Sun, Feb 16, 2014 at 10:20 PM, simone monteleone <ares...@gmail.com> wrote:

Hi Russ,

The script is now running in a normal bash shell, where I import the DJANGO_SETTINGS_MODULE.

So it's correct to run the script on the machine where the server goes?

Or it's possible (and better) runs the commands inside Django?

You're missing my point - there's no such thing as "inside Django". Logging onto the server and running the script is the right way to do what you are describing.

A running Django web server isn't a "place" you can run code - it's essentially just a way to turn http://myservice.com into a function call that returns a web page. Requests is just a tool for using a Python script to generate HTTP requests. 

Yes, you can use requests to connect to a Django web server. However, if you're in control of the web server, using requests to do what you describe, instead of just logging onto the server and running a script directly is a bit like saying "I'm going to pick up the cup that is sitting on that table using a fishing rod". Yes, you can probably do it, but unless you've got a good reason, it's much easier to just walk over and pick up the cup with your hand.:-)

Yours,

Russ Magee %-)

 

[simone monteleone]

Hi Russ,

:) +1 for your example.

I read the Django documentation and I find the Middleware chapter.

It's possible using the Middleware layer to intercept the HTTP requests.

The "script solution" remain the simplest solution.

My goal is the implementation of system, composed by two devices:

- a data logger

- a web service

The data logger have to send the registered data to the web server.

I think to use an https comunication between the two devices.

So on the same machine, where I install the web server,  I need another web server to handle the
data logger requests.

This is a possible solution.

SM

[Russell Keith-Magee]

On Tue, Feb 18, 2014 at 4:10 PM, simone monteleone <ares...@gmail.com> wrote:

Hi Russ,

:) +1 for your example.

I read the Django documentation and I find the Middleware chapter.

It's possible using the Middleware layer to intercept the HTTP requests.

The "script solution" remain the simplest solution.

My goal is the implementation of system, composed by two devices:

- a data logger

- a web service

The data logger have to send the registered data to the web server.

I think to use an https comunication between the two devices.

So on the same machine, where I install the web server,  I need another web server to handle the
data logger requests.

This is a possible solution.

Ok - this is why you need to provide all the details when you ask a question. What you've described here is a completely different problem to the one you've described so far, and one that *would* be well suited to receiving a file via URL, rather than using a script on the server.

So, yes - you write a view on the server that accepts a file upload, and when that file is uploaded, you read it and process it. The data logger itself can absolutely use requests to push this data; you could also use curl, or any other HTTP tool.

However, a bigger question -- how many "rows" of data will each CSV file contain?

If it's only one - perhaps you should consider using a HTTP POST, rather than sending a file with one line in it.

If it's a small number - maybe you should consider modifying your data logger so that it can send just one record at a time

If it's a *large* number of rows, consider the performance of your web server. Web servers are designed to respond quickly. They don't behave well with long lived processes. You may need to investigate handing the uploaded data using a background queue.

Yours,

Russ Magee %-)