Django database problem

[Mikko Meronen]

Hi,

I'm quite new with django and python and I wish someone could help me.

I'm building a webpage where my python program/bot collects data from webpages.

Is there a way that I can use my python program to store the data to django's database?

At the moment when I run my program, I collect the data to separate sqlite database, but I don't know how to collect/connect it to django's sqlite database (or models).

- Mikko

[Anirudh Jain]

You will have to create a database on your system first (it is better to use mysql instead of sqlite) and then run commands :-

1. python manage.py makemigrations

2. python manage.py migrate

Also you will have to create forms (forms.ModelForm or forms.Forms) for taking input.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAGD0jjLbzK1XVVSUTD4Z66p_RUoK0pBW%2BrEYtVM_HwAd9d%3DwvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Mikhailo Keda]

You could generate models from sqlite database - https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
Than switch to PostgresQL

[Okware Aldo]

Mikko, 

Check out this link below, it will help you understand how to connect you Django project to any DB manager. 

https://docs.djangoproject.com/en/2.1/intro/tutorial02/

[Mikko Meronen]

Hi and thanks for your help.

I have tried postgress and now I have a problem using it. Here's my code following an error. I appreciate any help.

def do():

    x = 1

    while x == 1:

        create_table()

        print('finding data')

        find_data()

        time.sleep(120)

    

def create_table():

    c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT, published TEXT, UNIQUE(url))')

def find_data():

            r = requests.get(homepage)

            soup = BeautifulSoup(r.text, 'html.parser')

            results = soup.find_all('data')

            for result in results:

                title = result.find('h1').text

                url = result.find('a')['href']

                published = result.find('time')['datetime']

            

                c.execute("INSERT OR IGNORE INTO testi (url, title, published) VALUES (%s, %s, %s)",

                          ('url', 'title', 'published'))

                connection.commit()

                

do()

ERROR:

Traceback (most recent call last):

  File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43, in <module>

    do()

  File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21, in do

    find_data()

  File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39, in find_data

    ('url', 'title', 'published'))

psycopg2.ProgrammingError: syntax error at or near "OR"

LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES (...

-Mikko

2018-08-17 19:09 GMT+03:00 Mikhailo Keda <mri...@gmail.com>:

You could generate models from sqlite database - https://docs.djangoproject.com/en/2.1/howto/legacy-databases/
Than switch to PostgresQL

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com.

[Jani Tiainen]

Hi,

Instead of doing everything by hand I would recommend doing a custom management command in Django.

Because management commands live inside Django ecosystem you get full benefits of Django. Migrations to setup database tables from model definitions. Models and ORM to interact with database. (Model)forms to validate your incoming data etc.

Also I really suggest that you do the official tutorial from the docs to get hang of basic django concepts.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAGD0jjJof7v9nMPF593DJiQHRv6z_%3DmQ_ZtQWob47PBT1gvL9Q%40mail.gmail.com.

[Mikko Meronen]

Hi,

Thanks for advice ^^

I have one new concern. Is there a limit for for Django model IDs?

Now I have automated the data insert with python code loop, but the ID-number is not consecutive. It jumps for example from ...56 to 571... and from ...587 to 3763... and so forth.

Do you know why it is doing that?

I'm not using django custom command (if that makes the difference or can be used for my case, I will get familiar with it a bit later), I'm just inserting the data to django's database with separate python code.

-Mikko 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHn91oex8cAE1yFshSABt9QZpdiiRVtALtLx0AWDNU84w0%2Bqvw%40mail.gmail.com.

[Jason]

yeah, there is a limit, and its set by the data type you use for the id field.  by default, id is set to IntegerField which has a maximum value of 2147483647 

sounds like the issue with the ids jumping around like that is with your management command, not django or the db.  

[Andréas Kühne]

The management command could be wrapped within a transaction, this would then in the database create a lot of models, but if the transaction fails - the rollback will not reset the database id.

Regards,

Andréas

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAGD0jjKcf%2BMDDjoB83fnBsGiRD16vvHe-uLAcEW6XZhU31UDug%40mail.gmail.com.

[Jason]

oh, good catch.  I didn't think of that, but you're right.

https://stackoverflow.com/questions/449346/mysql-auto-increment-does-not-rollback

[Andréas Kühne]

I have looked at this several times myself. :-)

Regards,

Andréas

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/81f734c8-3140-4d1e-ae41-3e20bf59d9e1%40googlegroups.com.

[Mikko Meronen]

Yes it seems that I can influence to the 'jumps' with my management command. First my bot went through tens of items per loop, however when I changed the bot to check only 2 items per loop, the 'jumps' got smaller. So even though the table rows do not accrue (nothing is added) when no new items are found in a loop, the ID numbers still accrue behind the scenes in every loop.

So I just gotta figure out if there are any other way than INSERT OR IGNORE command, or just decrease the sleep time of the bot and keep the checked item numbers low per a loop.

I actually tried ROLLBACK and some other commands as well, but they killed the loop.

Thanks for the help ^^

-Mikko

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCfpMr2EL5GQY83TyFb7x8z9BwDAEAhLE_rLX3uTs02oag%40mail.gmail.com.