parse xml from url to database
19 views
Skip to first unread message
Nuno Vieira
Jun 12, 2019, 4:22:01 PM6/12/19
to Django users
Hi there.
I´m having some problems on parsing a real estate xml to database.
The user profile have url field to import properties automatically from xml feed (url). The user can insert mannually or automatically, if the user insert the url feed then it should be able to get the objects and save them to mysql database automatically.
The function is not working and i do not know why, gives no error but don´t populate the database.
Models.py:
class Listing(models.Model):
'''Create table listing in db '''
realtor = models.ForeignKey(User, on_delete=models.CASCADE)
title = models.CharField(max_length=200)
address = models.CharField(max_length=200)
city = models.CharField(max_length=100)
state = models.CharField(max_length=100)
zipcode = models.CharField(max_length=20)
longitude = models.DecimalField(max_digits=9, decimal_places=6,default=0)
latitude = models.DecimalField(max_digits=9, decimal_places=6, default=0)
description = models.TextField(blank=True)
price = models.IntegerField()
bedrooms = models.IntegerField()
bathrooms = models.DecimalField(max_digits=2, decimal_places=1)
garage = models.IntegerField(default=0)
sqmt = models.IntegerField(blank=True)
lot_size = models.DecimalField(max_digits=5, decimal_places=1,blank=True)
On_Processing = 'On Processing'
AMais = 'A+'
A = 'A'
B = 'B'
BMenos = 'B-'
C = 'C'
D = 'D'
E = 'E'
F = 'F'
ENERGY_RATE_CHOICES = [
(On_Processing, 'On Processing'),
(AMais, 'A+'),
(A, 'A'),
(B, 'B'),
(BMenos, 'B-'),
(C, 'C'),
(D, 'D'),
(E, 'E'),
(F, 'F'),
]
energy_rate = models.CharField(
max_length=12,
choices=ENERGY_RATE_CHOICES,
default=On_Processing,
)
photo_main = models.ImageField(upload_to='photos/%Y/%m/%d/', default='default.jpg')
photo_main_url = models.URLField(blank = True)
is_published = models.BooleanField(default=True)
list_date = models.DateTimeField(default=datetime.now, blank=True)
video_url = models.CharField(max_length=100, blank=True)
virtual_tour = models.URLField(blank=True)
visit_num = models.PositiveIntegerField(default=0)
def __str__(self):
return self.title
def get_absolute_url(self):
return reverse('listing-detail', kwargs={'pk': self.pk})
Function on xml_parse.py:
...
import xml.etree.ElementTree as ET
import requests
from users.models import Profile
def xml_parse(self):
prof_list = Profile.objects.all()
for prof in prof_list:
if prof.xml_ky is not None:
url = xml_ky
response = request.get(url)
with open('xml/%s' % self.id + '/feed.xml', 'wb') as file:
file.write(response.content)
tree = ET.parse('xml/%s' % self.id + '/feed.xml')
root = tree.getroot()
for props in root.iter('property'):
listing, created = Listing.object.create(
title = props.find('type').text + ' ' + props.find('town').text,
address = props.find('location_detail').text,
city = props.find('town').text,
state = props.find('province').text,
zipcode = props.find('zipcode').text,
loc = props.find('location'),
longitude = loc.find('longitude').text,
latitude = loc.find('latitude').text,
desc = props.find('desc'),
description = desc.find('en').text,
price = props.find('price').text,
bedrooms = props.find('beds').text,
bathrooms = props.find('baths').text,
garage = props.find('garage').text,
area = props.find('surface_area'),
sqmt = area.find('built').text,
energy_rate = props.find('energy_rate').text,
realtor = self.request.user,
img = props.find('images'),
image = img.get('id=1'),
image_url = image.find('url').text,
photo_main_url = image_url,
video_url = props.find('video_url').text,
virtual_tour = props.find('virtual_tour').text,
)
if created:
return listing.save()
If you could help me and point me in the right direction it would be great.
Thanks and sorry for the black background!
Ewen Le Bihan
Jun 12, 2019, 11:04:33 PM6/12/19
to Django users
Hi, where does prof.xml_ky comes from ?
Taylor Hughes-Scott
Jun 13, 2019, 1:17:48 AM6/13/19
to django...@googlegroups.com
Are you able to provide an example url or xml file. That would help to figure out why no listings are created.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20aa727c-d1e8-4d5f-a17d-ac17947ba5f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nuno Vieira
Jun 13, 2019, 8:53:48 AM6/13/19
to Django users
Thanks for the reply.
XML Example:
<?xml version="1.0" encoding="UTF-8"?>
<root>
<kyero>
<feed_version>3</feed_version>
</kyero>
<property>
<id>2150</id>
<date>2018-06-27 11:18:37</date>
<ref>BC10002</ref>
<price>1599900</price>
<currency>EUR</currency>
<price_freq>sale</price_freq>
<part_ownership>0</part_ownership>
<leasehold>0</leasehold>
<new_build>0</new_build>
<surface_area>
<built>6000</built>
</surface_area>
<type>Villa</type>
<location>
<latitude>37.134720</latitude>
<longitude>-8.020436</longitude>
</location>
<location_detail>optional location detail</location_detail>
<town>Loulé</town>
<province>Algarve</province>
<country>Portugal</country>
<beds>5</beds>
<baths>5</baths>
<pool>1</pool>
<images>
<image id="1">
</image>
<image id="2">
</image>
<image id="3">
</image>
<image id="4">
</image>
<image id="5">
</image>
</images>
<features>
<feature/>
</features>
<url>
</url>
<desc>
<en>This truly stunning Private modern 5 bedrooms villa.
Set in 6,000m2 accessed by two security gates.
Located just below the old market town of Loule and only 8 minute drive to Vilasol and Vilamoura and everything you will need for luxury living .
10 minutes drive of the villa you will find championship golf courses, numerous restaurants, golden beaches, sports and leisure facilities. The famous water park is within 5 min drive .
Underfloor heating
Surround sound all rooms
Central vac
Solor power
Outside kitchen
Sold fully furnished
Brand new borehole
Alarm
Intercom
Central shutters full lock down
Pool
Water softer
Fast internet</en>
</desc>
</property>
<property>
<id>2777</id>
<date>2018-06-20 14:12:13</date>
<ref>BC10004</ref>
<price>690000</price>
<currency>EUR</currency>
<price_freq>sale</price_freq>
<part_ownership>0</part_ownership>
<leasehold>0</leasehold>
<new_build>0</new_build>
<surface_area>
<built>250</built>
</surface_area>
<type>Villa</type>
<location>
<latitude>37.134720</latitude>
<longitude>-8.020436</longitude>
</location>
<location_detail>optional location detail</location_detail>
<town>São Brás de Alportel</town>
<province>Algarve</province>
<country>Portugal</country>
<beds>4</beds>
<baths>4</baths>
<pool>1</pool>
<images>
<image id="1">
</image>
<image id="2">
</image>
<image id="3">
</image>
<image id="4">
</image>
<image id="5">
</image>
</images>
<features>
<feature/>
</features>
<url>
</url>
<desc>
<en>Villa situated in São Brás de Alportel, 20 minutes from the beach and Faro airport.
Isolated, with pool and alarm, 4 bedrooms (2 en suite). A privileged view over the countryside and town. A place for lovers of nature and tranquility, where family days are very well enjoyed both outdoors and in the games room that the house has. An excellent investment both for fixed address and for holidays.
</en>
</desc>
</property>
<property>
<id>2779</id>
<date>2018-06-20 14:15:54</date>
<ref>BC10001</ref>
<price>649000</price>
<currency>EUR</currency>
<price_freq>sale</price_freq>
<part_ownership>0</part_ownership>
<leasehold>0</leasehold>
<new_build>0</new_build>
<surface_area>
<built>2924</built>
</surface_area>
<type>Villa</type>
<location>
<latitude>37.134720</latitude>
<longitude>-8.020436</longitude>
</location>
<location_detail>optional location detail</location_detail>
<town>São Brás de Alportel</town>
<province>Algarve</province>
<country>Portugal</country>
<beds>4</beds>
<baths>3</baths>
<pool>1</pool>
<images>
<image id="1">
</image>
<image id="2">
</image>
<image id="3">
</image>
<image id="4">
</image>
<image id="5">
</image>
</images>
<features>
<feature/>
</features>
<url>
</url>
<desc>
<en>A perfect villa for a family, located in São Brás de Alportel, a few kilometers from the beach, the airport and the city center. In a place where you can enjoy the tranquility and nature. It has excellent finishings, where we would like to highlight the skylight in the living room, allowing the sun light to go inside directly from the roof.
It has swimming pool, 4 bedrooms (one of them as a suite), 3 toilets in total, 2 covered garages, two entrances for cars with automatic gates, barbecue, alarm, air-conditioning, stove and many other functionalities.
A fantastic opportunity not to be missed!</en>
</desc>
</property>
</root>
Nuno Vieira
Jun 13, 2019, 12:50:22 PM6/13/19
to Django users
Hi Ewen,
thanks for the reply.
I have extended user model with a profile model and a new field (xml_ky). The goal is to verify each user profile and if they have an xml url on field xml_ky, than populate the database with the properties located on that url.
Thanks for your help.
Reply all
Reply to author
Forward
0 new messages