admin管理员组

文章数量:1025244

I am new to Python and web scraping. I am a beginner in programming and am still practicing. I am using Python and Selenium for web scraping. And using Chat GPT to help me. Keep in mind it is still a work in progress:)

I am trying to scrape the data from exhibition websites like (Heimtextil). My goal is to find and scrape the company's website links, as well as the addresses and contact information from the exhibitor pages. From that, the details get put into an Excel document.

I'm specifically having trouble with finding the class="a-link--no-focus" & herf="link".

Exhibitor List

Exhibitor Page

I can get the company name just fine, I just can't get the code to find/open the exhibitor pages from the exhibitor list.

Here is a picture of the end result and output on the terminal

End Result

I would really appreciate your help!

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import TimeoutException, NoSuchElementException

# Set up Selenium WebDriver
service = Service('/LOCATION/chromedriver')
driver = webdriver.Chrome(service=service)

try:
    # Navigate to the exhibitor search page
    url = '.html?country=AUT'
    driver.get(url)

    # Wait for the main container to load (use explicit wait)
    try:
        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, 'ex-exhibitor-search-result-item'))
        )
        print("Exhibitor containers loaded")
    except TimeoutException:
        print("Loading took too much time!")

    # Scroll down multiple times to load more elements if lazy loading is present
    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(5)  # Wait for new content to load after scrolling

    # Locate all exhibitor items
    exhibitors = driver.find_elements(By.CLASS_NAME, 'ex-exhibitor-search-result-item')
    print(f"Found {len(exhibitors)} exhibitors")

    # Extract data
    data = []
    base_url = ''  # Base URL for detail pages

    for exhibitor in exhibitors:
        # Scroll each exhibitor into view
        driver.execute_script("arguments[0].scrollIntoView();", exhibitor)
        time.sleep(1)  # Small delay to allow scrolling to complete

        # Get the company name
        try:
            company_name = exhibitor.find_element(By.CLASS_NAME, 'ex-exhibitor-search-result-item__headline').text.strip()
        except NoSuchElementException:
            company_name = 'N/A'
            print("Error getting company name")

        # Get the link to the exhibitor's detail page
        detail_page_url = 'N/A'
        try:
            detail_page_tag = exhibitor.find_element(By.CSS_SELECTOR, 'a.a-link--no-focus')
            detail_page_url = base_url + detail_page_tag.get_attribute('href')
            print(f"Detail page URL found: {detail_page_url}")
        except NoSuchElementException:
            print("No detail page link found for exhibitor")

        # Now navigate to the detail page if it exists
        website_url = 'N/A'
        if detail_page_url != 'N/A':
            driver.get(detail_page_url)
            time.sleep(5)  # Wait for the detail page to load

            # Try to locate the website URL in the detail page
            try:
                website_tag = driver.find_element(By.CSS_SELECTOR, 'a.ex-contact-box__website-link')
                website_url = website_tag.get_attribute('href')
                print(f"Found website URL: {website_url}")
            except NoSuchElementException:
                print("No website URL found on detail page")

            # Return to the main page
            driver.back()
            time.sleep(5)

        # Append extracted data to the list
        data.append({
            'Company': company_name,
            'Website': website_url,
        })

    # Convert to DataFrame
    df = pd.DataFrame(data)

    # Check if DataFrame contains the expected data
    print("DataFrame columns:", df.columns)
    print(df.head())  # Display first few rows to verify data

    # Save to Excel
    df.to_excel('heimtextil_exhibitors.xlsx', index=False)

finally:
    # Close the browser
    driver.quit()
    print("Browser closed")

I am new to Python and web scraping. I am a beginner in programming and am still practicing. I am using Python and Selenium for web scraping. And using Chat GPT to help me. Keep in mind it is still a work in progress:)

I am trying to scrape the data from exhibition websites like (Heimtextil). My goal is to find and scrape the company's website links, as well as the addresses and contact information from the exhibitor pages. From that, the details get put into an Excel document.

I'm specifically having trouble with finding the class="a-link--no-focus" & herf="link".

Exhibitor List

Exhibitor Page

I can get the company name just fine, I just can't get the code to find/open the exhibitor pages from the exhibitor list.

Here is a picture of the end result and output on the terminal

End Result

I would really appreciate your help!

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import TimeoutException, NoSuchElementException

# Set up Selenium WebDriver
service = Service('/LOCATION/chromedriver')
driver = webdriver.Chrome(service=service)

try:
    # Navigate to the exhibitor search page
    url = 'https://heimtextil.messefrankfurt/frankfurt/en/exhibitor-search.html?country=AUT'
    driver.get(url)

    # Wait for the main container to load (use explicit wait)
    try:
        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, 'ex-exhibitor-search-result-item'))
        )
        print("Exhibitor containers loaded")
    except TimeoutException:
        print("Loading took too much time!")

    # Scroll down multiple times to load more elements if lazy loading is present
    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(5)  # Wait for new content to load after scrolling

    # Locate all exhibitor items
    exhibitors = driver.find_elements(By.CLASS_NAME, 'ex-exhibitor-search-result-item')
    print(f"Found {len(exhibitors)} exhibitors")

    # Extract data
    data = []
    base_url = 'https://heimtextil.messefrankfurt'  # Base URL for detail pages

    for exhibitor in exhibitors:
        # Scroll each exhibitor into view
        driver.execute_script("arguments[0].scrollIntoView();", exhibitor)
        time.sleep(1)  # Small delay to allow scrolling to complete

        # Get the company name
        try:
            company_name = exhibitor.find_element(By.CLASS_NAME, 'ex-exhibitor-search-result-item__headline').text.strip()
        except NoSuchElementException:
            company_name = 'N/A'
            print("Error getting company name")

        # Get the link to the exhibitor's detail page
        detail_page_url = 'N/A'
        try:
            detail_page_tag = exhibitor.find_element(By.CSS_SELECTOR, 'a.a-link--no-focus')
            detail_page_url = base_url + detail_page_tag.get_attribute('href')
            print(f"Detail page URL found: {detail_page_url}")
        except NoSuchElementException:
            print("No detail page link found for exhibitor")

        # Now navigate to the detail page if it exists
        website_url = 'N/A'
        if detail_page_url != 'N/A':
            driver.get(detail_page_url)
            time.sleep(5)  # Wait for the detail page to load

            # Try to locate the website URL in the detail page
            try:
                website_tag = driver.find_element(By.CSS_SELECTOR, 'a.ex-contact-box__website-link')
                website_url = website_tag.get_attribute('href')
                print(f"Found website URL: {website_url}")
            except NoSuchElementException:
                print("No website URL found on detail page")

            # Return to the main page
            driver.back()
            time.sleep(5)

        # Append extracted data to the list
        data.append({
            'Company': company_name,
            'Website': website_url,
        })

    # Convert to DataFrame
    df = pd.DataFrame(data)

    # Check if DataFrame contains the expected data
    print("DataFrame columns:", df.columns)
    print(df.head())  # Display first few rows to verify data

    # Save to Excel
    df.to_excel('heimtextil_exhibitors.xlsx', index=False)

finally:
    # Close the browser
    driver.quit()
    print("Browser closed")
  • python
  • selenium-webdriver
  • web-scraping
  • hyperlink
Share Improve this question edited Nov 18, 2024 at 12:19 Uzi asked Nov 18, 2024 at 11:45 UziUzi 113 bronze badges 1
  • Each website will have their own class in their html and you cannot expect to get the data from the class class="a-link--no-focus" & herf="link". always. – Moses01 Commented Nov 18, 2024 at 12:24
Add a comment  | 

1 Answer 1

Reset to default 1

I have gone through the website you provided. All the mentioned companies website addresses are available in the html with class icon icon-news-before ex-contact-box__website-link

<a class="icon icon-news-before ex-contact-box__website-link" href="http://www.bandex" rel="noopener noreferrer" target="_blank"><span><strong>Our website</strong></span></a>

Here is the updated code

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import TimeoutException, NoSuchElementException

service = Service('/LOCATION/chromedriver') 
driver = webdriver.Chrome(service=service)

try:

    url = 'https://heimtextil.messefrankfurt/frankfurt/en/exhibitor-search.html?country=AUT'
    driver.get(url)

    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'ex-exhibitor-search-result-item'))
    )

    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(5)

    exhibitors = driver.find_elements(By.CLASS_NAME, 'ex-exhibitor-search-result-item')

    data = []
    base_url = 'https://heimtextil.messefrankfurt'  # Base URL for detail pages

    for exhibitor in exhibitors:
        driver.execute_script("arguments[0].scrollIntoView();", exhibitor)
        time.sleep(1)  # Small delay to allow scrolling to complete
        try:
            company_name = exhibitor.find_element(By.CLASS_NAME, 'ex-exhibitor-search-result-item__headline').text.strip()
        except NoSuchElementException:
            company_name = 'N/A'

        detail_page_url = 'N/A'
        try:
            detail_page_tag = exhibitor.find_element(By.CSS_SELECTOR, 'a.a-link--no-focus')
            detail_page_url = base_url + detail_page_tag.get_attribute('href')
        except NoSuchElementException:
            pass

        website_urls = []
        if detail_page_url != 'N/A':
            driver.get(detail_page_url)
            time.sleep(5)  

            # Find all 'a' elements with the class 'ex-contact-box__website-link'
            try:
                website_tags = driver.find_elements(By.CSS_SELECTOR, 'a.ex-contact-box__website-link')
                website_urls = [website_tag.get_attribute('href') for website_tag in website_tags]
            except NoSuchElementException:
                pass

            
            website_url = ', '.join(website_urls) if website_urls else 'N/A'


            driver.get(url)
            time.sleep(5)

        data.append({
            'Company': company_name,
            'Website': website_url,
        })

    df = pd.DataFrame(data)

    df.to_excel('heimtextil_exhibitors.xlsx', index=False)

    print("Data successfully saved to heimtextil_exhibitors.xlsx")

finally:
    driver.quit()
    print("Browser closed")

I am new to Python and web scraping. I am a beginner in programming and am still practicing. I am using Python and Selenium for web scraping. And using Chat GPT to help me. Keep in mind it is still a work in progress:)

I am trying to scrape the data from exhibition websites like (Heimtextil). My goal is to find and scrape the company's website links, as well as the addresses and contact information from the exhibitor pages. From that, the details get put into an Excel document.

I'm specifically having trouble with finding the class="a-link--no-focus" & herf="link".

Exhibitor List

Exhibitor Page

I can get the company name just fine, I just can't get the code to find/open the exhibitor pages from the exhibitor list.

Here is a picture of the end result and output on the terminal

End Result

I would really appreciate your help!

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import TimeoutException, NoSuchElementException

# Set up Selenium WebDriver
service = Service('/LOCATION/chromedriver')
driver = webdriver.Chrome(service=service)

try:
    # Navigate to the exhibitor search page
    url = '.html?country=AUT'
    driver.get(url)

    # Wait for the main container to load (use explicit wait)
    try:
        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, 'ex-exhibitor-search-result-item'))
        )
        print("Exhibitor containers loaded")
    except TimeoutException:
        print("Loading took too much time!")

    # Scroll down multiple times to load more elements if lazy loading is present
    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(5)  # Wait for new content to load after scrolling

    # Locate all exhibitor items
    exhibitors = driver.find_elements(By.CLASS_NAME, 'ex-exhibitor-search-result-item')
    print(f"Found {len(exhibitors)} exhibitors")

    # Extract data
    data = []
    base_url = ''  # Base URL for detail pages

    for exhibitor in exhibitors:
        # Scroll each exhibitor into view
        driver.execute_script("arguments[0].scrollIntoView();", exhibitor)
        time.sleep(1)  # Small delay to allow scrolling to complete

        # Get the company name
        try:
            company_name = exhibitor.find_element(By.CLASS_NAME, 'ex-exhibitor-search-result-item__headline').text.strip()
        except NoSuchElementException:
            company_name = 'N/A'
            print("Error getting company name")

        # Get the link to the exhibitor's detail page
        detail_page_url = 'N/A'
        try:
            detail_page_tag = exhibitor.find_element(By.CSS_SELECTOR, 'a.a-link--no-focus')
            detail_page_url = base_url + detail_page_tag.get_attribute('href')
            print(f"Detail page URL found: {detail_page_url}")
        except NoSuchElementException:
            print("No detail page link found for exhibitor")

        # Now navigate to the detail page if it exists
        website_url = 'N/A'
        if detail_page_url != 'N/A':
            driver.get(detail_page_url)
            time.sleep(5)  # Wait for the detail page to load

            # Try to locate the website URL in the detail page
            try:
                website_tag = driver.find_element(By.CSS_SELECTOR, 'a.ex-contact-box__website-link')
                website_url = website_tag.get_attribute('href')
                print(f"Found website URL: {website_url}")
            except NoSuchElementException:
                print("No website URL found on detail page")

            # Return to the main page
            driver.back()
            time.sleep(5)

        # Append extracted data to the list
        data.append({
            'Company': company_name,
            'Website': website_url,
        })

    # Convert to DataFrame
    df = pd.DataFrame(data)

    # Check if DataFrame contains the expected data
    print("DataFrame columns:", df.columns)
    print(df.head())  # Display first few rows to verify data

    # Save to Excel
    df.to_excel('heimtextil_exhibitors.xlsx', index=False)

finally:
    # Close the browser
    driver.quit()
    print("Browser closed")

I am new to Python and web scraping. I am a beginner in programming and am still practicing. I am using Python and Selenium for web scraping. And using Chat GPT to help me. Keep in mind it is still a work in progress:)

I am trying to scrape the data from exhibition websites like (Heimtextil). My goal is to find and scrape the company's website links, as well as the addresses and contact information from the exhibitor pages. From that, the details get put into an Excel document.

I'm specifically having trouble with finding the class="a-link--no-focus" & herf="link".

Exhibitor List

Exhibitor Page

I can get the company name just fine, I just can't get the code to find/open the exhibitor pages from the exhibitor list.

Here is a picture of the end result and output on the terminal

End Result

I would really appreciate your help!

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import TimeoutException, NoSuchElementException

# Set up Selenium WebDriver
service = Service('/LOCATION/chromedriver')
driver = webdriver.Chrome(service=service)

try:
    # Navigate to the exhibitor search page
    url = 'https://heimtextil.messefrankfurt/frankfurt/en/exhibitor-search.html?country=AUT'
    driver.get(url)

    # Wait for the main container to load (use explicit wait)
    try:
        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, 'ex-exhibitor-search-result-item'))
        )
        print("Exhibitor containers loaded")
    except TimeoutException:
        print("Loading took too much time!")

    # Scroll down multiple times to load more elements if lazy loading is present
    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(5)  # Wait for new content to load after scrolling

    # Locate all exhibitor items
    exhibitors = driver.find_elements(By.CLASS_NAME, 'ex-exhibitor-search-result-item')
    print(f"Found {len(exhibitors)} exhibitors")

    # Extract data
    data = []
    base_url = 'https://heimtextil.messefrankfurt'  # Base URL for detail pages

    for exhibitor in exhibitors:
        # Scroll each exhibitor into view
        driver.execute_script("arguments[0].scrollIntoView();", exhibitor)
        time.sleep(1)  # Small delay to allow scrolling to complete

        # Get the company name
        try:
            company_name = exhibitor.find_element(By.CLASS_NAME, 'ex-exhibitor-search-result-item__headline').text.strip()
        except NoSuchElementException:
            company_name = 'N/A'
            print("Error getting company name")

        # Get the link to the exhibitor's detail page
        detail_page_url = 'N/A'
        try:
            detail_page_tag = exhibitor.find_element(By.CSS_SELECTOR, 'a.a-link--no-focus')
            detail_page_url = base_url + detail_page_tag.get_attribute('href')
            print(f"Detail page URL found: {detail_page_url}")
        except NoSuchElementException:
            print("No detail page link found for exhibitor")

        # Now navigate to the detail page if it exists
        website_url = 'N/A'
        if detail_page_url != 'N/A':
            driver.get(detail_page_url)
            time.sleep(5)  # Wait for the detail page to load

            # Try to locate the website URL in the detail page
            try:
                website_tag = driver.find_element(By.CSS_SELECTOR, 'a.ex-contact-box__website-link')
                website_url = website_tag.get_attribute('href')
                print(f"Found website URL: {website_url}")
            except NoSuchElementException:
                print("No website URL found on detail page")

            # Return to the main page
            driver.back()
            time.sleep(5)

        # Append extracted data to the list
        data.append({
            'Company': company_name,
            'Website': website_url,
        })

    # Convert to DataFrame
    df = pd.DataFrame(data)

    # Check if DataFrame contains the expected data
    print("DataFrame columns:", df.columns)
    print(df.head())  # Display first few rows to verify data

    # Save to Excel
    df.to_excel('heimtextil_exhibitors.xlsx', index=False)

finally:
    # Close the browser
    driver.quit()
    print("Browser closed")
  • python
  • selenium-webdriver
  • web-scraping
  • hyperlink
Share Improve this question edited Nov 18, 2024 at 12:19 Uzi asked Nov 18, 2024 at 11:45 UziUzi 113 bronze badges 1
  • Each website will have their own class in their html and you cannot expect to get the data from the class class="a-link--no-focus" & herf="link". always. – Moses01 Commented Nov 18, 2024 at 12:24
Add a comment  | 

1 Answer 1

Reset to default 1

I have gone through the website you provided. All the mentioned companies website addresses are available in the html with class icon icon-news-before ex-contact-box__website-link

<a class="icon icon-news-before ex-contact-box__website-link" href="http://www.bandex" rel="noopener noreferrer" target="_blank"><span><strong>Our website</strong></span></a>

Here is the updated code

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import TimeoutException, NoSuchElementException

service = Service('/LOCATION/chromedriver') 
driver = webdriver.Chrome(service=service)

try:

    url = 'https://heimtextil.messefrankfurt/frankfurt/en/exhibitor-search.html?country=AUT'
    driver.get(url)

    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'ex-exhibitor-search-result-item'))
    )

    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(5)

    exhibitors = driver.find_elements(By.CLASS_NAME, 'ex-exhibitor-search-result-item')

    data = []
    base_url = 'https://heimtextil.messefrankfurt'  # Base URL for detail pages

    for exhibitor in exhibitors:
        driver.execute_script("arguments[0].scrollIntoView();", exhibitor)
        time.sleep(1)  # Small delay to allow scrolling to complete
        try:
            company_name = exhibitor.find_element(By.CLASS_NAME, 'ex-exhibitor-search-result-item__headline').text.strip()
        except NoSuchElementException:
            company_name = 'N/A'

        detail_page_url = 'N/A'
        try:
            detail_page_tag = exhibitor.find_element(By.CSS_SELECTOR, 'a.a-link--no-focus')
            detail_page_url = base_url + detail_page_tag.get_attribute('href')
        except NoSuchElementException:
            pass

        website_urls = []
        if detail_page_url != 'N/A':
            driver.get(detail_page_url)
            time.sleep(5)  

            # Find all 'a' elements with the class 'ex-contact-box__website-link'
            try:
                website_tags = driver.find_elements(By.CSS_SELECTOR, 'a.ex-contact-box__website-link')
                website_urls = [website_tag.get_attribute('href') for website_tag in website_tags]
            except NoSuchElementException:
                pass

            
            website_url = ', '.join(website_urls) if website_urls else 'N/A'


            driver.get(url)
            time.sleep(5)

        data.append({
            'Company': company_name,
            'Website': website_url,
        })

    df = pd.DataFrame(data)

    df.to_excel('heimtextil_exhibitors.xlsx', index=False)

    print("Data successfully saved to heimtextil_exhibitors.xlsx")

finally:
    driver.quit()
    print("Browser closed")

本文标签： selenium webdriverI am unable to find links from python scraping code for exhibition websitesStack Overflow

版权声明：本文标题：selenium webdriver - I am unable to find links from python scraping code for exhibition websites - Stack Overflow 内容由热心网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://it.en369.cn/questions/1745620409a2159537.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。