1

I am trying to run an automation script (python-selenium script) that scrapes data from websites after taking some input from the client side. I am trying to send the credentials using a form and process it then using shell_exec or exec or symfony/process library. So far nothing works for me.

This is my form -

@extends('layouts.app')
@section('content')
    <div class="container">
        @include('includes.messages')
        <h1>Create Scraper</h1>
        {!! Form::open(['action' => 'ScraperController@store', 'method' => 'POST', 'novalidate' => 'novalidate']) !!}
            <div class="form-group">
                {{ Form::label('url', 'Search URL') }}
                {{ Form::text('url', '', ['class' => 'form-control', 'placeholder' => 'Search URL']) }}
            </div>

            <div class="form-group">
                {{ Form::label('email', 'Email') }}
                {{ Form::email('email', '', ['class' => 'form-control', 'placeholder' => 'Email']) }}
            </div>

            <div class="form-group">
                {{ Form::label('password', 'Password') }}
                {{ Form::password('password', ['class' => 'form-control', 'placeholder' => 'Password']) }}
            </div>

            {{ Form::submit('Submit', ['class' => 'btn btn-primary']) }}
        {!! Form::close() !!}
    </div>
@endsection

This is my controller -

I have tried it with process library it gives nothing no error, no response executes the below code straight away but scraper run unsuccessful

use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;

public function store(Request $request) {
    $this->validate($request, [
        'url' => 'required',
        'email' => 'required|email:rfc,dns',
        'password' => 'required'
    ]);

    $process = new Process(['C:\xampp\htdocs\scraper\public\lkscraper\venv\Scripts\python', 'C:\xampp\htdocs\scraper\public\lkscraper\lk.py']);
    $process->run();
    if (!$process->isSuccessful()) {
        throw new ProcessFailedException($process);
    }
    echo $process->getOutput();        
}

I have also tried the following - I have made all the dependency global in my machine an run it. But again not successful in laravel. But the same shell_exec run fine and I get the desired output when run as single file outside the laravel project.

public function store(Request $request) {
    $this->validate($request, [
        'url' => 'required',
        'email' => 'required|email:rfc,dns',
        'password' => 'required'
    ]);

    $output = shell_exec("python C:\xampp\htdocs\scraper\public\lkscraper\lk.py");
    var_dump($output);       
}

When I run the following keeping inside a seperate php file it works fine. But inside laravel it is not working

$output = shell_exec("python C:\xampp\htdocs\scraper\public\lkscraper\lk.py");
var_dump($output);

Please help me solve this. What am I doing wrong! Thank you for your time

The Error I am getting:

string(1418) "Traceback (most recent call last):
  File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\common\service.py", line 72, in start
    self.process = subprocess.Popen(cmd, env=self.env,
  File "C:\Users\raman\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\raman\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\xampp\htdocs\scraper/lkscraper/lk.py", line 15, in 
    browser = webdriver.Chrome('driver/chromedriver.exe')
  File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
    self.service.start()
  File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
    raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
5
  • Check this symfony.com/doc/current/components/process.html Commented Jun 17, 2020 at 6:22
  • I have already tried enough! As I said the code is working using shell_exec function from separate php file. It is just inside laravel controller not working. Commented Jun 17, 2020 at 6:25
  • Whata about this? stackoverflow.com/questions/41020068/… Commented Jun 17, 2020 at 6:26
  • I have gone through the above link. That is why I am asking this question. Otherwise there is no point in asking the same again. Commented Jun 17, 2020 at 6:28
  • Hi, Same problem. Did you fix this problem? Commented Sep 26, 2022 at 14:25

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.