Monitoring Laravel background queues in realtime

When building large applications, making it scale is usually a concern. Stats like how long it takes for the page to load is usually very important. Thus, doing things like processing large images, sending emails and SMS can be pushed to a background queue and processed at a later time.

However, because queues work in the background, they can fail sometimes. It may then be necessary to be able to monitor background queues.

In this article, we will consider ways to monitor Laravel’s background queues in realtime. We will assume we created an application that sends emails. The emails are to be queued in the background and sent later. We will then have a report page with the emails that have been sent and those that haven’t.

This is a screen recording of what we will be building:

Tutorial requirements

To follow along in this tutorial, we would need the following things:

• PHP 7.0+ installed on your machine.
• Laravel CLI installed on your machine.
• Composer installed on your machine.
• Knowledge of PHP and Laravel.
• Node.js and NPM installed on your machine.
• Basic knowledge of Vue.js and JavaScript.
• A Pusher application. Create a free sandbox Pusher account or sign in.
• A Mailtrap account to test emails sent. Create one here.

Once you have these requirements ready, let’s start.

Setting up Your Laravel application

Open the terminal and run the command below to create a Laravel application:

    $ laravel new app_name

Setting up a database connection and migration

When installation is complete, we can move on to setting up the database. Open the .env file and replace the configuration below:

bash
1DB_CONNECTION=mysql
2    DB_HOST=127.0.0.1
3    DB_PORT=3306
4    DB_DATABASE=homestead
5    DB_USERNAME=homestead
6    DB_PASSWORD=secret
Copy

with:

    DB_CONNECTION=sqlite

This will set SQLite as our default database connection (you can use MySQL or any other database connection you want).

In the terminal cd to the root directory of your project. Run the command below to create the SQLite database file:

    $ touch database/database.sqlite

The command above will create an empty file that will be used by SQLite. Run the command below to create a migration:

    $ php artisan make:migration create_queued_emails_table

Open up the migration file that was just created by the command above and replace the up method with the code below:

php
1public function up()
2    {
3        Schema::create('queued_emails', function (Blueprint $table) {
4            $table->increments('id');
5            $table->string('email');
6            $table->string('description');
7            $table->boolean('run')->default(false);
8            $table->timestamps();
9        });
10    }
Copy

Now run the command below to migrate our database:

    $ php artisan migrate

Setting up Mailtrap for email testing

Open your .env file and enter the keys you got from the Mailtrap dashboard. The relevant keys are listed below:

bash
1MAIL_DRIVER=smtp
2    MAIL_HOST=smtp.mailtrap.io
3    MAIL_PORT=2525
4    MAIL_USERNAME=null
5    MAIL_PASSWORD=null
6    MAIL_ENCRYPTION=null
7    MAIL_FROM="john@doe.com"
8    MAIL_NAME="John Doe"
Copy

Now when emails are sent, the emails will be visible in the Mailtrap inbox.

Setting up authentication

The next thing we need to do is set up authentication. Open your terminal and enter the command below:

    $ php artisan make:auth

This will generate an authentication scaffold. That is all that you need to do regarding authentication.

Configuring Pusher

Replace the PUSHER_* keys in the .env file with the correct keys you got from your Pusher dashboard:

bash
1PUSHER_APP_ID="PUSHER_APP_ID"
2    PUSHER_APP_KEY="PUSHER_APP_KEY"
3    PUSHER_APP_SECRET="PUSHER_APP_SECRET"
Copy

Open the terminal and enter the command below to install the Pusher PHP SDK:

    $ composer require pusher/pusher-php-server "~3.0"

After installation is complete, open the config/broadcasting.php file and scroll to the pusher section. Replace the options key with the following:

php
1'options' => [
2        'encrypt' => true,
3        'cluster' => 'PUSHER_APP_CLUSTER'
4    ],
Copy

Configuring other miscellaneous things

Open the .env file and change the BROADCAST_DRIVER to pusher, and the QUEUE_DRIVER to database. To make sure we have the tables necessary to use database as our QUEUE_DRIVER run the command below to generate the database migration:

    $ php artisan queue:table

Then run the migrate command to migrate the database:

    $ php artisan migrate

This will create the database table required to use our database as a queue driver.

💡 In a production environment, it is better to use an in-memory storage like Redis or Memcached as the queue driver. In-memory storage is faster and thus has better performance than using a relational database.

Building the backend of our application

Now let’s create the backend of our application. Run the command below in your terminal:

    $ php artisan make:model QueuedEmails

This will create a new model in the app directory. Open the file and replace the contents with the following:

php
1<?php
2    
3    namespace App;
4    
5    use Illuminate\Database\Eloquent\Model;
6    
7    class QueuedEmails extends Model
8    {
9        protected $fillable = ['description', 'run', 'email'];
10        protected $casts = ['run' => "boolean"];
11    }
Copy

In the code above, we define the fillable property of the class. This will prevent a mass assignment exception error when we try to create a new entry to the database. We also specify a casts property which will instruct Eloquent to typecast attributes to data types.

Next, open the HomeController and and replace the contents with the code below:

php
1<?php
2    namespace App\Http\Controllers;
3    
4    use Mail;
5    use App\QueuedEmails;
6    use App\Mail\SimulateMail;
7    use Faker\Factory as Faker;
8    
9    class HomeController extends Controller
10    {
11        /**
12         * Create a new controller instance.
13         *
14         * @return void
15         */
16        public function __construct()
17        {
18            $this->middleware('auth');
19    
20            $this->faker = Faker::create();
21        }
22    
23        /**
24         * Show the application dashboard.
25         *
26         * @return \Illuminate\Http\Response
27         */
28        public function index()
29        {
30            return view('home', ['jobs' => $this->jobs()]);
31        }
32        
33        /**
34         * Return all the jobs.
35         *
36         * @return array
37         */
38        public function jobs()
39        {
40            return QueuedEmails::orderBy('created_at', 'DESC')->get()->toArray();
41        }
42        
43        /**
44         * Simulate sending the email.
45         *
46         * @return mixed
47         */
48        public function simulate()
49        {
50            $email = $this->faker->email;
51            
52            Mail::to($email)->send(
53                new SimulateMail([
54                    "email" => $email,
55                    "description" => $this->faker->sentence()
56                ])
57            );
58            
59            return redirect()->route('home');
60        }
61    }
Copy

In the controller above, we have 4 methods that are mostly self-explanatory. In the class we use the Faker library which helps us generate random fake values. In the simulate method, we are using the faker library to generate a fake email address and description. We instantiate a SimulateMail mailable.

Open the terminal and enter the command below:

    $ php artisan make:mail SimulateMail

Open the SimulateMail class and enter the code below:

php
1<?php
2    namespace App\Mail;
3    
4    use App\QueuedEmails;
5    use Illuminate\Bus\Queueable;
6    use Illuminate\Mail\Mailable;
7    use Illuminate\Queue\SerializesModels;
8    use App\Events\{EmailQueued, EmailSent};
9    use Illuminate\Contracts\Queue\ShouldQueue;
10    use Illuminate\Contracts\Queue\Factory as Queue;
11    use Illuminate\Contracts\Mail\Mailer as MailerContract;
12    
13    class SimulateMail extends Mailable implements ShouldQueue
14    {
15        use Queueable, SerializesModels;
16        
17        protected $mail;
18    
19        /**
20         * Create a new message instance.
21         *
22         * @return void
23         */
24        public function __construct(array $mail)
25        {
26            $this->mail = QueuedEmails::create($mail);
27        }
28        
29        /**
30         * Build the message.
31         *
32         * @return $this
33         */
34        public function build()
35        {
36            return $this->subject("Queuer: Welcome to queuer")->view('email.welcome');
37        }
38    
39        /**
40         * Send the mail
41         */
42        public function send(MailerContract $mailer)
43        {
44            $this->mail->update(['run' => true]);
45            
46            event(new EmailSent($this->mail));
47    
48            parent::send($mailer);
49        }
50        
51        /**
52         * Queue the email
53         */
54        public function queue(Queue $queue)
55        {
56            event(new EmailQueued($this->mail));
57    
58            return parent::queue($queue);
59        }
60    }
Copy

PRO TIP: By implementing the ShouldQueue interface, we are telling Laravel that the email should be queued and not sent immediately.**

In the class above, we have a constructor that creates a new entry into the queued_emails table. In the build method, we build the mail message we are going to be sending.

In the send method, we mark the queued_emails entry’s run column to true. We also fire an event called EmailSent. In the queue method, we also trigger an event called EmailQueued.

Let’s create the events we triggered in the methods above. In your terminal run the command below:

php
1$ php artisan make:event EmailSent
2    $ php artisan make:event EmailQueued
Copy

In the EmailSent event class, paste the following code:

php
1<?php
2    namespace App\Events;
3    
4    use App\QueuedEmails;
5    use Illuminate\Broadcasting\Channel;
6    use Illuminate\Queue\SerializesModels;
7    use Illuminate\Foundation\Events\Dispatchable;
8    use Illuminate\Broadcasting\InteractsWithSockets;
9    use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
10    
11    class EmailSent implements ShouldBroadcast
12    {
13        use Dispatchable, InteractsWithSockets, SerializesModels;
14    
15        public $mail;
16    
17        public function __construct($mail)
18        {
19            $this->mail = $mail;
20        }
21        
22        public function broadcastOn()
23        {
24            return new Channel('email-queue');
25        }
26        
27        public function broadcastAs()
28        {
29            return 'sent';
30        }
31    }
Copy

In the code above, we just use Broadcasting in Laravel to send some data to Pusher.

Open the EmailQueued event class and paste the code below:

php
1<?php
2    
3    namespace App\Events;
4    
5    use App\QueuedEmails;
6    use Illuminate\Broadcasting\Channel;
7    use Illuminate\Queue\SerializesModels;
8    use Illuminate\Foundation\Events\Dispatchable;
9    use Illuminate\Broadcasting\InteractsWithSockets;
10    use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
11    
12    class EmailQueued implements ShouldBroadcast
13    {
14        use Dispatchable, InteractsWithSockets, SerializesModels;
15        
16        public $mail;
17    
18        public function __construct($mail)
19        {
20            $this->mail = $mail;
21        }
22        
23        public function broadcastOn()
24        {
25            return new Channel('email-queue');
26        }
27        
28        public function broadcastAs()
29        {
30            return 'add';
31        }
32    }
Copy

This class is almost the same as the EmailSent event class. The minor difference is the broadcastAs method. It returns a different alias to broadcast the event as.

Finally, open the routes file routes/web.php and replace the code with this:

php
1Auth::routes();
2    Route::name('jobs')->get('/jobs', 'HomeController@jobs');
3    Route::name('simulate')->get('/simulate', 'HomeController@simulate');
4    Route::name('home')->get('/home', 'HomeController@index');
5    Route::view('/', 'welcome');
Copy

Great! Now let’s move on to the frontend of the application.

Building the frontend of our application

Now that we have set up most of the backend, we will create the frontend of the application. Open the resources/views/home.blade.php file and replace the code with the following:

php
1@extends('layouts.app')
2    
3    @section('content')
4    <div class="container">
5        <div class="row">
6            <div class="col-md-12">
7                <div class="panel panel-default">
8                    <div class="panel-heading clearfix">
9                        <span class="pull-left">Queue Reports</span>
10                        <a href="{{ route('simulate') }}" class="btn btn-sm btn-primary pull-right">Simulate</a>
11                    </div>
12                    <div class="panel-body">
13                        <jobs :jobs='@json($jobs)'></jobs></jobs>
14                    </div>
15                </div>
16            </div>
17        </div>
18    </div>
19    @endsection
Copy

The noteworthy aspect of the code above is the jobs tag. This is a reference to the Vue component we will create next. We also have a “Simulate” button that leads to a /simulate route. This route simulates queuing an email to be sent.

Open your terminal and type in the command below:

    $ npm install --save laravel-echo pusher-js

This will install Laravel Echo and the Pusher JS SDK. When the installation is complete, run the command below to install the other NPM dependencies:

    $ npm install

Building our Vue component

Let’s build the jobs Vue component we referenced earlier. Open the resources/assets/js/app.js file and replace the code below:

    Vue.component('example', require('./components/ExampleComponent.vue'));

with:

    Vue.component('jobs', require('./components/JobsComponent.vue'));

Now create a new JobsComponent.vue file in the resources/assets/js/components/ directory. In the file, paste in the following code:

javascript
1<template>
2        <table class="table">
3            <tbody>
4                <tr v-for="(job, index) in allJobs" :key="index" v-bind:class="{success: job.run, danger: !job.run}">
5                    <td width="80%">{{ job.description }}</td>
6                    <td>{{ job.created_at }}</td>
7                </tr>
8            </tbody>
9        </table>
10    </template>
11    
12    <script>
13    export default {
14        props: ['jobs'],
15        data() {
16            return {allJobs: this.jobs}
17        },
18        created() {
19            let vm = this
20            vm.refreshAllJobs = (e) => axios.get('/jobs').then((e) => (vm.allJobs = e.data))
21            Echo.channel('email-queue')
22                .listen('.add', (e)  => vm.refreshAllJobs(e))
23                .listen('.sent', (e) => vm.refreshAllJobs(e))
24        }
25    }
26    </script>
Copy

In the Vue component above, we have defined a template. In there, we loop through the jobs array and list each job’s description and timestamp.

In the created method of the Vue component script, we have a refreshAllJobs function that uses Axios (a HTTP request library built-in Laravel by default) to make a request to the /jobs route. We then assign the response to the allJobs property.

In the same method, we use Laravel Echo to listen to a Pusher channel and wait for an event to be triggered. Whenever the events .add and .sent are triggered, we call the refreshAllJobs method.

**PRO TIP: **The event names have a dot before them because, in Laravel, whenever you use the broadcastAs method to define an alias you need to add the dot. Without the dot your event will not be caught by the listener. If you do not provide an alias, Laravel will use the namespace + class as the name of the broadcast event.

Open the resources/assets/js/bootstrap.js file. At the bottom of the file, add the following code:

javascript
1import Echo from 'laravel-echo'
2    
3    window.Pusher = require('pusher-js');
4    
5    window.Echo = new Echo({
6        broadcaster: 'pusher',
7        key: 'PUSHER_APP_KEY',
8        encrypt: true,
9        cluster: 'PUSHER_APP_CLUSTER'
10    });
Copy

IMPORTANT: Make sure you replace the PUSHER_APP_KEY and PUSHER_APP_CLUSTER with your Pusher application key and cluster.

Finally, run the command below to build your assets:

    $ npm run dev

Testing our application

After the build is complete, start a PHP server if you have not already by running the command below:

    $ php artisan serve

This will create a PHP server so we can preview our application. The URL will be provided on the terminal but the default is http://127.0.0.1:8000.

When you see the Laravel homepage, create a new account using the ”Register” link on the top right corner of the page. Now click the “Simulate” button and you should see a new queued email entry.

Now we will manually execute the processes on our queue using the queue:listen artisan command. Open a new terminal window and run the command below:

    $ php artisan queue:listen

This should start executing any queues it sees. As long as the terminal is open and the queue:listen command is running, when you click the “Simulate” button the queue will run immediately. If you kill the queue:listen command, the queue entries will remain there and not be triggered.

PRO TIP: In a production environment, you cannot keep queue:listen running and you might need a worker running on a background proces. You can read more about how you can do that here.

Conclusion

In this article, we have been able to create a realtime Laravel queue monitor using Pusher and Vue. Having queues that you can track and quantify can be useful. Hopefully, you picked something from this article. If you have any questions or feedback, feel free to ask in the comments section.

The source code is available on GitHub.