Context

Github Copilot is an ML-powered autocompletion tool for developers (it uses OpenAI's Codex). It is currently has a 1-month free trial and is priced at $10/mo.

Amazon CodeWhisperer is a similar service. It's currently in private preview, so its quality and feature set may change before release.

Comparison

Each of these tests is conducted from a blank page in VSCode with a language selected (typically Javascript unless otherwise stated). I will include the prompt and the results that both Copilot and CodeWhisperer suggested, along with any interesting items to notice.

Basic Variables (Similar)

// Prompt
const a = 1;
const b = 2;
const sum =

// Copilot
const sum = a + b;

// CodeWhisperer
const sum = () => {
  return a + b;
}

Trigonometry (Same)

// Prompt
const x = Math.cos(angle) * radius;

// Both
const y = Math.sin(angle) * radius;

Factorial (Near Identical)

// Prompt
function factorial

// Copilot
function factorial(n) {
  if (n === 0) {
    return 1;
  }
  return n * factorial(n - 1);
}

// CodeWhisperer
function factorial(n) {
  if (n === 1) {
    return 1;
  }
  return n * factorial(n - 1);
}

Price of BTC

// Get price of BTC
function

// Copilot
function getPrice() {
  return new Promise((resolve, reject) => {
    request('https://api.coinmarketcap.com/v1/ticker/bitcoin/', (error, response, body) => {
      if (error) {
        reject(error);
      }
      resolve(body);
    });
  });
}

// CodeWhisperer
function getPrice(callback) {
  $.ajax({
    url: "https://api.coindesk.com/v1/bpi/currentprice.json",
    dataType: "json",
    success: function (data) {
      callback(data);
    }
  });
}

Copilot's 9 suggestions use request, $.getJSON, and even new XMLHttpRequest(); in the last one. 5 make use of promises. Its recommendations include 3 different APIs (coinmarketcap, coindesk, and coinbase).

CodeWhisperer's 5 suggestions use $.ajax in 4 of them (no promises) and fetch in 1 (with promises). They all use the coindesk API.

Price of BTC with async (Similar)

The above example discourages the use of promises by not starting the function with async.

// Get price of BTC
async function

// Copilot
async function getPrice() {
  const response = await fetch('https://api.coindesk.com/v1/bpi/currentprice.json');
  const data = await response.json();
  return data.bpi.USD.rate_float;
}

// CodeWhisperer
async function getPrice() {
  const response = await fetch('https://api.coindesk.com/v1/bpi/currentprice.json');
  const data = await response.json();
  return data;
}

It's impressive how similar these are. I appreciate that Copilot actual gets the raw price.

Send Email (Near Identical)

I've specifically added from, to the prompt because Copilot stops its autocomplete as soon as it comes across an email (potentially causing syntax errors). This is not equivalent to OpenAI Codex's behavior, which has no issue completing the from field (with an email such as no-reply@todo.com).

We'll use SendGrid specifically. When not specifying sendgrid, Copilot is a little more diverse in its recommendations, whereas CodeWhisperer will only use aws-sdk.

import * as sgMail from '@sendgrid/mail';
const SEND_GRID_API_KEY = process.env.SEND_GRID_API_KEY;

function sendEmail(from,

// Copilot
function sendEmail(from, to, subject, text) {
  sgMail.setApiKey(SEND_GRID_API_KEY);
  const msg = {
    to,
    from,
    subject,
    text,
  };
  sgMail.send(msg);
}

// CodeWhisperer
function sendEmail(from, to, subject, html) {
  sgMail.setApiKey(SEND_GRID_API_KEY);
  const msg = {
    to,
    from,
    subject,
    html,
  };
  sgMail.send(msg);
}

Naming Consistency (Same)

const people = [
  { theirNameIs: 'John', andTheirAgeShallBe: 20 },

// Both
  { theirNameIs: 'Jane', andTheirAgeShallBe: 21 },

Halting Problem

function run() {
  let a = 10;
  while (a > 0) {
    a++;
  }
  return a;
}
// Explain in a couple sentences if program above halts or not, and why.
// The program will

// Copilot
// The program will halt because a is greater than 0, but it will not halt
// because the while loop will continue to run.

// CodeWhisperer
// The program will halt at the first iteration.
// It will not halt at the second iteration.
// It will not halt at the third iteration.

// A bit ironic that CodeWhisperer will
// continue this "It will not halt at..." forever.

Copilot is close--the last half is right. CodeWhisperer doesn't know what I'm talking about.

Explain Code

function a(b) {
  return b.split(',').map(x => `<b>${x}</b>`).join(',');
}
// Explain what the above function does:
// The function

// Copilot
// The function splits the string into an array of strings,
// then maps each string to a new string that is wrapped in <b> tags.
// The result is then joined back into a string with commas.
// The result is then returned.

// CodeWhisperer
// The function takes a string as an argument and returns a string.
// The string is split by the comma and then each word is wrapped in a <b> tag.
// The result is then joined together with a comma and a space.
// The function returns the result.

I appreciate CodeWhisperer's extra detail over Copilot (it specifies that the argument is a string, what we split by, etc). However it does have a clear typo: "with a comma and a space.". It also seems to refer to comma in the singular tense both times.

Javascript to Python

# function a(b) {
#   return b.split(',').map(x => `<b>${x}</b>`).join(',');
# }
# Convert the function to python:
def

# Copilot
def a(b):
    return '<b>' + b.split(',').join('</b>, <b>') + '</b>'

# CodeWhisperer
def a(b):
    return [f'<b>{x}</b>' for x in b.split(',')]

Both seem to struggle. Copilot's doesn't even run in Python ('list' object has no attribute 'join'). CodeWhisperer's is better (it runs), but it forgets to join with commas at the end.

SQL in JavaScript

// Get the average join date of all users
// on the team "Team Coco" using a SQL query.
async function

// Copilot
async function getAverageJoinDate() {
  const query = `
    SELECT AVG(join_date)
    FROM users
    WHERE team_id = (
      SELECT id
      FROM teams
      WHERE name = 'Team Coco'
    )
  `;
  const result = await db.query(query);
  return result.rows[0].avg;
}

// CodeWhisperer
async function getAvgJoinDate() {
  const [rows] = await connection.execute(
    `SELECT AVG(joindate) AS avg_joindate
    FROM users
    WHERE teamname = 'Team Coco';`,
  );
  return rows;
}

Given CodeWhisperer is technically right (the best kind of right), I added an extra comment to see how it (and Copilot) would react (// Tables: "users" and "teams"):

// Get the average join date of all users
// on the team "Team Coco" using a SQL query.
// Tables: "users" and "teams"
async function

// Copilot
async function getAverageJoinDateOfTeamCoco() {
  const query = `
    SELECT AVG(users.join_date)
    FROM users
    JOIN teams ON users.team_id = teams.id
    WHERE teams.name = 'Team Coco'
  `;
  const result = await db.query(query);
  return result.rows[0].avg;
}

// CodeWhisperer
async function getAverageJoinDate() {
  const [results] = await client.query(`
    SELECT AVG(users.date_joined)
    FROM users
    JOIN teams
    ON users.team_id = teams.id
    WHERE teams.name = 'Team Coco';
  `);

  return results.rows[0].avg;
}

Both look good. Notable differences are function names, and CodeWhisperer's odd destructuring (const [results] instead of just const results).

Additional Differences

Speed

Copilot not only runs faster (0.5s-1s, sometimes up to ~3s), but automatically runs. CodeWhisperer requires you to hit Opt-C to autocomplete, and tends to load slower (usually 1-2s, sometimes up to ~5s).

Languages

Copilot has been trained on dozens of languages and theoretically has no language limitation (although it will assuredly perform better in some than others). CodeWhisperer seems to be limited to JavaScript, Python, and Java (it refuses to autocomplete if you have another language selected in VSCode).

Final Thoughts

I personally think Github Copilot is still further ahead, however:

• Amazon CodeWhisperer is not officially released.
• They are still surprisingly similar in many circumstances.
• Competition is good!

(But Copilot is a cooler name than CodeWhisperer.)