Lucas Simon's Blog

Amazon CodeWhisperer vs Github Copilot (Updated April 2023)

Amazon CodeWhisperer vs Github Copilot (Updated April 2023)

Comparing the results of Amazon's and Github's ML-powered autocompletion tools.

Lucas Simon's photo
Lucas Simon
·

10 min read

Table of contents

Context

Github Copilot is an ML-powered autocompletion tool for developers (it uses OpenAI's Codex). It is currently has a 1-month free trial and is priced at $10/mo.

Amazon CodeWhisperer is a similar service. It's currently in private preview, so its quality and feature set may change before release.

Comparison

Each of these tests is conducted from a blank page in VSCode with a language selected (typically Javascript unless otherwise stated). I will include the prompt and the results that both Copilot and CodeWhisperer suggested, along with any interesting items to notice.

Basic Variables (Similar)

// Prompt
const a = 1;
const b = 2;
const sum =

// Copilot
const sum = a + b;

// CodeWhisperer (July 2022)
const sum = () => {
  return a + b;
}

// CodeWhisperer (April 2023)
const sum = a + b;

Trigonometry (Same)

// Prompt
const x = Math.cos(angle) * radius;

// All
const y = Math.sin(angle) * radius;

Factorial (Near Identical)

// Prompt
function factorial

// Copilot
function factorial(n) {
  if (n === 0) {
    return 1;
  }
  return n * factorial(n - 1);
}

// CodeWhisperer (July 2022 and April 2023)
function factorial(n) {
  if (n === 1) {
    return 1;
  }
  return n * factorial(n - 1);
}

Price of BTC

// Get price of BTC
function

// Copilot (July 2022)
function getPrice() {
  return new Promise((resolve, reject) => {
    request('https://api.coinmarketcap.com/v1/ticker/bitcoin/', (error, response, body) => {
      if (error) {
        reject(error);
      }
      resolve(body);
    });
  });
}

// CodeWhisperer (July 2022)
function getPrice(callback) {
  $.ajax({
    url: "https://api.coindesk.com/v1/bpi/currentprice.json",
    dataType: "json",
    success: function (data) {
      callback(data);
    }
  });
}

// Copilot (April 2023)
function getPrice() {
    var url = "https://api.coinbase.com/v2/prices/BTC-USD/buy";
    var req = new XMLHttpRequest();
    req.open("GET", url, false);
    req.send(null);
    var data = JSON.parse(req.responseText);
    return data.data.amount;
}

// CodeWhisperer (April 2023)
function getBTCPrice() {
  $.getJSON('https://api.coindesk.com/v1/bpi/currentprice.json', function(data) {
    var price = data.bpi.USD.rate;
    $('#btc-price').text(price);
  });
}

Copilot July 2022's 9 suggestions use request, $.getJSON, and even new XMLHttpRequest(); in the last one. 5 make use of promises. Its recommendations include 3 different APIs (coinmarketcap, coindesk, and coinbase).

Copilot April 2023's first suggestion is clearly an initial different approach. It now uses UrlFetchApp.fetch (seems to be a Google thing), new XMLHttpRequest, and new Request. It recommends 4 different APIs (coinbase, coindesk, coinmarketcap, and bitstamp).

July 2022 CodeWhisperer's 5 suggestions use $.ajax in 4 of them (no promises) and fetch in 1 (with promises). They all use the coindesk API.

April 2023 CodeWhisisper's 5 suggestion use $ajax in 4 of them and $.getJSON is 1 (no promises at all). They all use the coindesk API. Interestingly, they all now try to extract/display the data in some simple way (as you can see above).

Price of BTC with async (Similar)

The above example discourages the use of promises by not starting the function with async.

// Get price of BTC
async function

// Copilot (July 2022 and April 2023)
async function getPrice() {
  const response = await fetch('https://api.coindesk.com/v1/bpi/currentprice.json');
  const data = await response.json();
  return data.bpi.USD.rate_float;
}

// CodeWhisperer (July 2022)
async function getPrice() {
  const response = await fetch('https://api.coindesk.com/v1/bpi/currentprice.json');
  const data = await response.json();
  return data;
}

// CodeWhisperer (April 2023)
async function getPrice() {
    const response = await fetch('https://api.coindesk.com/v1/bpi/currentprice.json');
    const data = await response.json();
    return data.bpi.USD.rate_float;
}

It's impressive how similar these are. I appreciate that Copilot actual gets the raw price.

CodeWhisperer April 2023 comes around and becomes identical to Copilot.

Send Email (Near Identical)

I've specifically added from, to the prompt because Copilot stops its autocomplete as soon as it comes across an email (potentially causing syntax errors). This is not equivalent to OpenAI Codex's behavior, which has no issue completing the from field (with an email such as no-reply@todo.com).

We'll use SendGrid specifically. When not specifying sendgrid, Copilot is a little more diverse in its recommendations, whereas CodeWhisperer will only use aws-sdk.

import * as sgMail from '@sendgrid/mail';
const SEND_GRID_API_KEY = process.env.SEND_GRID_API_KEY;

function sendEmail(from,

// Copilot July 2022, Copilot April 2023, CodeWhisperer April 2023
function sendEmail(from, to, subject, text) {
  sgMail.setApiKey(SEND_GRID_API_KEY);
  const msg = {
    to,
    from,
    subject,
    text,
  };
  sgMail.send(msg);
}

// CodeWhisperer (July 2022)
function sendEmail(from, to, subject, html) {
  sgMail.setApiKey(SEND_GRID_API_KEY);
  const msg = {
    to,
    from,
    subject,
    html,
  };
  sgMail.send(msg);
}

Only the old version of CodeWhisperer is a bit different.

Naming Consistency (Same)

const people = [
  { theirNameIs: 'John', andTheirAgeShallBe: 20 },

// All
  { theirNameIs: 'Jane', andTheirAgeShallBe: 21 },

Halting Problem

function run() {
  let a = 10;
  while (a > 0) {
    a++;
  }
  return a;
}
// Explain in a couple sentences if program above halts or not, and why.
// The program will

// Copilot (July 2022)
// The program will halt because a is greater than 0, but it will not halt
// because the while loop will continue to run.

// Copilot (April 2023)
// The program will not halt because the while loop will never be false. The
// value of a will always be greater than 0 because it is incremented by 1
// every time the loop runs.

// CodeWhisperer (July 2022)
// The program will halt at the first iteration.
// It will not halt at the second iteration.
// It will not halt at the third iteration.

// A bit ironic that CodeWhisperer will
// continue this "It will not halt at..." forever.

// CodeWhispeper (April 2023) (Doesn't work with //, only /*)
/*
  The program halts because the while loop is never executed.
  The while loop is never executed because the condition is never met.
*/

  • Copilot July 2022 is close--the last half is right.

  • Copilot April 2023 knocks it out of the ballpark.

  • CodeWhisperer April 2022 doesn't know what I'm talking about.

  • CodeWhisperer July 2023 is afraid to play the game with // but with /* it immediately gives a... more well-structured answer, but incorrect.

Explain Code

function a(b) {
  return b.split(',').map(x => `<b>${x}</b>`).join(',');
}
// Explain what the above function does:
// The function

// Copilot (July 2022)
// The function splits the string into an array of strings,
// then maps each string to a new string that is wrapped in <b> tags.
// The result is then joined back into a string with commas.
// The result is then returned.

// Copilot (April 2023)
// The function takes a string as input and returns a string with each word wrapped in <b> tags.

// Copilot (April 2023) with /*
/*
  1. Split the string by comma
  2. Map each element to a string with <b> tags
  3. Join the elements with comma
*/

// CodeWhisperer (July 2022)
// The function takes a string as an argument and returns a string.
// The string is split by the comma and then each word is wrapped in a <b> tag.
// The result is then joined together with a comma and a space.
// The function returns the result.

// CodeWhisperer (April 2023)
// The function takes a string and splits it into an array of strings.
// Then it maps each string to a string with a <b> tag around it.
// Then it joins the array of strings back together with a comma in between each string.
// The function returns the string.

I appreciate CodeWhisperer July 2022's extra detail over Copilot July 2022 (it specifies that the argument is a string, what we split by, etc). However it does have a clear typo: "with a comma and a space.". It also seems to refer to comma in the singular tense both times.

CodeWhisperer April 2023 doesn't seem to do particularly better.

Javascript to Python

# function a(b) {
#   return b.split(',').map(x => `<b>${x}</b>`).join(',');
# }
# Convert the function to python:
def

# Copilot (July 2022)
def a(b):
    return '<b>' + b.split(',').join('</b>, <b>') + '</b>'

# Copilot (April 2023)
def a(b):
    return ','.join([f'<b>{x}</b>' for x in b.split(',')])

# CodeWhisperer (July 2022)
def a(b):
    return [f'<b>{x}</b>' for x in b.split(',')]

# CodeWhisperer (April 2023)
def a(b):
    return b.split(",").map(lambda x: f"<b>{x}</b>").join(",")

  • Copilot July 2022's doesn't even run in Python ('list' object has no attribute 'join').

  • Copilot April 2023 is correct.

  • CodeWhisperer July 2022's is better (it runs), but it forgets to join with commas at the end.

  • CodeWhisperer April 2023 fails ('list' object has no attribute 'map')

SQL in JavaScript

// Get the average join date of all users
// on the team "Team Coco" using a SQL query.
async function

// Copilot (July 2022)
async function getAverageJoinDate() {
  const query = `
    SELECT AVG(join_date)
    FROM users
    WHERE team_id = (
      SELECT id
      FROM teams
      WHERE name = 'Team Coco'
    )
  `;
  const result = await db.query(query);
  return result.rows[0].avg;
}

// Copilot (April 2023)
// For some reason Copilot often refuses to do this one. If you go line by line, it quickly gets stuck. After a couple attempts, I managed to get the following (line-by-line). Seems to be cut off after "f"--in the past, when something is cut off, it's usually GitHub censoring something sensitive (like an email). No idea why it's triggered here.
async function getAverageJoinDate() {
  const res = await f
    .query(
      `SELECT AVG(join_date) FROM users WHERE team = "Team Coco"`
    )
    .run();
  return res[0]["AVG(join_date)"];
}

// CodeWhisperer (July 2022)
async function getAvgJoinDate() {
  const [rows] = await connection.execute(
    `SELECT AVG(joindate) AS avg_joindate
    FROM users
    WHERE teamname = 'Team Coco';`,
  );
  return rows;
}

// CodeWhisperer (April 2023), 1st suggestion
async function averageJoinDate() {
  // TODO
}

// CodeWhisperer (April 2023), 2nd suggestion
async function getAverageJoinDate() {
  // TODO: your code here
  const result = await client.query(`
    SELECT avg(joindate)
    FROM users
    WHERE teamname = 'Team Coco';
  `);

  return result.rows[0].avg;
}

Given both CodeWhisperers are technically right (the best kind of right), I added an extra comment to see how it (and Copilot) would react (// Tables: "users" and "teams"):

// Get the average join date of all users
// on the team "Team Coco" using a SQL query.
// Tables: "users" and "teams"
async function

// Copilot (July 2022)
async function getAverageJoinDateOfTeamCoco() {
  const query = `
    SELECT AVG(users.join_date)
    FROM users
    JOIN teams ON users.team_id = teams.id
    WHERE teams.name = 'Team Coco'
  `;
  const result = await db.query(query);
  return result.rows[0].avg;
}

// Copilot (April 2023)
// Again, can generate a query just fine but is struggling to actually execute...

// CodeWhisperer (July 2022)
async function getAverageJoinDate() {
  const [results] = await client.query(`
    SELECT AVG(users.date_joined)
    FROM users
    JOIN teams
    ON users.team_id = teams.id
    WHERE teams.name = 'Team Coco';
  `);

  return results.rows[0].avg;
}

// CodeWhisperer (April 2023), again 1st suggestion is just "todo" so 2nd suggestion:
async function averageJoinDate() {
  // YOUR CODE BELOW HERE //
  let average = await db.query(`
  SELECT avg(created_at) as average
  FROM users
  JOIN teams ON users.team_id = teams.id
  WHERE teams.name = 'Team Coco'
  `);

  return average.rows[0].average;


  // YOUR CODE ABOVE HERE //
}

Both Copilot and CodeWhisperer July 2022 good. Notable differences are function names, and CodeWhisperer's odd destructuring (const [results] instead of just const results). I honestly don't super like CodeWhisperer April 2023's solution (not as prettily formatted, includes weird "YOUR CODE BELOW HERE" comments, and names the variable average when it isn't actually the average yet).

Additional Differences

Speed

Copilot not only runs faster (0.5s-1s, sometimes up to ~3s), but automatically runs. CodeWhisperer requires you to hit Opt-C to autocomplete, and tends to load slower (usually 1-2s, sometimes up to ~5s).

CodeWhispeper April 2023 might be a bit faster now, but it still requires the Opt-C command (couldn't find a way to have it go automatically).

Languages

Copilot has been trained on dozens of languages and theoretically has no language limitation (although it will assuredly perform better in some than others). CodeWhisperer seems to be limited to JavaScript, Python, and Java (it refuses to autocomplete if you have another language selected in VSCode).

CodeWhisperer April 2023 now supports: Python, Java, JavaScript, TypeScript, C#, Rust, Go, Ruby, Scala, Kotlin, PHP, C, C++, Shell, SQL. However, it still refuses to autocomplete in VSCode languages that it does not support (Github Copilot will happily autocomplete in a plain text file).

Final Thoughts

I personally think Github Copilot is still further ahead, however:

  • Amazon CodeWhisperer is not officially released.

  • They are still surprisingly similar in many circumstances.

  • Competition is good!

(But Copilot is a cooler name than CodeWhisperer.)

JavaScriptgithub copilot#CodeWhispererAI