Scott Giminiani

Introducing ConJob

Mon, 30 Jan 2023 08:00:00 GMT

What is ConJob?

ConJob is a service for running “job” containers, with a focus on CI use cases.

For a container to be considered a job, it should have 3 defining properties:

it’s ephemeral (exits of its own accord in a reasonable amount of time)
it accepts input
it returns any stdout and stderr output to the caller

The idea is that you can run these job images easily in any environment, with ConJob or just with Docker, and reproduce the same results.

What problem does it solve?

An important aspect missing from nearly all CI offerings is local reproducibility: builds pass on your local machine but fail in CI. This is challenging to debug and makes it slow to iterate on changes.

The first time I can remember being frustrated with a CI job was the first time I encountered a failing one. Like many others, we used Jenkins at my company. The jobs were defined by a script typed into a text area in the Jenkins GUI, and the environment was a number of special snowflake servers we had provisioned manually.

More times than I can remember there would be a failure on CI that we couldn’t reproduce locally. Attempts to get the build green usually involved making some code changes, pushing them, then seeing if it had the desired effect.

The spiral into insanity would usually look like this:

Carefully analyze the problem, write up some elegant code, craft a nicely written commit message, and push.
[Doesn’t work]
Do everything the same except faster and less elegant.

…

Throw shit at the wall, then run my finely tuned one-liner to “iterate” on my newest idea:

git commit --no-verify -a -m 'HAAAAAAAAANDS' && git push --force --no-verify

Although things with CI have gotten better recently with code based configuration (usually YAML files), this hasn’t solved the problem of local reproducibility. Those YAML files can only be consumed by the remote CI service.

There are some tools like act for GitHub Actions which attempt to replicate what’s happening on CI when you push your code, but there’s no substitute for the real thing. In fact, the first time I tried to use act to replicate a CI issue it was unable to do so.

Since I started using ConJob as the CI server for all my projects, I have yet to find an issue I cannot replicate on my local machine.

The solution

ConJob provides as thin of a layer as possible on top of Docker. You tell it what image to run, optionally provide input to the job, then it returns to you any output. This also allows it to be used in a “serverless” capacity (with the understanding that there will be the usual overhead of starting a container for each request).

If you must, you could say it’s “Docker as a service”, but please don’t :)

To reproduce a CI build failure locally, you’d simply run the same image with the same inputs.

Using ConJob

Getting started with ConJob is easy. Since those using ConJob are likely to be avid users of Docker, the best way to run ConJob is:

docker run -it \
  -p 8080:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  scottg489/conjob

Then make a request to run a job:

curl 'localhost:8080/job/run?image=library/hello-world:latest'

Or try making a request with input:

curl --data 'foobar' 'localhost:8080/job/run?image=scottg489/echo-job:latest'

More info can be found in the project’s README on how to build and run from source.

A note on serverless

Since ConJob can run any image and return the resulting output, it also has the possibility to be used as a serverless backend. This use case is probably best saved for requests that are asynchronous or slow enough so that the overhead of spinning up a container won’t be noticeable.

If you’re curious what this would look like, check out my other project metadiff-ui (which is, of course, also built with ConJob) and hosted at metadiff.com as a proof of concept of using ConJob in this way, albeit maybe not the best use case.

A little extra

As avid Docker users, you’d probably like to use it inside your jobs, particularly if you’re using it for CI. To support this, ConJob allows you to specify a different container runtime. This will be the runtime that the containers (jobs) being run by ConJob are started with:

docker run -it \
  -p 8080:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e CONTAINER_RUNTIME=sysbox_runc \
  scottg489/conjob

The Sysbox runtime is perfect for running Docker-in-Docker and is really the underlying technology that made this project feasible. Note that you’ll need to have the Sysbox runtime installed for this to work.

There are plenty of other configuration options as well such as auth, job limits, and container registry login for using a container registry other than Docker Hub (e.g. GitHub Container registry). For a full list of all configuration options see the default-config.yml file.

Wrapping up

So that’s ConJob! There’s a lot more going on under the surface such as caching between runs and support for secrets. Future plans include adding per-job configuration and a UI. So let me know in the comments if you’d be interested in a deeper dive. I’m also looking for any kind of feedback, ideas, or problems you run into while trying it yourself.

For the time being, you can try ConJob with nothing more than an internet connection:

https://try.conjob.io/job/run?image=library/hello-world:latest

Just change the image query param to an image that can be run as a job (i.e. exits on its own). Also see the curl examples above for providing input to your job.

Be aware that there are some job limitations (duration, etc.) at the moment.

Updating Dropwizard configuration at runtime

Fri, 20 Nov 2020 05:32:36 GMT

An example Dropwizard application demonstrating runtime configuration

Although this feature isn’t supported out of the box by dropwizard, you’re able to update your configuration at runtime fairly easy with the tools they give you.

Before I get started, note that this doesn’t persist the updated config values to the config.yml. However, this would be easy enough to implement yourself simply by writing to the config file from the application. If anyone would like to write this implementation feel free to open a PR on the example project I’ve linked below.

An example application can be found at scottg489/dropwizard-runtime-config-example

Code

Start off with a minimal config:

config.yml

myConfigValue: "hello"

And it’s corresponding configuration file:

ExampleConfiguration.java

public class ExampleConfiguration extends Configuration {
    private String myConfigValue;

    public String getMyConfigValue() {
        return myConfigValue;
    }

    public void setMyConfigValue(String value) {
        myConfigValue = value;
    }
}

Then create a task which updates the config:

UpdateConfigTask.java

public class UpdateConfigTask extends Task {
    ExampleConfiguration config;

    public UpdateConfigTask(ExampleConfiguration config) {
        super("updateconfig");
        this.config = config;
    }

    @Override
    public void execute(Map<String, List<String>> parameters, PrintWriter output) {
        config.setMyConfigValue("goodbye");
    }
}

Also, for demonstration purposes, create a resource which allows you to get the config value:

ConfigResource.java

@Path("/config")
public class ConfigResource {
    private final ExampleConfiguration config;

    public ConfigResource(ExampleConfiguration config) {
        this.config = config;
    }

    @GET
    public Response handleGet() {
        return Response.ok().entity(config.getMyConfigValue()).build();
    }
}

Finally, wire everything up in your application:

ExampleApplication.java

public class ExampleApplication extends Application<ExampleConfiguration> {
    public static void main(String[] args) throws Exception {
        new ExampleApplication().run(args);
    }

    @Override
    public String getName() {
        return "dropwizard-runtime-config-example";
    }

    @Override
    public void run(ExampleConfiguration configuration, Environment environment) {
        environment.jersey().register(new ConfigResource(configuration));
        environment.admin().addTask(new UpdateConfigTask(configuration));
    }
}

Usage

Start up the application

./gradlew clean build install && \
    ./build/install/dropwizard-runtime-config-example/bin/dropwizard-runtime-config-example server config.yml

Then verify everything is working as expected:

$ curl 'http://localhost:8080/config'
hello
$ curl -X POST 'http://localhost:8081/tasks/updateconfig'
$ curl 'http://localhost:8080/config'
goodbye

How it works

This works simply by passing the same reference to the constructor of ConfigResource.java and UpdateConfigTask.java. If you aren’t familiar with the concept see here: https://stackoverflow.com/questions/40480/is-java-pass-by-reference-or-pass-by-value

The linked classes above are to a project I’ve created which demonstrates this as a complete solution:

scottg489/dropwizard-runtime-config-example

Footnote: I haven’t verified this works with the built in configuration. However, the dropwizard Configuration class (which you need to extend for your own configuration) does have various “setters” for the internal configuration.

Using the Google Drive API with Node.js

Mon, 26 Oct 2020 04:44:48 GMT

A simple Node.js application with the Google Drive API

This guide will explain how to integrate with the Google API for Google Drive using the googleapi npm package. I try to make as few assumptions as possible and will be starting from scratch. If you’ve already done some setup yourself, feel free to skip to the relevant section. The app is a bare-bones command line implementation meant to get you started. You’ll then be ready to explore and play around with the API yourself.

If you’re just looking for the code it can be found as a GitHub gist

Or run the following:

git clone git@github.com:b6d134d37dcbb4e8f9e07aca48a9cde1.git google-drive-api-project
cd google-drive-api-project
npm install

Then copy in your service account’s credentials.json key file and run:

ts-node index.ts

Google Cloud API setup
Google Drive configuration
1. Find your service account’s client_email
2. Share your files with the service account
Application code and configuration
Conclusion

Google Cloud API setup

Setting up a Google Cloud account

In order to access Google APIs you need to set up Google Cloud. Navigate to their getting-started page and click on Get started for free in the top right corner. If you already have a Google account (who doesn’t?), follow the guide to sign up and then you should be brought to the Google Cloud Dashboard.

I set up my Google Cloud account years ago, so I apologize if this sign up procedure is a little off.

Creating a project

Google should create your first project for you but if you want to create a new one for yourself, or manage all of your projects in the future, you can go to the Cloud Resource Manager to do so.

Configuring your project to use the Google Drive API

Once you’re in your project navigate, to your project’s API dashboard. Here you can click on ENABLE APIS AND SERVICES. Search for “drive” and click on Google Drive API once it pops up. You can also navigate to it directly by clicking the previous link. Once you’re on this page click ENABLE.

Setting up credentials

After you’ve enabled Google Drive for your project you should be brought to the Google Drive API Overview page. On the left nav bar click Credentials.

Create a service account

In the nav bar near the top, click +CREATE CREDENTIALS and then in the dropdown, Service account. Give the service account a name and description then click CREATE. In the next step you’ll also need to grant the service account access to your project. I opted for the “Owner” role which gives “Full access to all resources”. However, depending on your use case, there may be a more targeted role you’ll want to choose.

Add a key to your service account

Once you’re back on the Credentials page, under the Service Accounts section, click on the service account you just created to edit it. At the bottom of the Service account details page, under the Keys section, click on ADD KEY. In the dialog window that pops up choose “JSON” then click CREATE. For the purposes of this guide this is the format we’ll be using and is important for later steps. The download dialog window should then pop up, and you should save the file into your project’s directory. Later on in the guide I’ll refer to this file as credentials.json.

I should note that this was the first project I’ve set up in Google Cloud. So keep that in mind in case you run into something that conflicts with this guide. However, I’ve made every effort to make sure the steps are accurate to my experience.

Otherwise, this should be all the setup you need to do in Google Cloud. Next we’ll start creating our application!

Google Drive configuration

In order for the service account you’ve created to have access to your personal Google account’s Drive you need to share the file(s) or directory(s) with it. Personally I found this a bit awkward, and it would be nice for there to be a way to give my application access to everything my personal account has access to in Drive. If anyone knows how to do this (without an OAuth popup) I’d be interested to know.

Find your service account’s `client_email`

In the credentials.json file you downloaded earlier there should be a field called client_email. Note this down as you’ll need it for the next step. If you happen to have jq installed you can run jq -r .client_email credentials.json.

Navigate to Google Drive and find the file(s) or directory(s) you wish to share with the service account. Right click on them and in the menu select Share. In he Add people and groups text box enter the client_email value from the last step. To the right of this text box there’s a dropdown that will let you select what permissions the service account will have. Choose whatever is appropriate for your use case. For the purpose of this guide, we only need read access, so I selected “Viewer”. You can also uncheck the Notify people checkbox since there’s no use in notifying a service account. Then click the Share button in the bottom right of the dialog box.

Application code and configuration

Install prerequisites

For this project we’ll be using:

node v10.19.0
npm v6.14.4
ts-node v9.0.0

Node.js is very popular and can be found and installed by most popular package managers. You can also download it from the Node.js downloads page. Once you’ve installed node you’ll also have npm installed. To install ts-node run npm install -g ts-node or follow the ts-node installation instructions.

Project configuration

To configure our npm project we’ll need a package.json and a tsconfig.json. Below you’ll find minimal versions of these files.

package.json

{
  "name": "google-drive-api-project",
  "version": "1.0.0",
  "description": "Google Drive API with Node.js",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "https://gist.github.com/ScottG489/b6d134d37dcbb4e8f9e07aca48a9cde1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typescript": "^4.0.3",
    "googleapis": "^61.0.0"
  }
}

This package.json has been generated from npm init and I’ve added the latest version (at the time of writing) of the package’s we’ll need. You may want to install the latest versions yourself. You can do so by running npm install typescript googleapis.

tsconfig.json

{
  "compilerOptions": {
    "target": "ES2015",
    "module": "commonjs",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  },
  "$schema": "https://json.schemastore.org/tsconfig",
  "display": "Recommended"
}

This tsconfig.json was taken from the recommended tsconfig.json from the Centralized Recommendations for TSConfig bases GitHub project.

After you have these two files in place run npm install.

Application code

The example code is a simple demonstration that lists all the files that were shared with the service account.

index.ts

import {google} from 'googleapis';

const auth = new google.auth.GoogleAuth({
    keyFile: './credentials.json',
    scopes: ['https://www.googleapis.com/auth/drive'],
});
const drive = google.drive({version: 'v3', auth});

(async () => {
    const driveResponse = await drive.files.list({
        fields: '*',
    });
    console.log(driveResponse.data.files);
})().catch(e => {
    console.log(e);
});

Note the reference to your credentials.json file. This needs to be in the same directory before index.ts can be run.

Running the application

ts-node index.ts

The application will print out the JSON object representation of the file(s) you previously shared with the service account.

Conclusion

You now have a working example integrated with the Google Drive API. However, you likely want to do more than just print out the JSON object representations of your Google Drive files.

The best starting point is to check out the documentation on the npm googleapis package page. You can also find some nice samples in the GitHub google-api-nodejs-client repository.

How to create your GitHub profile README

Fri, 02 Oct 2020 04:30:07 GMT

Personalize your GitHub profile page

In the past, your ability to customize your GitHub profile overview page was a bit limited. You could select what repositories you wanted to pin or choose to display your activity overview below your yearly contributions. However, these options didn’t really allow you to express yourself.

With the introduction of the GitHub profile README, you’re able to add some personality to your profile with the full power of Markdown. The new feature is a bit of an Easter egg. It doesn’t seem to have ever been officially announced, though it’s fully documented. It appears to have been released in early July 2020.

If you’d like to set this up for your own profile simply:

Create a repo with the same name as your username
Create a README.md file in that repo’s root
Put some content in it!

The README will then render at the top of your profile page. For more information, see the GitHub documentation on Managing your profile README.

This is my GitHub profile README at the time of writing (layout may differ):

Check out my GitHub profile page to view the current version.

First blog post

Thu, 01 Oct 2020 02:51:29 GMT

Frist psot

First!

:)

A Full and Comprehensive Style Test

Thu, 01 Jan 1970 00:00:00 GMT

Below is just about everything you’ll need to style in the theme. Check the source code to see the many embedded elements within paragraphs.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Lorem ipsum dolor sit amet, test link adipiscing elit. This is strong. Nullam dignissim convallis est. Quisque aliquam. This is emphasized. Donec faucibus. Nunc iaculis suscipit dui. 5³ = 125. Water is H₂O. Nam sit amet sem. Aliquam libero nisi, imperdiet at, tincidunt nec, gravida vehicula, nisl. The New York Times (That’s a citation). Underline. Maecenas ornare tortor. Donec sed tellus eget sapien fringilla nonummy. Mauris a ante. Suspendisse quam sem, consequat at, commodo vitae, feugiat in, nunc. Morbi imperdiet augue quis tellus.

HTML and CSS are our tools. Mauris a ante. Suspendisse quam sem, consequat at, commodo vitae, feugiat in, nunc. Morbi imperdiet augue quis tellus. Praesent mattis, massa quis luctus fermentum, turpis mi volutpat justo, eu volutpat enim diam eget metus. To copy a file type COPY filename. ~~Dinner’s at 5:00.~~ Let’s make that 7. This ~~text~~ has been struck.

Media

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore.

Big Image

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Small Image

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore.

Labore et dolore.

List Types

Definition List

Definition List Title : This is a definition list division.

Definition : An exact statement or description of the nature, scope, or meaning of something: our definition of what constitutes poetry.

Ordered List

List Item 1
List Item 2
1. Nested list item A
2. Nested list item B
List Item 3

Unordered List

List Item 1
List Item 2
- Nested list item A
- Nested list item B
List Item 3

Table

Table Header 1	Table Header 2	Table Header 3
Division 1	Division 2	Division 3
Division 1	Division 2	Division 3
Division 1	Division 2	Division 3

Preformatted Text

Typographically, preformatted text is not the same thing as code. Sometimes, a faithful execution of the text requires preformatted text that may not have anything to do with code. Most browsers use Courier and that’s a good default — with one slight adjustment, Courier 10 Pitch over regular Courier for Linux users.

Code

Code can be presented inline, like <?php bloginfo('stylesheet_url'); ?>, or using jekyll’s highlight filter to highlight a block of code. Because we have more specific typographic needs for code, we’ll specify Consolas and Monaco ahead of the browser-defined monospace font.

#container {
  float: left;
  margin: 0 -240px 0 0;
  width: 100%;
}

Blockquotes

Let’s keep it simple. Italics are good to help set it off from the body text. Be sure to style the citation.

Good afternoon, gentlemen. I am a HAL 9000 computer. I became operational at the H.A.L. plant in Urbana, Illinois on the 12th of January 1992. My instructor was Mr. Langley, and he taught me to sing a song. If you’d like to hear it I can sing it for you. — HAL 9000

And here’s a bit of trailing text.

Text-level semantics

HTML elements

The a element example
The abbr element and abbr element with title examples
The b element example
The cite element example
The code element example
The ~~del element~~ example
The dfn element and dfn element with title examples
The em element example
The i element example
The ins element example
The kbd element example
The mark element example
The q element inside a q element example
The ~~s element~~ example
The samp element example
The small element example
The span element example
The strong element example
The _{sub element} example
The ^{sup element} example
The var element example
The u element example

* * *

Embeds

Sometimes all you want to do is embed a little love from another location and set your post alive.

Video

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Culpa qui officia deserunt mollit anim id est laborum.

Slides

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Culpa qui officia deserunt mollit anim id est laborum.

Audio

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Culpa qui officia deserunt mollit anim id est laborum.

Code

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt.

See the Pen Simple Rotating Spinner by Rob Glazebrook (@rglazebrook) on CodePen.

Isn’t it beautiful.

_[HTML]: Hyper Text Markup Language _[CSS]: Cascading Style Sheets

Scott Giminiani

Introducing ConJob

What is ConJob?

What problem does it solve?

The solution

Using ConJob

A note on serverless

A little extra

Wrapping up

Updating Dropwizard configuration at runtime

An example Dropwizard application demonstrating runtime configuration

An example application can be found at scottg489/dropwizard-runtime-config-example

Code

Usage

How it works

scottg489/dropwizard-runtime-config-example

Using the Google Drive API with Node.js

A simple Node.js application with the Google Drive API

If you’re just looking for the code it can be found as a GitHub gist

Table of contents

Google Cloud API setup

Setting up a Google Cloud account

Creating a project

Configuring your project to use the Google Drive API

Setting up credentials

Create a service account

Add a key to your service account

Google Drive configuration

Find your service account’s client_email

Share your files with the service account

Application code and configuration

Install prerequisites

Project configuration

Application code

Running the application

Conclusion

How to create your GitHub profile README

Personalize your GitHub profile page

First blog post

Frist psot

A Full and Comprehensive Style Test

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Media

Big Image

Small Image

List Types

Definition List

Ordered List

Unordered List

Table

Preformatted Text

Code

Blockquotes

Text-level semantics

Embeds

Video

Slides

Audio

Code

Find your service account’s `client_email`