By evaluating setup time, deployment processes and performance metrics — we aim to guide the selection of the best platform for various application needs. As the demand for scalable and cost-effective solutions continues to grow, understanding the nuances of these platforms becomes essential. Our mission is to empower the community with practical knowledge for informed decisions in cloud-based application deployment. The goal is to compare Heroku and Cloud Run, determining their developer experience and cost effectiveness, and to decide whether there is a clear scenario where both services may be used.
Methodology
This case study utilizes a pre-existing, simple backend service written in Go. The service will be deployed on both Heroku and Google Cloud Platform (GCP) in sequence, with all its dependencies, which in this case, include only a PostgreSQL database. To compare the costs, the service will be deployed with minimal runtime requirements and minimal database configuration, limited to the platforms' smallest tier options.
For the comparison, I’ve chosen the following evaluation criteria:
- Ease and straightforwardness of the setup and deployment
- Flexibility and customization
- Cost
- Monitoring
- Scalability
The order is not arbitrary. Given our focus on a budget and time-constrained MVP solution, scalability, ease of maintenance and security — typically vital for production runtimes — are of lower significance.
This is a subjective, empirical evaluation that is in no shape or form based on statistically significant data. Different evaluators might have different opinions.
Heroku
Creating a New App
The first thing that I experienced was that Heroku, for some reason, doesn't allow you to completely choose your app name. Some values return the “is not available” error shown above without a clear reason. However, adding the dev- prefix solved this issue, allowing me to proceed.
Connecting a GitHub Repository
First, I experimented with a connected GitHub repository. After connecting the backend repo to Heroku, I triggered a manual deploy button. The app was built and deployed to Heroku.
Despite confirming the “successful” deployment, the application was not running properly because I hadn’t configured the runtime environment variables yet. Clicking the view button showed that the application had encountered an error, but no further information was available.
When opening the Logs page, I saw an error stating that “No web process running.” While this message is not very helpful, it is understandable given that I hadn’t configured the app to run yet.
2023-11-03T07:29:06.350630+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/favicon.ico" host=dev-backend-api-09c031bdf982.herokuapp.com request_id=98b025b3-77db-4c71-ac4c-299c28521b83 fwd="62.168.3.170" dyno= connect= service= status=503 bytes= protocol=httpsConfiguring the Application
Using the Heroku CLI, I connected to the existing Heroku app from my local Git repository with the simple command: heroku git:remote -a dev-backend-api.
I could then set individual config variables using the Heroku CLI, such as: heroku config:get PORT=8080.
Unfortunately, whenever you set a config variable, the application restarts. This meant my app restarted 24 times just to set the configuration. There are also other issues with this kind of configuration:
- Heroku only allows setting environment variable values directly, meaning there is no alternative to Secret Manager like those offered by AWS and GCP. Consequently, anyone with access to the app can view the configuration values.
- There is no centralized management of configuration values. This implies that two applications must have their environments configured separately, which leaves room for human error and misconfiguration and increases the risk of leakage of the value. For example, this is problematic when using an Auth service for handling JWT creation and other services for validating the JWT.
At this point, the application is still not running because the dynos are turned OFF by default. Turning on dynos (the cheapest ECO tier that costs $5/month) triggers application startup. However, the app still crashes due to missing credentials.
2023-11-03T08:46:55.253014+00:00 heroku[api.1]: State changed from starting to up
2023-11-03T08:46:55.285886+00:00 app[api.1]: {"level":"fatal","time":"2023-11-03T08:46:55Z","caller":"api/main.go:61","msg":"setup firebase auth","error","getting
Firebase Auth client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information"}Since the app uses Firebase Admin SDK, Heroku doesn’t expose the service account credentials in the environment the way GCP does out of the box. This is unfortunate because the application is expected to run in a GCP-enabled environment and doesn’t provide a mechanism to inject firebase configuration to its config at the moment.
Google documentation suggests using the GOOGLE_APPLICATION_CREDENTIALS environment variable, which should point to a JSON file. This means we would need to either:
- Push that file to the repo (unsafe, not a good idea).
- Choose a different flow and build a custom Docker container containing the credential file (unsafe).
- Implement a custom configuration mechanism that reads this configuration from the environment directly.
To solve this, I chose the last option and implemented a custom configuration. This is already a significant slowdown to the Heroku flow.
Once implemented and pushed, Heroku automatically deployed the changes, but the application is unfortunately still not functional. To get visibility into the state of the app, I had to enable a higher Tier and upgrade my dynos.
After 60 seconds, an error was thrown:
2023-11-03T10:44:21.914848+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind $PORT within 60 seconds of launch
2023-11-03T10:44:21.914848+00:00 heroku[web.1]: Stopping process with SIGKILL
2023-11-03T10:44:21.914848+00:00 heroku[web.1]: Process exited with status 137
2023-11-03T10:44:21.914848+00:00 heroku[web.1]: State changed from starting to crashedApparently, Heroku sets the PORT variable that the application is supposed to bind to, and there is no way to configure this port through the GUI. Since my application by default uses prefixed environment variables to distinguish between multiple services running on the host, this required yet another configuration change.
After adjusting this configuration, I was finally able to run the application successfully. The total setup time so far was about 2 hours.
Database
So far, I have been using an existing external database hosted on Supabase. Supabase provides this service for free for development purposes, making it a decent and reasonable choice for MVP projects. To finish the evaluation, I am going to set up Heroku Postgres instead.
Heroku Postgres is deployed with a single click, that’s pretty straightforward. However, the application configuration is again problematic, as I have to change four configuration values. Heroku restarts the application on every change, meaning the application will crash three times before all the config values are set.
Once updated, I was not able to connect to the database. The application throws the following error:
2023-11-03T12:37:16.430024+00:00 app[web.1]: {"level":"fatal","time":"2023-11-03T12:37:16Z","caller":"api/main.go:52","msg":"setup datavase","error":"opening database: failed to connect to 'host=ec2-34-202-53-101.compute-1.amazonaws.com user=nuugcyzdjilbeb datebase=dcicrhumcpq00d': server error (FATAL: no pg_hba.conf entry for host \"52.90.172.71\", user \"nuugcyzdjilbeb\", database \"dcicrhumcpq00d\", no encryption (SQLSTATE 28000))"}This happens because the default connection mode of the client is set to SSL_MODE=disable. It’s actually a good thing that this is raised since it promotes proper security practices, although it’s a bit unfortunate that Heroku doesn’t allow changing this setting easily.
After resolving this issue, the Heroku setup is complete for the MVP of a Go API.
Cloud Run
As a disclaimer, let’s assume there already exists a GCP project that you can use. Setting up the project itself is trivial; however, an organization must already exist. Setting up an organization can get a bit tedious, so we will skip that for now and assume it already exists. This is a valid assumption in the context of STRV since A) the organization is set up by the client, and B) there is often a requirement from other platforms to set up Firebase, which also requires a GCP project to function.
Creating a New App
In the context of Cloud Run, the app is called a service. We have two options: either deploy a revision from an existing container image or to continuously deploy new revisions from a source Git repository. Since I tried the path of least resistance with Heroku, I will choose continuous deployment from a source repository, even though this may not be the most likely development method in the context of STRV.
I chose the same setup as Heroku, meaning the simplest way possible: allowing direct internet access and unauthenticated invocations.
The prerequisite to creating a service is to set up Cloud Build. Clicking on the button “SET UP WITH CLOUD BUILD” lets me select a repository provider by connecting my GitHub repository via the Google Cloud Build GitHub app. Unlike Heroku, Cloud Run allows you to directly pick “Dockerfile” as the build type, meaning that you have way more control over the build from the source repo.
Configuring the Application
Once clicked on the “SAVE” button, the build immediately starts. Cloud Run logs every step of the build quite extensively, and the logs are clear, concise, easily accessible and searchable, which provides a better developer experience than in Heroku’s case.
As expected, the process fails in the deployment step because we haven’t configured the service yet. The build log informs you it was unable to start the service and points you to the logs for the particular revision, i.e., the logs of your service. All the error messages and logs are also visible from the revision’s page, which is rather pleasant.
Clicking on “EDIT & DEPLOY NEW REVISION” takes you to the service configuration, where we can add environment variables and even reference secrets from the secret manager. From now on, we’ll follow the same process as we did on Heroku and add all the configuration as environment variables, even though this would be discouraged in the production use case. The preferred way would be to reference sensitive information from the secrets. However, before we add all the env variables, we have to set up a database for the service first.
Database
Still in the “EDIT & DEPLOY NEW REVISION” tab, there is an option to add a connection to the Cloud SQL, a relational database service from GCP. Clicking on the field prompts you to enable the API, and you will find an option to connect via a connection string if you already have an instance created. Otherwise, you have to create a new Cloud SQL instance.
To create a new instance, I headed over to the Cloud SQL tab and proceeded to create an instance by clicking on the "CREATE INSTANCE" field and selecting PostgreSQL. The database is highly customizable, with predefined setups for development, staging, and production instances, and the ability to configure your database parameters. However, I wasn’t able to match Heroku’s smallest MINI database tier that I chose during the setup (priced at $0.007 per hour, with 1GB storage limit and a 10,000 row limit). Cloud SQL offers a minimum of 10GB storage with no row limits for $0.01 per hour.
Comparing the lowest database tiers on either platform, a month of runtime on GCP costs roughly $9 vs. Heroku’s cap of $5 a month. Even though the smallest tier prices favor Heroku, when comparing higher equivalent tiers, GCP turns out to be the cheaper alternative. At 256 GB of storage and 8 CPUs (a staging or production setup for a small project), Heroku caps at $200/month while GCP’s estimated spend is just $145.
Once you confirm the configuration of the database, you should go have a coffee because it takes roughly 10 minutes to finalize. Compared to Heroku, this is slower and hinders the experience of an otherwise smooth flow. Once created, we can pass the database connection parameters as environment variables along with the rest of the needed env variables to the revision configuration. That completes the setup of the service.
One thing to note here: as I mentioned earlier, Heroku enforces SSL connection by default. GCP, however, does not. The setting is configurable, and once enabled, Cloud SQL requires SSL connection with the certificate that it automatically generates. The certificate is valid for 10 years, so no further management of the certificate is needed for quite a while.
Evaluation
The evaluation is based on the comparison criteria that I described in Methodology.
Ease and Straightforwardness of the Setup and Deployment
Deploying on Heroku required adjustments to the application to fit its environment, and with debugging issues, including poor logging and lackluster built-in monitoring, it took longer than anticipated. Despite these obstacles, the application was up and running in about 2 hours with a 10 to 15-minute coffee break. Don’t get me wrong, that’s still superb compared to how much it would take to create a full-fledged deployment on AWS with no premade templates or IaaC.
I expected that it would take me around 20 to 30 minutes altogether if I’d known what I was doing and if I didn’t have to debug any issues – i.e., the app would already be prepared for Heroku’s environment.
I spent 1 hour and 30 minutes deploying the application to GCP, with no changes to the code needed and a 15 minute coffee break during the database setup. The flow was much smoother, and I was navigated by GCP throughout the whole process without having to leave GCP to look up any sort of documentation. The logs (both build and runtime) were much clearer, easily accessible and better searchable. The built-in Docker pipeline also made me more comfortable since I knew what was happening in the background, and the process was more controllable. By far, the worst experience with Heroku was environment configuration, which was super annoying. Luckily, GCP allowed for both secrets and env variables to be easily configured with the GUI in one go.
Flexibility and Customization
To be completely blunt, Heroku can’t compete (and it’s not even its intention to) with the sheer amount of customizable parameters that GCP’s Cloud Run and Cloud SQL offer. In particular, I appreciate the ability to fully customize my database instance (as opposed to Heroku offering just a predefined tier with a limited number of rows) or to be able to hide everything from the public and use the private authenticated connection to the service (for security purposes, this might come in handy sometime). Last but not least, secret management in GCP is way more portable to the real production environment than the plain text environment config of Heroku.
Cost
Cost might become the top priority for the client. Heroku has been seen as the cheapest solution for quite some time, but is it really? Comparing purely the runtime, Heroku’s cheapest option is $5/month for the eco run, GCP’s API service runtime is essentially free for the development load. Regarding databases, Cloud SQL's cheapest instance is a bit pricier than Heroku's PostgreSQL addon ($9 vs $5 monthly), but higher Cloud SQL tiers are cheaper. Considering a $4 monthly difference is likely insignificant, in the broader context, it is arguably a better choice to start with a database that's cheaper for production. As a side note, for ultra-cheap development, plugging Supabase (free tier for one project) into GCP's Cloud Run service is the way to go!
Monitoring
Most developers know that monitoring is essential for production, but some underestimate the importance of proper monitoring setup during development. On Heroku, achieving a reasonable monitoring setup required using a paid Graphite add-on, increasing costs by an extra $25 a month (capped). However, I found the experience with Graphite unsatisfactory compared to solutions like PromQL + Grafana that I'm used to. On the other hand, GCP offers Google Cloud Monitoring plugged directly into Cloud Run. Out of the box, you can configure your dashboard, add custom metrics and alerts from existing ones, or write a PromQL query to your application and plug it into Cloud Monitoring. With minimal setup, this came free with the GCP solution.
Scaling
Scaling is not something you'd typically worry about during development, right? On the contrary, it’s crucial to consider. First and foremost, code should be written with scalability in mind. After all, applications are built to serve potentially millions of users eventually. It doesn’t matter that only a handful of them succeed in doing so, all of them should be able to.
While I haven't had the opportunity to test scaling on either Heroku or GCP, the following are theoretical assumptions. With Heroku, autoscaling is available out of the box but is based on the tier, starting with the professional tier by default (get ready for a $250 bill). On the other hand, Cloud Run scales automatically based on the needs, without requiring changes to the billing tier. It's also possible to set limits on the minimum and maximum number of instances to prevent excessive costs.
Results
The primary objective of the case study was to compare Heroku and Cloud Run based on evaluation criteria outlined in the Methodology section, and to determine whether there is a clear case to use either of the services in specific scenarios.
Both services are comparable in terms of ease and straightforwardness of setup, as well as cost, with Cloud Run being slightly more expensive but still within the acceptable range for an MVP. However, the developer experience with Cloud Run was much more pleasant. Considering that Cloud Run offers more customization options (practically limitless), I would lean towards Cloud Run in the future. Personally, I don’t see any reason to choose Heroku over Cloud Run for an MVP project with minimal requirements, such as the one used in this case study.
