|
| 1 | +# Google BigQuery JDBC Client for Java |
| 2 | + |
| 3 | +Java idiomatic client for [BigQuery JDBC][product-docs]. |
| 4 | + |
| 5 | +[![Maven][maven-version-image]][maven-version-link] |
| 6 | +![Stability][stability-image] |
| 7 | + |
| 8 | +- [Product Documentation][product-docs] |
| 9 | +- [Client Library Documentation][javadocs] |
| 10 | + |
| 11 | + |
| 12 | +## Quickstart |
| 13 | + |
| 14 | + |
| 15 | +If you are using Maven, add this to your pom.xml file: |
| 16 | + |
| 17 | +```xml |
| 18 | +<dependency> |
| 19 | + <groupId>com.google.cloud</groupId> |
| 20 | + <artifactId>google-cloud-bigquery-jdbc</artifactId> |
| 21 | + <version>LATEST_VERSION</version> |
| 22 | +</dependency> |
| 23 | +``` |
| 24 | + |
| 25 | +If you are using Gradle without BOM, add this to your dependencies: |
| 26 | + |
| 27 | +```Groovy |
| 28 | +implementation 'com.google.cloud:google-cloud-bigquery-jdbc:LATEST_VERSION' |
| 29 | +``` |
| 30 | + |
| 31 | +If you are using SBT, add this to your dependencies: |
| 32 | + |
| 33 | +```Scala |
| 34 | +libraryDependencies += "com.google.cloud" % "google-cloud-bigquery-jdbc" % "LATEST_VERSION" |
| 35 | +``` |
| 36 | + |
| 37 | +## Authentication |
| 38 | + |
| 39 | +See the [Authentication][authentication] section in the base directory's README. |
| 40 | + |
| 41 | +## Authorization |
| 42 | + |
| 43 | +The client application making API calls must be granted [authorization scopes][auth-scopes] required for the desired BigQuery JDBC APIs, and the authenticated principal must have the [IAM role(s)][predefined-iam-roles] required to access GCP resources using the BigQuery JDBC API calls. |
| 44 | + |
| 45 | +## Developer Guide |
| 46 | + |
| 47 | +### Prerequisites |
| 48 | + |
| 49 | +You need to have either Java with Maven installed or Docker. You might want to install [`Make`](https://www.gnu.org/software/make/) to simplify running commands, otherwise please look into Makefile to check for specific configurations. |
| 50 | + |
| 51 | +### Setup |
| 52 | + |
| 53 | +`make install` primarily relies on `mvn install` command. All following commands are primarily applicable for the `google-cloud-bigquery-jdbc` project. |
| 54 | +You can also use `make clean` to clean the project and `make lint` to format the code. |
| 55 | + |
| 56 | +### Running tests |
| 57 | + |
| 58 | +#### Unittests |
| 59 | + |
| 60 | +Run all unittests |
| 61 | + |
| 62 | +`make unittest` |
| 63 | + |
| 64 | +Run specific unittests |
| 65 | + |
| 66 | +`make unittest test=<filter>` |
| 67 | + |
| 68 | +Please reference [Maven documentation](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) for details about `<filter>` |
| 69 | + |
| 70 | +Example: `make unittest test=BigQueryArrowStructTest` |
| 71 | + |
| 72 | +#### Integration tests |
| 73 | + |
| 74 | +IMPORTANT: Running integration tests will skip unit tests by default. To include unit tests, run `make integration-test skipSurefire=false`. Primary focus of this command is to run specific set of tests without a lot of overhead. |
| 75 | + |
| 76 | +Set the following environment variables to run the integration tests: |
| 77 | + |
| 78 | +``` |
| 79 | +# Default gcloud auth setup |
| 80 | +export GOOGLE_APPLICATION_CREDENTIALS=<path-to-service-account-json-config-file> |
| 81 | +
|
| 82 | +# Test specfic envs |
| 83 | +export SA_EMAIL=email@email.com |
| 84 | +export SA_SECRET=<path-to-service-account-json-config-file> |
| 85 | +# Alternatively it can be json content: |
| 86 | +export SA_SECRET=`cat <path-to-service-account-json-config-file>` |
| 87 | +``` |
| 88 | + |
| 89 | +Run all integration tests (currently takes 15+ minutes, so this is discouraged). |
| 90 | + |
| 91 | +`make integration-test` |
| 92 | + |
| 93 | +Run specific integration test |
| 94 | + |
| 95 | +`make integration-test test=<filter>` |
| 96 | + |
| 97 | +Please reference [Maven documentation](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) for details about `<filter>` |
| 98 | + |
| 99 | +Example: `make integration-test test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey` |
| 100 | + |
| 101 | +### Dockerized environment |
| 102 | + |
| 103 | +If you don't have Java or Maven, or if you want to test changes with a different Java version, you can leverage dockerized environment. |
| 104 | + |
| 105 | +One-time run commands are similar to local development make commands: |
| 106 | + |
| 107 | +``` |
| 108 | +make docker-build |
| 109 | +make docker-unittest |
| 110 | +make docker-integration-test |
| 111 | +``` |
| 112 | + |
| 113 | +Please note that running unit or integration tests within docker doesn't leverage maven cache because it is not persisted. |
| 114 | +If you want to run multiple commands, you can start a session and treat this shell session as your local environment. |
| 115 | + |
| 116 | +``` |
| 117 | +make docker-session |
| 118 | +``` |
| 119 | + |
| 120 | +All Docker commands rely only on `GOOGLE_APPLICATION_CREDENTIALS` env being present, it will create rest of env vars as needed. |
| 121 | + |
| 122 | +### Packaging |
| 123 | + |
| 124 | +There are a few ways to package the Google JDBC Driver. The output of the packaging commands can be found in the `target` directory. |
| 125 | + |
| 126 | +`make package` or `make docker-package` will create both a thin jar and a zip file with all dependencies. |
| 127 | +`make package-all-dependencies` or `make docker-package-all-dependencies` will create a single jar with all dependencies included. |
| 128 | +`make package-all-dependencies-shaded` or `make docker-package-all-dependencies-shaded` will create a single shaded jar with all dependencies included. |
| 129 | + |
| 130 | +#### Thin jar |
| 131 | + |
| 132 | +The thin jar is created with `make package`. The thin jar is packaged as a zip file with its dependencies listed in a `dependencies.txt` file, compatible with `pom.xml`. |
| 133 | + |
| 134 | +#### All dependencies |
| 135 | + |
| 136 | +The jar with all dependencies is also created. This jar includes all dependencies and can be used as a standalone jar with tools like R-Studio. You can run `make package-all-dependencies` or `make docker-package-all-dependencies` to build only the jar with all dependencies. |
| 137 | + |
| 138 | +#### Shaded Jar |
| 139 | +You can also build a shaded jar with all dependencies. This can be done by running `make package-all-dependencies-shaded` or `make docker-package-all-dependencies-shaded`. |
| 140 | + |
| 141 | +### Nightly builds |
| 142 | + |
| 143 | +The nightly build runs the full set of integration tests, including the `ITBigQueryJDBCTest` and `ITNightlyBigQueryTest` test suites. It also includes some long-running tests (takes 20+ minutes to complete). |
| 144 | + |
| 145 | +**Note:** These builds are intended for testing and development purposes only and are not recommended for production use. |
| 146 | + |
| 147 | +Nightly Integration Tests include a step to build a full and thin jars and upload them to Google Storage. |
| 148 | + |
| 149 | +They can be retrieved via following commands: |
| 150 | + |
| 151 | +``` |
| 152 | +gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest-all.jar . |
| 153 | +gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest.zip . |
| 154 | +``` |
| 155 | + |
| 156 | +#### Performance tests |
| 157 | + |
| 158 | +Cloud Build Pipeline is uploading latest full jar to the internal location for perf tests once a week. |
| 159 | + |
| 160 | +### Code Coverage |
| 161 | + |
| 162 | +We're using [JaCoCo](https://www.eclemma.org/jacoco/) to track Code Coverage. `Makefile` has 2 separate set of commands for unittests and integration tests reports. |
| 163 | + |
| 164 | +You can run `make unit-test-coverage` to generate a coverage report for unit tests. The output will be in `jacoco-unittests.zip`. |
| 165 | +You can run `make full-coverage` to generate a coverage report for both unit and integration tests. The output will be in `jacoco-full.zip`. |
| 166 | + |
| 167 | +You can also run `make docker-coverage` which will produce both results. You can find `jacoco-unittests.zip` and `jacoco-full.zip` files in the root with results. |
| 168 | + |
| 169 | +## Getting Started |
| 170 | + |
| 171 | +### Prerequisites |
| 172 | + |
| 173 | +You will need a [Google Cloud Platform Console][developer-console] project with the BigQuery JDBC [API enabled][enable-api]. |
| 174 | + |
| 175 | +[Follow these instructions][create-project] to get your project set up. You will also need to set up the local development environment by |
| 176 | +[installing the Google Cloud Command Line Interface][cloud-cli] and running the following commands in command line: |
| 177 | +`gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`. |
| 178 | + |
| 179 | +### Installation and setup |
| 180 | + |
| 181 | +You'll need to obtain the `google-cloud-bigquery-jdbc` library. See the [Quickstart](#quickstart) section |
| 182 | +to add `google-cloud-bigquery-jdbc` as a dependency in your code. |
| 183 | + |
| 184 | +## About BigQuery JDBC |
| 185 | + |
| 186 | + |
| 187 | +[BigQuery JDBC][product-docs] |
| 188 | + |
| 189 | +See the [BigQuery JDBC client library docs][javadocs] to learn how to |
| 190 | +use this BigQuery JDBC Client Library. |
| 191 | + |
| 192 | + |
| 193 | + |
| 194 | + |
| 195 | + |
| 196 | + |
| 197 | +## Troubleshooting |
| 198 | + |
| 199 | +To get help, follow the instructions in the [shared Troubleshooting document][troubleshooting]. |
| 200 | + |
| 201 | +## Supported Java Versions |
| 202 | + |
| 203 | +Java 8 or above is required for using this client. |
| 204 | + |
| 205 | +Google's Java client libraries, |
| 206 | +[Google Cloud Client Libraries][cloudlibs] |
| 207 | +and |
| 208 | +[Google Cloud API Libraries][apilibs], |
| 209 | +follow the |
| 210 | +[Oracle Java SE support roadmap][oracle] |
| 211 | +(see the Oracle Java SE Product Releases section). |
| 212 | + |
| 213 | +### For new development |
| 214 | + |
| 215 | +In general, new feature development occurs with support for the lowest Java |
| 216 | +LTS version covered by Oracle's Premier Support (which typically lasts 5 years |
| 217 | +from initial General Availability). If the minimum required JVM for a given |
| 218 | +library is changed, it is accompanied by a [semver][semver] major release. |
| 219 | + |
| 220 | +Java 11 and (in September 2021) Java 17 are the best choices for new |
| 221 | +development. |
| 222 | + |
| 223 | +### Keeping production systems current |
| 224 | + |
| 225 | +Google tests its client libraries with all current LTS versions covered by |
| 226 | +Oracle's Extended Support (which typically lasts 8 years from initial |
| 227 | +General Availability). |
| 228 | + |
| 229 | +#### Legacy support |
| 230 | + |
| 231 | +Google's client libraries support legacy versions of Java runtimes with long |
| 232 | +term stable libraries that don't receive feature updates on a best efforts basis |
| 233 | +as it may not be possible to backport all patches. |
| 234 | + |
| 235 | +Google provides updates on a best efforts basis to apps that continue to use |
| 236 | +Java 7, though apps might need to upgrade to current versions of the library |
| 237 | +that supports their JVM. |
| 238 | + |
| 239 | +#### Where to find specific information |
| 240 | + |
| 241 | +The latest versions and the supported Java versions are identified on |
| 242 | +the individual GitHub repository `github.com/GoogleAPIs/java-SERVICENAME` |
| 243 | +and on [google-cloud-java][g-c-j]. |
| 244 | + |
| 245 | +## Versioning |
| 246 | + |
| 247 | + |
| 248 | +This library follows [Semantic Versioning](http://semver.org/). |
| 249 | + |
| 250 | + |
| 251 | + |
| 252 | +## Contributing |
| 253 | + |
| 254 | + |
| 255 | +Contributions to this library are always welcome and highly encouraged. |
| 256 | + |
| 257 | +See [CONTRIBUTING][contributing] for more information how to get started. |
| 258 | + |
| 259 | +Please note that this project is released with a Contributor Code of Conduct. By participating in |
| 260 | +this project you agree to abide by its terms. See [Code of Conduct][code-of-conduct] for more |
| 261 | +information. |
| 262 | + |
| 263 | + |
| 264 | +## License |
| 265 | + |
| 266 | +Apache 2.0 - See [LICENSE][license] for more information. |
| 267 | + |
| 268 | +Java is a registered trademark of Oracle and/or its affiliates. |
| 269 | + |
| 270 | +[product-docs]: https://cloud.google.com/bigquery |
| 271 | +[javadocs]: https://cloud.google.com/java/docs/reference/google-cloud-bigquery/latest/history |
| 272 | +[stability-image]: https://img.shields.io/badge/stability-unknown-red |
| 273 | +[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-bigquery-jdbc.svg |
| 274 | +[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-bigquery-jdbc/0.0.0 |
| 275 | +[authentication]: https://github.com/googleapis/google-cloud-java#authentication |
| 276 | +[auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes |
| 277 | +[predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles |
| 278 | +[iam-policy]: https://cloud.google.com/iam/docs/overview#cloud-iam-policy |
| 279 | +[developer-console]: https://console.developers.google.com/ |
| 280 | +[create-project]: https://cloud.google.com/resource-manager/docs/creating-managing-projects |
| 281 | +[cloud-cli]: https://cloud.google.com/cli |
| 282 | +[troubleshooting]: https://github.com/googleapis/google-cloud-java/blob/main/TROUBLESHOOTING.md |
| 283 | +[contributing]: https://github.com/googleapis/java-bigquery/blob/main/CONTRIBUTING.md |
| 284 | +[code-of-conduct]: https://github.com/googleapis/java-bigquery/blob/main/CODE_OF_CONDUCT.md |
| 285 | +[license]: https://github.com/googleapis/java-bigquery/blob/main/LICENSE |
| 286 | + |
| 287 | +[semver]: https://semver.org/ |
| 288 | +[cloudlibs]: https://cloud.google.com/apis/docs/client-libraries-explained |
| 289 | +[apilibs]: https://cloud.google.com/apis/docs/client-libraries-explained#google_api_client_libraries |
| 290 | +[oracle]: https://www.oracle.com/java/technologies/java-se-support-roadmap.html |
| 291 | +[g-c-j]: https://github.com/googleapis/google-cloud-java |
0 commit comments