Merge the baseline code into master branch (#55)

* Baseline code to query Spanner using DynamoDB queries (#3) * Baseline code to query Spanner using DynamoDB queries * readme file * DYN-40: integration testing framework added * DYN-39: added base functions for output change * DYN-41: GetItem API added * DYN-41 test cases imporved * DYN-41: getItem API output & test case changed * DYN-43: BatchGet api input and output changes done * DYN-43: test cases added & end-point correction * DYN-41: Query api change * DYN-42: test cases added * DYN-42: query api output & testcases changed * DYN-44: scan input & output changed * DYN-44: corrected url & added test cases * DYN-31: input & output chnages for UpdateItem * DYN-31: Put API input & output changes * DYN-31: test cases & output change for Update * DYN-31: removed commented code for test * DYN-47: DeleteItem API implementation & test cases * DYN-48: BatchWriteItem API input changes * DYN-48: BatchWriteItem API test cases added * DYN-48: test cases for spanner has been corrected * DYN-47: Delete API test cases * corrected 1 test case * DYN-33: initial test data has been added * DYN-51: imporved readme file * utils: unit test added * config_test added * condition test cases added * services test cases added * test case for services added * CCI03-52 : license header added * CCI03-52: package comments added * Update README.md Co-authored-by: Knut Olav Løite <[email protected]> * CCI03-53: readme file & config file improved * prod config file improved * added proper space in readme * fix : issues related linting fixed * fix: linting issues for log and model fixed * fix: logger_test linting issue fixed * fix: logger_test linting issue fixed * updated README.md and configured integration test to run setup and cleanup logic * updated config to contain description * Fix: Typos - Fixed environment name. - Replaced `.` with `-` for the config file names to match the expected conterpart. - Added copyright header to integrationtest/setup.go. * integration test: to check ddl table updated correctly before execution tests. Also, added CONTRIBUTING.md file Co-authored-by: ankitmalikg2 <[email protected]> Co-authored-by: Knut Olav Løite <[email protected]> Co-authored-by: Saurav Ghosh <[email protected]> * Updates for automated testing (#25) Co-authored-by: skuruppu <[email protected]> * Adding a starter .gitignore to the project. (#26) * Changing up the config loader and documentation to simplify the code and support more than just production and staging environments. (#31) * Simplifying the logger initilization and wrappers. (#32) * Adding router function to parse X-Amz-Target and send the request to the appropiate handler. (#34) * Fixing queryResponse to grab the correct results list. (#36) * Adding support for Numeric data type. (#35) * Example application written in Golang (#27) * Initial commit of example data, scripts and golang application. * Added more read access patterns * Pulling region from env vars. * Updates to handle the query endpoints for the adatper. * Adding config for the indexes. * Removing the code to make extra sessions. * Updating to go 1.17 * Cleaning up the README some and updating to the new config file names. * Found a couple more places were config file names needed to be fixed. * Apply suggestions from code review Co-authored-by: skuruppu <[email protected]> Co-authored-by: skuruppu <[email protected]> * Various fixes for linting and tests (#33) * Fixing broken test in config_test.go * Fixing linter errors. * Cleaning up redundant declaration and adding skip for short tests. * Updates to the CircleCI config to increase parallerism and fix linting. * Syntax fix in the circleci config. * Cleaning up deadcode linting errors. * Cleaning up missing error checks. * Cleaning up some unused code. * Fixing gosimple errors from the linter. * Fixing a missing err check. * more gosimple fixes. * More linter fixes. * increasing the linter timeout. * Cleaning up the last few linter errors. * Fixed another staticcheck error. * Output updates for the integraiton tests. * Moved changeTableNameForSP to utils and updated other files use the new centralized version. (#40) * Documentation updates and clean up (#41) * Add compatibility info and resturctured READMEs. Restructured the README to reference the examples and moved the integration testing instructions to their own README. Also added supported DynamoDB operations and data types to the README. * Minor tweaks and spelling fixes. * Fixes for issues found during review. * CircleCI config and sytles for markdown linting. * Linting fixes. * Changing the matching to use regex and added tests for varying cases. (#49) * Changing the matching to use regex and added tests for varying cases. * Changing the regexes to use the case insensitive flag. * Adding models for the BatchWriteItem response and error handling. (#50) * Adding example functions for more DB operations (#48) * Adding UpdateItem example * Adding UpdateItem example * Adding PutItem example * Adding DeleteItem example * Adding ScanItem example * Adding more example functions * Adding BatchWriteItem example * Adding BatchWriteItem example * Fixing indentation and moving some code * Fixing indentation and moving some code * Fixing indentation and moving some code * Fixing indentation and moving some code * Fixing indentation and moving some code * Fixing indentation and moving some code * Fixing indentation and moving some code * Starting to add tests and some refactor for storage/spanner.go (#44) Changes: - Added spanner_test.go. Tests currently on cover parseRow - Renamed parseRowForNull to parseRow (issue #22) - createRowMap and parseRow did pretty much the same thing so deleted createRowMap and updated reference to use parseRow - Removed the unneeded cols argument from parseRow * Fixes for integration test failures in CircleCI (#54) * Changing to use the working directory for the config files and directly export the credentials environment var. * Removing branch restriction from integration_test * Fixing the spanner.json config to use the var Co-authored-by: Roshan Patil <[email protected]> Co-authored-by: ankitmalikg2 <[email protected]> Co-authored-by: Knut Olav Løite <[email protected]> Co-authored-by: Saurav Ghosh <[email protected]> Co-authored-by: skuruppu <[email protected]> Co-authored-by: Shobhit Gupta <[email protected]>
cloudspannerecosystem · Nov 11, 2021 · cc46ff8 · cc46ff8
1 parent 44dc1c5
commit cc46ff8
Show file tree

Hide file tree

Showing 69 changed files with 15,207 additions and 71 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -3,28 +3,111 @@
 # Check https://circleci.com/docs/2.0/language-go/ for more details
 version: 2
 jobs:
-  build_and_test:
+  build:
     docker:
       - image: circleci/golang:1.13
-
     working_directory: /go/src/github.com/cloudspannerecosystem/dynamodb-adapter
+    steps:
+      - checkout
+      - run: go build
 
+  lint_golang:
+    docker:
+      - image: golangci/golangci-lint:latest-alpine
     steps:
       - checkout
+      - run: golangci-lint run --timeout 5m
 
+  lint_markdown:
+    docker:
+      - image: circleci/ruby:2.7
+    steps:
+      - checkout
+      - run: gem install mdl
+      - run: mdl .
+
+  unit_test:
+    docker:
+      - image: circleci/golang:1.13
+    working_directory: /go/src/github.com/cloudspannerecosystem/dynamodb-adapter
+    steps:
+      - checkout
+      - run: go test -v -short ./...
+
+  integration_test:
+    docker:
+      - image: circleci/golang:1.13
+    working_directory: /go/src/github.com/cloudspannerecosystem/dynamodb-adapter
+    steps:
+      - checkout
       - run:
           name: set credentials
           command: |
             echo $GCLOUD_SERVICE_KEY > $HOME/gcloud-service-key.json
             echo 'export GOOGLE_APPLICATION_CREDENTIALS=$HOME/gcloud-service-key.json' >> $BASH_ENV
-      - run: go test -v ./...
+      - run:
+          name: integration tests
+          command: |
+            cat <<EOF > $CIRCLE_WORKING_DIRECTORY/config-files/staging/config.json
+            {
+                "GoogleProjectID": "$SPANNER_PROJECT",
+                "SpannerDb": "dynamodb-adapter-int",
+                "QueryLimit": 5000
+            }
+            EOF
+            cat <<EOF > $CIRCLE_WORKING_DIRECTORY/config-files/staging/spanner.json
+            {
+                "dynamodb_adapter_table_ddl": "$SPANNER_INSTANCE",
+                "dynamodb_adapter_config_manager": "$SPANNER_INSTANCE",
+                "department": "$SPANNER_INSTANCE",
+                "employee": "$SPANNER_INSTANCE"
+            }
+            EOF
+            cat <<EOF > $CIRCLE_WORKING_DIRECTORY/config-files/staging/tables.json
+            {
+                "employee":{
+                    "partitionKey":"emp_id",
+                    "sortKey": "",
+                    "attributeTypes": {
+                        "emp_id": "N",
+                        "first_name":"S",
+                        "last_name":"S",
+                        "address":"S",
+                        "age":"N"
+                    },
+                    "indices": {}
+                },
+                "department":{
+                    "partitionKey":"d_id",
+                    "sortKey": "",
+                    "attributeTypes": {
+                        "d_id": "N",
+                        "d_name":"S",
+                        "d_specialization":"S"
+                    },
+                    "indices": {}
+                }
+            }
+            EOF
+            go run integrationtest/setup.go setup
+            go test integrationtest/api_test.go
+            go run integrationtest/setup.go teardown
 
 workflows:
   version: 2
 
   commit:  # Run on every commit.
     jobs:
-      - build_and_test
+      - build
+      - lint_golang
+      - lint_markdown
+      - unit_test
+      - integration_test:
+          filters:
+            branches:
+              only:
+                  - master
+                  - baseline
 
   nightly:  # Run every night.
     triggers:
@@ -35,4 +118,8 @@ workflows:
               only:
                   - master
     jobs:
-      - build_and_test
+      - build
+      - lint_golang
+      - lint_markdown
+      - unit_test
+      - integration_test
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,17 @@
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Output of the go coverage tool, specifically when used with LiteIDE
+*.out
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+dynamodb-adapter
diff --git a/.mdl.rb b/.mdl.rb
@@ -0,0 +1,17 @@
+# Enable all rules by default
+all
+
+# Asterisks for unordered lists
+rule 'MD004', :style => :asterisk
+
+# Nested lists should e indented with four spaces.
+rule 'MD007', :indent => 4
+
+# Allow table and code lines to be longer than 80 chars
+rule 'MD013', :code_blocks => false, :tables => false
+
+# Ordered list item prefixes
+rule 'MD029', :style => :ordered
+
+# Spaces after list markers
+rule 'MD030', :ul_single => 3, :ul_multi => 3, :ol_single => 2, :ol_multi => 2
diff --git a/.mdlrc b/.mdlrc
@@ -0,0 +1 @@
+style '.mdl.rb'
diff --git a/docs/contributing.md → CONTRIBUTING.md b/docs/contributing.md → CONTRIBUTING.md
@@ -25,4 +25,4 @@ information on using pull requests.
 ## Community Guidelines
 
 This project follows [Google's Open Source Community
-Guidelines](https://opensource.google/conduct/).
+Guidelines](https://opensource.google/conduct/).
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,50 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM golang:1.12
+
+
+# Set the Current Working Directory inside the container
+WORKDIR /go/src/db-driver
+
+# Force the go compiler to use modules
+ENV GO111MODULE=on
+
+# We want to populate the module cache based on the go.{mod,sum} files.
+COPY go.mod .
+COPY go.sum .
+
+#This is the 'magic' step that will download all the dependencies that are specified in
+# the go.mod and go.sum file.
+# Because of how the layer caching system works in Docker, the  go mod download 
+# command will _ only_ be re-run when the go.mod or go.sum file change 
+# (or when we add another docker instruction this line)
+RUN go mod download
+
+# Copy everything from the current directory to the PWD(Present Working Directory) inside the container
+COPY . .
+
+# Set active environment
+ARG ACTIVE_ENV
+ENV ACTIVE_ENV ${ACTIVE_ENV}
+
+# Download all the dependencies
+# https://stackoverflow.com/questions/28031603/what-do-three-dots-mean-in-go-command-line-invocations
+#RUN go get -d -v ./...
+
+# Install the package
+RUN go install -v ./...
+
+# Run the executable
+CMD ["db-driver"]