Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable to run Local network with a consensus node and mirror node #1

Merged
merged 47 commits into from
Mar 2, 2022
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
9a41a9e
initial commit for single docker-compose.yml
Feb 18, 2022
774c3b7
update pointers
Feb 18, 2022
360e3a7
get latest changes
Feb 18, 2022
4f2bf4a
working changes for node
Feb 19, 2022
5fb19a8
add .gitkeep
Feb 21, 2022
b5c611b
add .gitkeep
Feb 21, 2022
0bd6e3a
update pointers
Feb 21, 2022
53fea2a
remove committing to logs
Feb 21, 2022
910d48b
disable monitor
Feb 21, 2022
8d1f028
change address book file
Feb 21, 2022
81a7eaa
create bucket in minio at startup
Feb 22, 2022
8a08aa3
add uploaders .. but not working
Feb 22, 2022
49fb5b1
changes for syncing to minio bucket
Feb 22, 2022
7bc8764
Updated minio config methodolgy
nathanklick Feb 23, 2022
d276c38
Update .env file
nathanklick Feb 23, 2022
85ebfaf
working record stream uploader
Feb 23, 2022
a17ebde
uploaders for eventstreams and accountbalances
Feb 23, 2022
180cbca
working uploaders
Feb 23, 2022
79f8259
fix extension for account balances
Feb 23, 2022
7fae48f
remove rosetta, monitor as per review comments and add bind mount to …
Feb 23, 2022
4408d53
remove redis and use polling
Feb 23, 2022
3c547b9
remove submodules and update README.md
Feb 24, 2022
325e846
remove event streams uploaders and set JAVA_HEAP_MAx to 2g
Feb 24, 2022
7e1cef8
Update gitignore and remove old address book (not in use)
nathanklick Feb 24, 2022
5a99ee6
Added forced mapping of the network-node IP
nathanklick Feb 24, 2022
95efcdb
Fix node IP address
nathanklick Feb 24, 2022
4c46adb
Fix for broken YAML
nathanklick Feb 24, 2022
e11043e
Prevent network-node from using hedera user (UID: 2000)
nathanklick Feb 24, 2022
cd4a852
Disable event streams
nathanklick Feb 24, 2022
5f1e653
Fix uploader
nathanklick Feb 24, 2022
f1c1854
Fix case of bucket path for record streams
nathanklick Feb 24, 2022
00ca5b7
Apply memlimits to importer and grpc
nathanklick Feb 24, 2022
dc5eb74
Apply memlimits to web3
nathanklick Feb 24, 2022
d87137a
Apply memlimits to rest
nathanklick Feb 24, 2022
4673f86
Increase importer mem limits
nathanklick Feb 24, 2022
c7ebe44
add README.md
Feb 24, 2022
99755da
remove empty settings
Feb 24, 2022
8e2bf7c
remove explicit branch name
Feb 24, 2022
7209ee4
remove not needed log
Feb 24, 2022
b17a25d
keep only one address in addressBook.txt
Feb 24, 2022
5989b5e
address a few review comments
Feb 25, 2022
ad7313c
delete .idea folder
Feb 25, 2022
787e158
add minio-server/data/ to gitignore
Feb 25, 2022
05c5769
split to separate networks
Mar 2, 2022
b6466c2
delete network-logs folder and create minio bucket runtime. Remove gc…
Mar 2, 2022
72014b7
update README and add volume mount for minio data
Mar 2, 2022
ed1caae
ignore all files in network-logs
Mar 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions .env
Original file line number Diff line number Diff line change
@@ -1,2 +1,35 @@
TAG=0.11.0
REGISTRY_PREFIX=gcr.io/hedera-registry/
#### Image Names & Prefixes ####
HAVEGED_IMAGE_PREFIX=gcr.io/hedera-registry/
NETWORK_NODE_IMAGE_PREFIX=gcr.io/hedera-registry/
NETWORK_NODE_IMAGE_NAME=main-network-node

#### Image Tags/Hashes ####
UBUNTU_IMAGE_SHA256=ceed028aae0eac7db9dd33bd89c14d5a9991d73443b0de24ba0db250f47491d2
POSTGRES_IMAGE_SHA256=7ced77a458b7d6f4f3e4abfa55d6428fdcd0016e1926cb7463250b54b06480a1
NETWORK_NODE_IMAGE_TAG=0.22.0
HAVEGED_IMAGE_TAG=0.22.0

#### PostgreSQL settings ####
POSTGRES_DB=fcfs
POSTGRES_USER=swirlds
POSTGRES_PASSWORD=password

#### Java Process Settings ####
JAVA_HEAP_MIN=256m
JAVA_HEAP_MAX=2g
JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xlog:gc*:gc.log"

#### Bind Mount Settings ####
MINIO_DATA_PATH=./network-logs/minio-server/data
NETWORK_NODE_LOGS_ROOT_PATH=./network-logs/node
APPLICATION_ROOT_PATH=./compose-network/network-node

#### Network Node Memory Limits ####
NETWORK_NODE_MEM_LIMIT=8gb

#### MirrorNode settings ####
MIRROR_POSTGRES_IMAGE=postgres:13.5-alpine
#### MINIO settings ####
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin

157 changes: 154 additions & 3 deletions .gitignore
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,3 +1,154 @@
compose-network/
db/
data/
## Temporary node files
network-logs/node/logs/*.log
network-logs/node/*.log

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should ignore log files generated anywhere.

Suggested change
network-logs/node/logs/*.log
network-logs/node/*.log
*.log

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Modified

network-logs/node/eventsStreams/**
network-logs/node/recordStreams/**
network-logs/node/accountBalances/**


### JetBrains template
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839

# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf

# Generated files
.idea/**/contentModel.xml

# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml

# Gradle
.idea/**/gradle.xml
.idea/**/libraries

# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# *.iml
# *.ipr

# CMake
cmake-build-*/

# Mongo Explorer plugin
.idea/**/mongoSettings.xml

# File-based project format
*.iws

# IntelliJ
out/

# mpeltonen/sbt-idea plugin
.idea_modules/

# JIRA plugin
atlassian-ide-plugin.xml

# Cursive Clojure plugin
.idea/replstate.xml

# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties

# Editor-based Rest Client
.idea/httpRequests

# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.ser

### Linux template
*~

# temporary files which can be created if a process still has a handle open of a deleted file
.fuse_hidden*

# KDE directory preferences
.directory

# Linux trash folder which might appear on any partition or disk
.Trash-*

# .nfs files are created when an open file is removed but is still being accessed
.nfs*

### Windows template
# Windows thumbnail cache files
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db

# Dump file
*.stackdump

# Folder config file
[Dd]esktop.ini

# Recycle Bin used on file shares
$RECYCLE.BIN/

# Windows Installer files
*.cab
*.msi
*.msix
*.msm
*.msp

# Windows shortcuts
*.lnk

### macOS template
# General
.DS_Store
.AppleDouble
.LSOverride

# Icon must end with two \r
Icon

# Thumbnails
._*

# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent

# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk

### GPG template
secring.*


6 changes: 0 additions & 6 deletions .gitmodules

This file was deleted.

5 changes: 5 additions & 0 deletions .idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions .idea/hedera-network-e2e.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 12 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
# How to run a a mirror node with a local services code
# How to run a LocalNode with a mirror node and consensus node

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# How to run a LocalNode with a mirror node and consensus node
# How to run a local node with a mirror node and consensus node

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


### Setup

Clone this repository and initialize submodules:
1. Clone this repository.
2. Run `docker-compose up -d` from the console.
3. After the run do `docker-compose down` to stop and remove the containers.

`git submodule update --init --recursive`
### Folder set up
1. `compose-network` folder has the static files needed for starting Local network.
2. `network-logs` folder will have all the log files generated after starting local node.
3. `network-logs/node/gc.log` and `network-logs/node/stdout.log` should not be removed from the repository even if they are empty.

### Start hedera-services
### NOTE
1. Ensure to use Docker Compose version 1.29.2 on macOS, due to known bug in Docker Compose V2.
2. Ensure the `gRPC FUSE for file sharing` and `Use Docker Compose V2` settings are disabled in the docker settings.

`docker-compose -f services/docker-compose.yml -f docker-compose.override-services.yml up -d`

### Start the mirrornode

`docker-compose -f mirror/docker-compose.yml -f docker-compose.override-mirror.yml up -d`
![docker-compose-settings.png](docker-compose-settings.png)
11 changes: 0 additions & 11 deletions application.yml

This file was deleted.

Binary file added compose-network/mirror-node/addressBook.bin
Binary file not shown.
31 changes: 31 additions & 0 deletions compose-network/mirror-node/application.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
hedera:
mirror:
grpc:
listener:
type: SHARED_POLL
importer:
parser:
record:
entity:
redis:
enabled: false
db:
host: 127.0.0.1
loadBalance: true
name: mirror_node
owner: ${hedera.mirror.importer.db.username}
ownerPassword: ${hedera.mirror.importer.db.password}
password: mirror_node_pass
port: 5432
restPassword: mirror_api_pass
restUsername: mirror_api
schema: public
username: mirror_node

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section can be removed as it is the application defaults.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. Thanks

downloader:
accessKey: minioadmin
cloudProvider: "S3"
secretKey: minioadmin
bucketName: hedera-streams
endpointOverride: http://minio:9000
initialAddressBook: "/usr/etc/hedera-mirror-importer/local-dev-1-node.addressbook.f102.json.bin"
network: OTHER
Loading