npm audit --production - list vulnerable packages while ignoring dev dependencies
npm ls <package-name> - trace vulnerable packages to the base level packages that are using them

Typescript - review appropriateness of typescript for this service if the dev knows typescript

Microservices implementation structural review

Team Projects

Schema support for db-migrate

We decided to assign search_path to each microservice user (service and migration users) in terraform to resolve this need.

Developer Productivity

Codeowners file and agreements
Document observability tools and access
- ELK: `kubernetes.container.image :"greenstand/treetracker-api:1.4.0"
  - deployed image version can be found by visiting the root path of any deployed api
  - (such as https://dev-k8s.treetracker.org/messaging/)
  - kubernetes.container.name: "treetracker-api"
  - u: admin p:xRrE6vZDUcRph%1P%0H
- grafana (Performance monitoring)
Document about how deployment to dev environment works.
Figure out if dicpher can work with node16 or if there is a replacement (npm run decrypt)
- Password for dipher is pinned

Improve Documentation

https://www.conventionalcommits.org/en/v1.0.0/
Sealed secrets: treetracker-infrastructure/sealed-secrets/scripts/create-database-secret.sh

Improve Microservices Template

resolve all lint warnings
- Update lint settings to our likely - are there rules we want to enable?
remove co-mocha?
- YES, remove from the template
require lint errors to be removed to commit.
- Reject on warnings in CI, but not in husky
- Reject on errors in husky
- Quaid adding his settings to microservice template
Resolve all vulnerabilities
- Check for vulnerabilities when a new PR comes in.
Resolve dependencies errors for test files
```
  import/no-extraneous-dependencies: [error, { devDependencies: true }]
```
* https://eslint.org/docs/user-guide/configuring/rules
- "import/no-extraneous-dependencies": [ "error", { "devDependencies": ["/*.test.js", "/*.spec.js"] } ]

Helpful Topics

CORS

Disable CORS for working on a microservice API with localhost

Disable CORS for development by importing "cors" and then use it to disable cors for 'development' in each microservice
if (process.env.NODE_ENV === 'development') { log.info('disable cors'); app.use(cors()); }
EXAMPLE w/ cors: https://github.com/Greenstand/treetracker-stakeholder-api/blob/main/server/app.js
EXAMPLE w/ header options: https://github.com/Greenstand/treetracker-web-map-api/blob/master/src/app.js
Chrome extension that will let you disable CORS when you want without having to shutdown
- Simple option: Cross Domain CORS or Moesif CORS - You can click the red button to make it active when you want to disable CORS.
- Lots of settings, but you have to create an account: Requestly
Try different browsers or the proxy setting for CRA. https://medium.com/swlh/avoiding-cors-errors-on-localhost-in-2020-5a656ed8cefa
Create your own proxy server: https://medium.com/nodejsmadeeasy/a-simple-cors-proxy-for-javascript-applications-9b36a8d39c51

Helpful reading!

Debugging: https://httptoolkit.tech/blog/how-to-debug-cors-errors/ Explain settings: https://web.dev/cross-origin-resource-sharing/ CORS module settings: https://expressjs.com/en/resources/middleware/cors.html

Domain Migration

Treetracker Domain Migration Sequencing

Migration Sequence

Migrate all legacy 'tree' table data into the fielddata.raw_capture table
1. Includes creation session and device records for these captures
Admin panel application: split capture verification tool from capture search tools.
Go through the migration process for capture verification
1. Documentation: https://github.com/Greenstand/system-design-docs/blob/master/domain-migration/project_plan.md
2. Steps
  1. Milestone 1: Get rabbitmq verification messages functioning all the way through into production
    Get it working on dev infra
    Release and test on test infra
    Release and test on beta and prod infra
  2. Milestone 2: Update admin panel to move approved data from field_data.raw_capture to treetracker.capture in the verification process (using the treetracker API), but for capture search still pulls from legacy database (using the legacy API)
    Get it working on dev infra
    Release and test on test infra
    Release and test on beta and prod infra
Migrate all remaining approved legacy 'tree' table data into the treetracker.capture table
1. Apply to dev
2. Apply to test
3. Apply to prod
Consolidation step to fix all planting organization assignments
1. Priority between planting org per planter, per tree, and reported in field data to be determined
2. Clean up all capture records to have valid organization assignments in the stakeholder tool.
Update admin panel to using treetracker API for capture search tools.
*Now, ready to get the capture matching tool working in deployment*
GIS task: Identify very small geographic area in Freetown as a pilot for capture matching, and supply this as a spatial shape file (.geojson or .shp)
GIS task: Identify all the capture records in the small geographic area that are the initial captures of the tree.
Extract production data from small geographic area and add that data to dev and test database.
Data task: insert treetracker.tree records for each initial capture. Now we have trees to capture match against.
1. Perform in dev, test, and prod
Implement capture matching frontend feature to allow filtering trees and captures by named geographic area. We will filter for initial use by the small geographic area defined above, which should be loaded into the regions API.
Now we can deploy the capture matching tool to the beta infrastructure and try it out.

Notes:

Some legacy planter registrations reported to not have device identifier.
1. This is inconvenient but not fixable with the v2 bulk pack format

Domain Migration M2

Goal: Admin panel to not read trees table directly. Build new treetracker service with API to create capture records.

Build a new service to be the source of truth of all approved captures.

This service is to own all the capture records along with tagging and other details created during the approval in the admin panel.

Service is already created, known as treetracker-api.
Some verbs (GET/POST/PATCH) may be missing but capture, tree, and planter resources have been defined.
We may need to add additional search parameters as needed by other services or for domain goals.

On creation of a capture record, emit events indicating the creation of the capture record to a Topic. This event is meant for web-map to build a view of all approved captures.

This appears to be implemented, but has not been tested or operated

The scope of this capture service is yet to be determined if it can own more than just capture records. We could call this as captures service or treetracker-capture-records.

We added planter (ground_user) and tree to this service already. That will be the scope of the service. The service is called treetracker serivce, and is stored in the treetracker-api repository

Change Admin panel

Change Admin panel to use field data API instead of directly reading `trees` table for tree capture verification.

Data populated in the admin panel verify tool, should ONLY be raw captures from the field data service that has not been verified (status=unprocessed)
This will require the admin panel frontend team to split the current verify tool into a 'verification' tool and a 'capture edit' tool.
Only verified captures can be 'edited' through the 'capture edit' tool, and only very limited fields can be edited. mostly this is tagging.
Later, there will also be some tools for cleaning field data, but that's out of scope for now.

Admin panel to invoke capture service API to record the approved capture, update the trees table as usual and then update field data service to mark the data as approved/rejected.

(1) POST to treetracker/capture
(2) PATCH to the legacy admin panel API
(3) PATCH back to the field service to mark as approved/rejected in field data schema.
TODO: Think through what happens if one of these calls fails.
- I think it's ok, it just causes some work to be repeated.
- For instance, if POST to (1) succeeds but PATCH to (2) fails,
  - Then raw capture will need to be reverified (because (3) did not execute)
  - So the data will just be reprocessed by the operator in this case
- We will probably resolve this by using an orchestration layer
  - The treetracker service itself can handle the orchestration, and also keep track of failed requests to (2) and (3) to retry.
  - This could also use a queue, or implement a separate service to handle the orchestration.
  - To some extent, this decision is up to the engineer implementing this step of the domain migration.

Remove the feature that emitted event for approved captures from Change 1 earlier (in the legacy API). The reason we will write to trees table from admin panel (or orchestration layer) is for back-ward compatability with web-map.

Add a consumer for the capture creation events in the web-map layer and build a view for all captures in web-map specific db.

This appears to already be complete. Need to be tested and operated.

Migrate data from trees table in main db to captures table in the new captures service of all approved tree entries.

ETL job. Can be implemented as one-off airflow job.

Update token generation process and refer to ids from captures instead of trees as reference association.

This refers to the token minting scripts, now being run in airflow
capture_id for each token needs to reference the id of treetracker.capture rather than public.trees. This also requires updating the HATEOAS link in the token payload (wallet API) to point to the new capture service.
Update all existing tokens to use treetracker.capture.id as capture_id

Wallet API service to emit events when transfer of tokens occur to be consumed by web-map layer for Impact owner view.

The events consumed in web-map layer will trigger api calls to get the tokens impacted and update the capture view in the web-layer to assign the updated wallet owner.
Requires endpoint to look up all tokens in any transfer by transfer id.

Trigger discussion about impact manager map (the view in web-map for planter orgs) in the web-map layer and its needs to shape and update milestone 3.

Discuss how organizational hierarchy is captured in the webmap schema (JSON blob or table structure)

Address domain naming issues (variable and class names)

Breaking changes will trigger a new major revision, and a new deployment mount point
- Frontend engineers will be responsible for migrating to the new mount point
- If database changes are necessary, if the service is not in production it's ok to make a breaking change. Otherwise check with engineering leadership.
planter, grower, and fielduser should all become 'ground_user'
planteridentifier should become 'ground_username'
planter class, table, and API resource should become 'ground_user'

Domain modeling notes

‘Ground user’ and ‘individual’ are synonyms. Also ‘person’

A mobile app user registers a wallet when using the mobile phone. Wallet registrations are uploaded along with their corresponding captures. A wallet is created for the wallet registration if it does not already exist. A grower account is created for each wallet, to track the identity of the person until they can be verified. Once a person is verified, a person record is created for them in the stakeholder system. Person records can receive payments for treetracker services.

Treetracker Schema Attributes

**Capture**

Not null, default uuid(), immutable

reference_id

Nullable, immutable

session_id

Not null, immutable

tree_id

Nullable, immutable but clearable

image_url

Not null, immutable

lat

Not null, immutable

lon

Not null, immutable

gps_accuracy

Not null, immutable

estimated_geometric_location

Not null, immutable

estimated_geographic_location

Not null, immutable

planter_id

Not null, immutable

planter_photo_url

Not null, immutable

planter_username

Not null, immutable

planting_organization_id

Not null, immutable

device_configuration_id

Not null, immutable

species_id

Nullable, mutable (for now)

morphology

Nullable, mutable (for now)

age

Nullable, mutable (for now)

attributes

Nullable, immutable

note

Nullable, immutable

domain_specific_data

Nullable, immutable

status

Not null, mutable

created_at

Default now() on create, not null, immutable

updated_at

Default now() on create or update, not null, mutable

**Tree**

Not null, default uuid(), immutable

latest_capture_id

Not null, mutable

image_url

Not null, mutable

lat

Not null, immutable

lon

Not null, immutable

gps_accuracy

Not null, immutable

estimated_geometric_location

Not null, immutable

estimated_geographic_location

Not null, immutable

species_id

Nullable, mutable (for now)

morphology

Nullable, mutable (for now)

age

Nullable, mutable (for now)

attributes

Nullable, immutable

status

Not null, mutable

created_at

Default now() on create, not null, immutable

updated_at

Default now() on create or update, not null, mutable

**Tag**

Not null, default uuid(), immutable

name

Not null, immutable

public

Not null, mutable

status

Not null, mutable

created_at

Default now() on create, not null, immutable

updated_at

Default now() on create or update, not null, mutable

**Capture_Tag**

Not null, default uuid(), immutable

capture_id

Not null, immutable

tag_id

Not null, immutable

status

Not null, mutable

created_at

Default now() on create, not null, immutable

updated_at

Default now() on create or update, not null, mutable

**Tree_Tag**

Not null, default uuid(), immutable

tree_id

Not null, immutable

tag_id

Not null, immutable

status

Not null, mutable

created_at

Default now() on create, not null, immutable

updated_at

Default now() on create or update, not null, mutable

**Grower_Account**

Not null, default uuid(), immutable

wallet_id

Not null, immutable

wallet

Not null, immutable

person_id

Nullable, uuid of stakeholder

organization_id

Nullable, uuid of stakeholder

name

Not null, mutable

Nullable, mutable - one of email or phone required to be non-null

phone

Nullable, mutable - one of email or phone required to be non-null

image_url

Not null, mutable

image_rotation

Default 0, not null

status

Not null, default active

first_registration_at

Immutable, timestamp of first field registration

created_at

Default now() on create, not null, immutable

updated_at

Default now() on create or update, not null, mutable

**Status Enumeration** active deleted Session - Field Data Schema Device Configuration - Field Data Schema

Implementation Pathway

Release treetracker API into all envs
Web map query schema & consumer into all env
1. Release
2. Validate data flow for raw captures
3. Validate data flow for captures
4. Validate or implement cluster generation
5. Validate or implement
Implement web map for raw captures in field data schema
Finalize and Release Capture Matching Tool
Finalize and Release Earnings Tool
Finalize and Release Stakeholder Tool
Migrate admin panel users to keycloak
Migrate web map to captures table in treetracker schema

Capture Verification

Update to 'verified' status requested to treetracker API
Start transaction against treetracker schema to insert a new capture record
1. If the record already exists, that's OK, just accept.
Call legacy API and update capture (trees table record) to approved
Commit transaction on treetracker schema
Call field data API and update raw_capture record to approved

if (3) fails, we rollback the treetracker schema transaction, therefore the raw_capture record remains unprocessed, and will be reprocessed by an operator in the future, should bring data into consistency.

if (4) fails, same as if (3) fails, we no update. Will be reprocessed

if (5) fails, the record will be reprocessed

When we POST to captures/, send the UUID of the raw_capture to become the UUID of the approved capture record - these should always be unique.

Allows for easy reprocessing when there are outages
Allows for multiple concurrent POST messages to be collapsed and only create 1 capture record

Bulk Pack

Bulk Pack Processing v2

Field data schem

Ingestion Process by Entity Type

wallet_registration

Wallet registrations need to be stored in the field data schema in the wallet_registrations table, but they also need to be used to upsert (create or update) grower account records in the treetracker API. A wallet registration also needs the grower account id field populated.

Suggested workflow:

Bulk pack transformer calls PUT on grower_account in treetracker API for grower account with identifier that matches wallet
1. If account exists, update
  1. Only update the name, phone, and email for now
2. Else, insert
  1. Insert identifier, name, phone, email, and photo
POST to /planter path on bulk-pack-transformer-1 to populate legacy schema
Populate grower account id in wallet registration data, and POST into field data api
1. If wallet registration id is already present, field data api will do nothing (return 200)
Return 200

device_configuration

device configuration records are simply inserted in v2, but in v1 they were upserted.

Suggested Workflow

POST device configuration payload to bulk-pack-transformer-1
1. bulk-pack-transformer-1 returns 200 if already present
POST device configuration payload to field data API
1. field data API returns 200 if already present
Return 200

session

session does not exist in the v1 bulk-pack-transformer.

Suggested Workflow

POST to session path on field data API
1. field data API returns 200 if already present
Return 200

capture

POST capture data to bulk-pack-transformer-1
1. If data exists this API returns 200, else a new record is inserted
2. This API also propagates the record to field data API, where a queue message is emitted
POST capture data to field data API
1. Since step 1 already propagated to this API, a 200 should be returned based on uuid
2. Since 200 is returned, queue message should not re-emitted

Orchestration Diagram

Improved orchestration architecture

Deprecated proposal for orchestration

Nullable Columns for Field Data Tables

Session

track_url
organization

Wallet Registration

phone or email is nullable, but not both

Raw Capture

session_id is nullable for v1 endpoints only
reference_id
note
extra_attributes
rejection_reason

Device Configuration

No nullable fields

Workflow Notes

Insert 'tree' as 'capture' will fail until corresponding device and registration records have been inserted, because there's no session Once device and registration records are inserted, then it is possible to insert a session id using SESSION UUID = device_identifier + planter_identifier In v2 bulk pack endpoint /v1/tree is responsible for ensuring that the session record exists for a v1 tree. This session can then be used as the session for inserting the v1 'tree' record as a v2 'capture' record

process sends tree record to /v1/tree in v2
v2 calculates SESSION UUID = device_identifier + planter_identifier using uuid-string
v2 checks for a session that matches this
If no session exists, it looks up device_configuration_id and wallet_registration_id using device_identifier and planter_identifier, and if they exist it creates a session record. If they don't exist yet, it fails until next time.
Assuming a session exists or was inserted in (4), we can now insert the capture record using his session uuid.

Release

Old versions:

greenstand/bulk-pack-processor:1.2.7
greenstand/bulk-pack-transformer:1.6.2

Development Paths

Capture Matching Release Project

Capture Match road map 2021:

Webmap query service

Messaging System Rollout Decisions

We will control available authors manually at first, by populating the authors table
We will control access to admins using our normal process
For announce and survey messages, we look up associated Grower Accounts in the treetracker API, and then match those against enabled authors
1. We need to extend this approach to respect the organizational hierarchy, which may mean a query service for grower accounts.
2. A query service for grower accounts would also more easily respect the region filters

Testing Methodology

Unit Tests

Integration Tests

Only access the API, do not perform database check
- First write seeded tests for GET endpoints
- Then write tests for create and update that use GET endpoint to validate functionality
APIs should return created objects, and get queryable via these object Ids
Seeding should be done using knex, and use the same seeding scenarios provided to the frontend developers
Best Practices
- Do specific seeding for each integration test, don't use a generic seed that isn't for the specific test
- Objects posted to an API should be explicitly expressed within the integration test file (?)
- Use async syntax
  - it('should do something', async function() { ...
- Log request errors in test to facilitate debugging
  - if(res.error ) { console.log(res.error); }
- Stub functions on a per-test basis
Running integration tests
- Use the test-integration-working command to specific a specific test file rather than running all
- Use .only to specify a single test
- Don't leave .only in place after getting the test to pass

TODO
- Need a solution to print response errors when expect fails on an error code
  - https://github.com/visionmedia/supertest/issues/12
- What tests are required?
  - Endpoint behaviors
  - How much validation?
    Payload structure vs 404 responses
    JOI tests could be unit tests

Software Layers

Handler

HTTP Domain
All error and success codes
API payload validation
Calls a service or a model function

Service

Orchestration between services and the domain model
- Database session
- External APIs
- Cloud Services (such as RabbitMQ or S3)

Model

Domain logic
Accesses repositories
Allowances
- We allow HTTP 404 errors to be thrown from within the model as a convenience for now

Repository

Accesses the database, performs CRUD operations
One repository for each database table in RDMS

Grower Unique Identifier Notes

treetracker.grower_account.wallet

is the same as messaging.author.handle
is the same as (public.planter.phone || public.planter.email)

messaging.author.handle

does not necessarily have a corresponding treetracker.grower_account.wallet
does not have a matching (public.planter.phone || public.planter.email) if it does not have a treetracker.grower_account.wallet

Contract Service

Diagram for contract tables:

https://lucid.app/lucidchart/4beb07f7-0ef0-4bf0-b88d-5548c2cda03b/edit?page=0_0#

Greenstand Microservices

Microservices Directory

Domain Services

Treetracker API

Field Data API

Stakeholder API

Earnings API

Regions API

Messaging API

Map Tile Service

Grower Account Query Service

Web Map Query Service Consumer

Bulk Pack Services

Bulk Pack Consumer

Bulk Pack Processor

Bulk Pack Transformer V1

Bulk Pack Transformer V2

Legacy Services

Admin API

Web Map API

Cloud Services

Airflow API

CKAN API

Service Review

Team Projects

Schema support for db-migrate

Developer Productivity

Improve Documentation

Improve Microservices Template

Helpful Topics

CORS

Disable CORS for working on a microservice API with localhost

Helpful reading!

Domain Migration

Treetracker Domain Migration Sequencing

Migration Sequence

Notes:

Domain Migration M2

Build a new service to be the source of truth of all approved captures.

This service is to own all the capture records along with tagging and other details created during the approval in the admin panel.

On creation of a capture record, emit events indicating the creation of the capture record to a Topic. This event is meant for web-map to build a view of all approved captures.

The scope of this capture service is yet to be determined if it can own more than just capture records. We could call this as captures service or treetracker-capture-records.

Change Admin panel

Change Admin panel to use field data API instead of directly reading trees table for tree capture verification.

Admin panel to invoke capture service API to record the approved capture, update the trees table as usual and then update field data service to mark the data as approved/rejected.

Add a consumer for the capture creation events in the web-map layer and build a view for all captures in web-map specific db.

Wallet API service to emit events when transfer of tokens occur to be consumed by web-map layer for Impact owner view.

Trigger discussion about impact manager map (the view in web-map for planter orgs) in the web-map layer and its needs to shape and update milestone 3.

Address domain naming issues (variable and class names)

Domain modeling notes

Treetracker Schema Attributes

Implementation Pathway

Capture Verification

Bulk Pack

Ingestion Process by Entity Type

wallet_registration

Suggested workflow:

device_configuration

Suggested Workflow

session

Suggested Workflow

capture

Orchestration Diagram

Improved orchestration architecture

Deprecated proposal for orchestration

Nullable Columns for Field Data Tables

Workflow Notes

Release

Development Paths

Messaging System Rollout Decisions

Testing Methodology

Unit Tests

Integration Tests

Software Layers

Handler

Service

Model

Repository

Grower Unique Identifier Notes

Contract Service

Change Admin panel to use field data API instead of directly reading `trees` table for tree capture verification.

Change Admin panel to use field data API instead of directly reading `trees` table for tree capture verification.